Databricks Pushes ETL Further Into Declarative SQL

Databricks is making ETL more declarative, so teams can build governed pipelines with less orchestration code and more built-in control.

Updated

What is this trend?

Databricks is moving ETL from procedural pipeline code into declarative SQL primitives, making data workflows easier to govern, observe, and maintain.

  • APPEND, AUTO CDC, and REPLACE WHERE reduce custom orchestration in SQL ETL.
  • Built-in lineage, observability, and data-quality controls tighten pipeline governance.
  • Lakebase keeps transactional app data inside Databricks for more end-to-end workflows.
  • The platform is shifting engineers from wiring pipelines to designing data contracts.

What’s the latest?

Databricks’ latest update adds APPEND, AUTO CDC, and REPLACE WHERE to its SQL ETL stack, extending the platform-control story from governed production AI into how data pipelines themselves are built and run.

Governed Agent Execution, AI Trust Scores, and Declarative SQL Reshape Data Work

How it developed

  1. Agent Control Planes Tighten, Production AI Gets Governed, and ML Becomes Audit-Ready

Go deeper

Curated long-form picks on this trend — podcasts, videos, and analysis, by seniority.

Stay ahead in Data Science & Machine Learning

Get the weekly Data Science & Machine Learning brief in your inbox — the developments, what they mean by seniority, and what to do next.