Author: @atilayberber
Think about a typical week for an ETL developer. A new file format from a source system. Broken dates. Duplicate records. Trying to work out why a pipeline failed overnight. Writing the notebooks that move data from bronze to silver and silver to gold, one by one. And at the end, the inevitable “do the numbers actually reconcile?”
Now imagine writing this instead: “Build bronze, silver and gold layers from our customer, product and sales files. Preserve the raw inputs, standardize and deduplicate records, create daily sales summaries, and verify that revenue totals reconcile.” And then watching an agent plan the work, write the notebooks, run them and check the results.
The Fabric data engineering agent (Project Osmos), built on the Osmos acquisition Microsoft announced in January 2026, went into public preview at FabCon Barcelona. It brought one of the most argued-about questions in the community along with it: is the ETL developer’s job over?
What’s in this post
- What Project Osmos is, and how it differs from a coding assistant
- How it actually works: task settings and safety boundaries
- What it can’t do yet: the limits of the preview
- The two camps: “nobody will hand-write ETL” vs. “it’s just a very fast intern”
- My take: not an ETL developer anymore, an “outcome engineer”
- A safe plan for teams that want to try it
What is Project Osmos?
Coding assistants like GitHub Copilot suggest code. Running it, checking the result and fixing what breaks is still your job. Osmos goes a step further. According to Microsoft Learn:
- You describe a complete outcome: sources, transformations, outputs and validation rules.
- The agent runs it as one remote task on SparkCore. It plans and sequences the work, creates tables and notebooks, runs the code and checks the results.
- The task runs persistently in Fabric. Close your laptop and the work keeps going. You follow progress in the lakehouse explorer or by asking your local agent.
The difference Microsoft emphasizes: Osmos inspects the live lakehouse, tests options and iterates on real execution results. A general coding assistant writes code without ever seeing the data.
You can’t use Osmos from Copilot in the Fabric portal. It runs through a local agent, and the supported clients are GitHub Copilot CLI, Claude Code and Codex.
How it actually works
Microsoft’s quickstart is, interestingly, a retail scenario: generate sample CSVs for customers, products and sales in an empty lakehouse, build bronze, silver and gold tables plus three notebooks, run them in order, and validate row counts, key uniqueness, referential integrity and revenue totals.
The part I find most valuable is the task settings. Before it starts, the agent recommends values for each of these, and you can change any of them:
| Setting | What it controls |
|---|---|
| Permission boundary | Which resources are read-only and which are writable |
| Safety pattern | Staging, clone-and-promote, or direct iteration |
| Promote step | When and how results move to the real target |
| Rerun semantics | Fail, append, deduplicate or replace |
| Schema evolution | Whether schema changes are allowed |
| Artifact format and destination | Where generated files go, and in what format |
| Reasoning effort | How deeply the agent thinks |
That table is essentially the checklist a good ETL developer carries around in their head. Which is a hint about where this debate actually lands.
What it can’t do yet
The excitement needs some brakes, and Microsoft is refreshingly direct about it:
- The feature is in preview. Capabilities are limited, it isn’t intended for production use, there’s no SLA, and it may only be available in selected regions.
- It’s not available in workspaces with outbound access protection enabled.
- The most important warning: a task can access any resource you have permission to access, including resources outside the target workspace. The permission boundary in the task settings guides the agent, but Fabric and OneLake permissions are the real enforcement boundary. Protect sensitive data with platform permissions, not with the prompt.
That last point is, for me, the key lesson of the preview. “I told the agent to only write to the staging schema” isn’t a security control. It’s a polite request.
The two camps
Camp 1: “Nobody will hand-write ETL anymore”
- Medallion architecture, deduplication, SCD Type 2, incremental loads: these are well-known, repeated patterns. Exactly the kind of work agents are best at.
- The agent doesn’t just write code, it runs and validates it. The risk of “runs but wrong” is lower than with a plain coding assistant.
- Microsoft’s sample outcomes (Dynamics 365 customer lifecycle, SAP order-to-cash, NetSuite financial reconciliation, Salesforce pipeline metrics) are precisely the integrations organizations sink the most effort into.
- A task can run for hours on its own. One engineer can supervise several at once.
Camp 2: “It’s just a very fast intern”
- The hard part of real ETL isn’t the code, it’s the source system’s quirks: an undocumented status code, a time zone that changes at midnight, a branch that used a different product coding scheme three years ago. Those live in people’s memories, not in the data.
- A preview isn’t production. Enterprise ETL expectations around reliability, traceability and rollback aren’t met yet.
- “You provide the validation criteria” means leaving the hardest part to a human anyway. Someone who doesn’t know what correct looks like can’t write correct criteria.
My take: not an ETL developer anymore, an “outcome engineer”
Both camps are right, and here’s how I reconcile them: the agent takes over the “how.” The “what” and “is it right?” stay with people.
Look at that task settings table again. Staging or direct write? Append or replace on rerun? Can the schema change? None of these are technical details. They’re business risk decisions. In a retailer, rerunning the sales table in “replace” mode versus “append” mode can be the difference between a correct month-end revenue report and one where every sale appears twice.
So the ETL developer’s job doesn’t vanish. It moves:
- Before: writing notebooks, debugging, rebuilding the same deduplication logic on every project.
- Now: defining the outcome, writing validation criteria (row counts, null checks, uniqueness, reconciliation), choosing the safety pattern, reviewing what the agent produced, and feeding source-system knowledge to the agent.
I call this new role the “outcome engineer.” And in that role the most valuable skill isn’t Python or Spark. It’s being able to answer “what does correct data look like?” An ETL developer who has spent years wrestling with source systems starts this new era with a real advantage for exactly that reason.
A safe plan for teams that want to try it
- Start with a separate workspace and lakehouse. Don’t touch production data during the preview.
- Narrow permissions at the platform level. The identity running the task shouldn’t have access to sensitive resources. Don’t rely on the boundary in the prompt.
- Give it a job you already know. Have the agent rebuild a hand-written pipeline whose output you trust, and compare the results.
- Write validation criteria before the task. Don’t leave them to the agent’s suggestions. Define the acceptance tests yourself.
- Use staging or clone-and-promote. Keep direct iteration for purely experimental work.
- Put generated notebooks through code review. Code the agent wrote is as much your responsibility as code you wrote.
Conclusion
Project Osmos has moved “agents in data engineering” from a demo to a working preview. It isn’t production-ready today, and Microsoft says so plainly. But the direction is clear: the ETL developer who writes notebooks by hand will be replaced by an engineer who defines the outcome, sets the criteria and supervises the agent. And the person who knows the source systems best will be the most valuable person in that transition.
Where do you stand? Would you hand your own pipelines to an agent, and under what conditions? Let’s talk in the comments or on LinkedIn.
Sources
- Get started with Fabric data engineering agent (Project Osmos) (Microsoft Learn)
- Announcements from FabCon Barcelona 2026 (James Serra)
- Microsoft Acquires Osmos To Bring Autonomous Data Engineering to Fabric (Redmond Magazine)
STAY IN THE LOOP
New guides, sorted by topic in a single email.
Every Monday, the week's new posts arrive grouped by topic. No new posts, no email.