Introducing the data engineering agent for Fabric (Project Osmos)

AI coding assistants are excellent at turning natural-language transformation instructions into code. For data engineers, the harder question is whether that code produces results they can trust.

You can describe transformation logic in natural language, and an AI coding assistant will happily generate hundreds of lines of Spark code. Impressive.

But a Spark job can execute successfully and still produce completely wrong results. The logic may work for a subset of the data but fail when run against the full dataset. The pipeline's logic and validation checks may not account for future changes to the data or schema. Incremental loads can cause duplicates. Revenue numbers might look perfectly reasonable until someone reconciles them against the source.

Anyone who's spent time working with real-world enterprise data knows that automating data engineering with AI comes with a unique set of challenges.

Data engineering involves more than code: it requires working with large datasets and retaining context about their schemas, relationships, and business rules over time.

Quality & governance matter a lot. The cost of errors or data loss is very high.

Data engineering work typically spans team-based, collaborative, long-running, milestone-driven projects; not a single conversation between one person and a coding agent.

What if specialized, data-aware agents could help teams investigate these problems, implement solutions, and validate the results—not just generate code?

Following Microsoft's acquisition of Osmos in January 2026, we've been working on bringing autonomous data engineering capabilities as a native experience in Microsoft Fabric.

The idea behind Project Osmos is simple: instead of asking AI to help you write data engineering code, you define a complete problem and the outcome you expect. It has the autonomy to use the tools and an orchestration framework to build, validate, troubleshoot, and refine its implementation, making decisions within the boundaries you set and working towards a high quality output.

The task's code execution and data operations run within your selected Fabric lakehouse.

Real-world data engineering is rarely straightforward

Let's take an insurance use case to illustrate the complexity.

Imagine an insurance company consolidating policy and claims data from enterprise systems such as SAP, Guidewire, and Oracle, a common challenge in an industry that grows through acquisitions. The project also needs to ingest external datasets and user-generated data.

The objective is to build a consolidated view of a claim, including its policy, coverage, and financial history.

Sounds reasonable, until you start looking at the data.

The same person, vehicle, or incident could have different IDs across systems. The same ID could also refer to completely different incidents across systems. Reconciling payments and ledger entries needs to be done with extreme caution. Discrepancies could result in overpayments and wrong forecasts.

You need to investigate the data, understand its relationships, establish the business rules, and validate that the resulting view is consistent with those rules. Getting it wrong can have serious downstream implications.

Quality and accuracy matter more than speed.

In this example, a team would ask the Project Osmos to investigate relationships across the datasets, implement the agreed business rules, and validate the consolidated claims view. The agent would autonomously work through the problem and regularly report updates to the user. Users can review the evidence and validate results before accepting the output.

A purpose-built orchestrator

Under the hood, Project Osmos uses a purpose-built orchestrator that coordinates multiple specialized agents to tackle the problem. Each agent has a well-defined role from code generation to validation to evaluation.

The agents interpret instructions, analyze the data, evaluate implementation approaches, generate code in isolated sandboxes, and verify the outputs. They use execution results to guide further iterations, with the aim of meeting your validation and quality requirements. Your team reviews the evidence to determine whether the results are acceptable.

Should the agent need additional information or get blocked, or if the agent determines that the task cannot be accomplished within the constraints, the user will be elicited for input and guidance, and task will pause waiting user input.

Project Osmos isn't merely simulating what the code might do. It submits PySpark code through the Fabric Livy API for execution in Fabric Spark sessions, so it can test the implementation and validate actual results. Whether changes use copies, staging tables, or in-place updates depends on the write pattern you select.

The goal isn't to remove humans from the process. It's to let the agents take on the investigation and implementation work, while allowing people to guide the outcome, resolve ambiguities, and review the evidence.

Figure: Project Osmos coordinates knowledge, Spark worker, and validation agents in Fabric.

Autonomy, with the right guardrails

Of course, giving an AI agent the autonomy to execute code and modify data raises the most important question: How do you make sure it doesn't do something it shouldn't? Before a task starts, you review the operating guardrails recommended by Project Osmos to define the scope and impact of its work. You specify which resources it can use, where outputs should go, whether changes should be staged, how reruns should behave, and whether schema changes are permitted.

Project Osmos also recommends a reasoning effort setting, which users can adjust. More extensive reasoning can be useful for complicated problems, but it also comes at the cost of additional time and compute.

These instructions guide the agent's work, but they aren't a substitute for proper access controls. Agents operate within the user's permissions, and Fabric and OneLake permissions continue to enforce what the agent is authorized to access.

And while Osmos can carry out extensive validation, your team ultimately determines whether the results meet the business requirements.

A persistent task that lives in Fabric

Real-world data engineering problems rarely get completed in a single sitting. Requirements change. New data arrives. Business rules get clarified. Often implementations that worked perfectly the previous month need to be revisited.

With Project Osmos, work is organized into persistent tasks hosted in Microsoft Fabric. Each task represents an ongoing data engineering project, rather than a single chat exchange.

Once created, a task runs independently in Fabric, consuming Fabric capacity units (CUs). You can start it from GitHub Copilot CLI, Claude Code, or Codex using the Skills for Fabric plug-in, then disconnect without stopping the work. You don't need to keep your terminal open, maintain an active chat session, or even keep your laptop running.

The task maintains its state, context, execution history, and outputs. You can monitor its progress in your Fabric Lakehouse, inspect what the agents are doing, review errors, and examine the outputs.

The task is reusable and can evolve with additional instructions. You can interrupt it to provide additional context, clarify a requirement, or pause and resume the work through your local coding agent.

Say you've built a data pipeline, and a new data drop arrives the following week. You don't have to start another task and explain the entire project all over again. You can reference the existing task and ask Project Osmos to process the new data.

It has the context of the previous work and can decide whether to reuse the existing implementation or update it to account for the new data.

Figure: Animated GIF of a persistent Project Osmos task and its activity history in Fabric.

Collaborative by design

Complex data engineering projects are rarely a one-person job.

They involve a team of data engineers, data architects, business analysts, data stewards, and domain experts. And each of these people brings something different to the table.

Getting the details right often requires conversations between people who understand different aspects of the problem.

Project Osmos allows authorized teammates to work with the same task, inspect what's been done, and provide additional guidance.

So instead of passing notebooks back and forth or having different people initiate separate AI conversations with slightly different assumptions, the team can collaborate on the same ongoing work.

The task retains the context as requirements evolve, making it possible for someone joining later to understand what has already been attempted and contribute to the next steps.

Figure: Teammates collaborate on a shared Project Osmos task.

What about the data engineering work you've already done?

So far, we've talked mostly about building new data engineering projects. But a significant amount of engineering effort also goes into maintaining and improving what's already there.

This is also work you can delegate to Project Osmos.

For instance, you could ask Project Osmos to convert an existing pipeline to an incremental load and validate its rerun behavior. You could also ask it to optimize Spark code in existing notebooks, improve the performance of Delta tables in your lakehouse, or assist with migrating data engineering workloads from another platform to Fabric while preserving business rules and validating the results.

Give Project Osmos a try!

Getting started with Project Osmos is straightforward.

Install the Skills for Fabric plugin in GitHub Copilot CLI, Claude Code, or Codex, sign in to Azure, and ask your coding agent to create a Project Osmos task.

Point it to your Fabric workspace and Lakehouse, describe the outcome you want, and specify how you'd like the results to be validated.

You'll need Contributor or higher workspace permissions, the required tenant settings, and an eligible Fabric capacity. Trial capacities are not supported. Review the quickstart for prerequisites and preview limitations before you begin.

We've put together a few resources to help you get started:

• Project Osmos Quickstart: A step-by-step walkthrough, including a sample medallion architecture project you can run in an empty Lakehouse.
• Project Osmos Overview: A deeper look at how tasks work and what the agent can do.
• Best Practices: Guidance for defining project outcomes, setting boundaries, and specifying validation requirements.

Have a messy dataset, a pipeline that needs fixing, or a migration you've been putting off? Give Project Osmos a try and see what it can do!