How are AI agents changing modern data engineering workflows?

A practical approach is to treat AI agents as an intelligence layer around the data platform, rather than letting them directly control everything.

In a Fabric setup, the pattern I’ve seen making the most sense is:

• Fabric Pipelines handle orchestration and dependencies.
• Notebooks / Spark handle transformations and data processing.
• Data quality rules and monitoring detect failures, schema changes, volume anomalies, or unexpected data patterns.
• An AI agent sits on top of these signals, analyzes logs and pipeline metadata, and helps explain the issue or suggest the next action.
• For documentation, the agent can use metadata, schemas, notebooks, and transformation logic to generate and maintain dataset documentation.

For example, if a pipeline fails because of a schema change, the agent shouldn't blindly fix and rerun it. It can identify the failure, compare the new schema with the previous version, explain the impact, and recommend whether the pipeline or source needs attention.

The key best practice is human-in-the-loop automation: let agents investigate, summarize, recommend, and handle low-risk repetitive tasks, while keeping production-impacting changes under proper controls.

I think this approach gives teams the benefits of AI without turning the data platform into a black box.