Copilot Studio, Foundry, Fabric Data Agent or Operations Agent? Which one to pick
Most confusion comes from the word "agent" being used for all four. The fastest way to choose is to ask two questions: who builds it (maker or developer) and what triggers it (a person asking, or data changing). Meet the four Copilot Studio is Microsoft's low-code agent builder on Power Platform. You describe the agent in plain language, add knowledge (SharePoint, Dataverse, websites), plug in 1,000+ connectors and Power Automate flows, then publish to Teams, Microsoft 365 Copilot or a website. It is also the easiest place to orchestrate other agents, including Foundry and Fabric agents. Microsoft Foundry (formerly Azure AI Foundry) is the pro-code platform for engineers. You pick from a large model catalog, write orchestration in code, run evaluations, trace every call and deploy behind private networking. Use it when the agent is a product component, not just a chat window. Fabric data agent is a read-only, conversational analyst over your Fabric data. It turns a question into SQL, DAX or KQL against up to five sources (lakehouse, warehouse, Power BI semantic model, KQL database, ontology) and respects each user's permissions. It is generally available. Fabric operations agent lives in Real-Time Intelligence. You give it an Eventhouse, a business goal and allowed actions. It writes its own monitoring rules, wakes up when a condition hits, reasons over the data, and messages people in Teams or runs a Power Automate action. It reached general availability in mid-2026. Which one should I pick? Ask the questions in order and stop at the first "yes". A "yes" does not rule the others out: a data agent or Foundry agent is often published through Copilot Studio. Side by side Copilot Studio Foundry Fabric data agent Fabric operations agent Core job Converse and automate Build custom AI apps and agents Answer questions over data Monitor live data and act Triggered by A person (or event/flow) Your code A person or another agent A data condition Builder Maker, IT, business AI engineer, developer Data/BI team Ops and RTI team Skill level Low-code Pro-code Low-code + good data modelling Low-code, chat-based setup Data it reaches Connectors, SharePoint, Dataverse, other agents Anything you wire up Lakehouse, warehouse, semantic model, KQL, ontology Eventhouse (KQL) streams Writes or acts? Yes, via flows and tools Yes, whatever you code No, read-only Yes, Teams messages and Power Automate actions Where users meet it Teams, M365 Copilot, web, other channels Your app, API, Teams Fabric, Copilot Studio, Foundry, M365 Copilot, MCP Teams chat or channel Billing Copilot Credits Azure consumption Fabric capacity (CU) Fabric capacity (CU) Pick it when You need speed and M365 reach You need model choice, evals, isolation People Real-world scenarios These are the patterns that come up most often in the community. Each names the tool that fits and why. HR policy helper in Teams → Copilot Studio. A 3,000-person company wants staff to ask "How many leave days do I have?" and "Book Thursday off." Knowledge sits in SharePoint, leave balances in an HR system with a connector. A maker builds it in days and publishes to Teams. No developer needed. Insurance claims triage inside a customer app → Foundry. An insurer reads uploaded photos and PDFs, extracts fields, scores fraud risk and returns a decision to its claims system through an API. It needs a vision model, evaluation datasets, tracing and a private network. That is a product feature, so it belongs in Foundry. Sales leaders asking about pipeline → Fabric data agent. "Which regions missed Q3 target, and by how much?" The data agent sits on the sales semantic model, writes the DAX, and only shows rows each person may see. Publish it into Microsoft 365 Copilot or Copilot Studio so leaders never open Fabric. Cold-chain monitoring for a food distributor → Fabric operations agent. Truck temperature sensors stream into an Eventhouse. The goal: "Keep chilled loads between 2 and 5 °C." When a truck drifts, the agent posts in the fleet Teams channel with the truck, route and trend, and can trigger a Power Automate flow to reroute or open a ticket. Retail "what should I do?" assistant → all four together. Microsoft's Zava reference demo shows this: an operations agent spots a stock risk, a Fabric data agent pulls the numbers, Foundry agents for sales and product reason over them, and Copilot Studio is the front door in Teams where the manager approves a reallocation. Better together The strongest designs use more than one. Copilot Studio meets people where they work, Foundry adds custom reasoning, and the two Fabric agents keep answers and actions grounded in governed data. The operations agent is the only one that starts on its own: it reads live events and reaches people in Teams without anyone asking first. Status as of October 2026, and gotchas Fabric data agent is GA, and its Copilot Studio integration went GA at FabCon Barcelona (October 2026). Data agents in Microsoft Copilot Chat were only a sneak peek there. The data agent inside Copilot in Power BI was retired on 26 August 2026. Use the data agent directly, or through Foundry, Copilot Studio, Microsoft 365 Copilot or its MCP endpoint. Operations agent is GA (June to July 2026). Newer abilities, such as root-cause investigations and pre-approved actions, are still in public preview. Operations agents need a paid Fabric capacity. Trial capacity is not supported, and an admin must turn on the Copilot/AI and operations agent tenant settings. A data agent is only as good as the model under it. Clear table names, descriptions, relationships and agent instructions matter more than prompt tricks. An operations agent automates your mistakes too. Start with "recommend" actions that a person approves, then move to automatic actions. Don't pick Foundry just for governance. Both Copilot Studio and Foundry agents now show up in the same Microsoft governance and security tooling. Pick on build style, model needs and network needs. Rule of thumb: prototype in Copilot Studio for a week. If you hit a wall (model choice, evals, isolation, scale), that wall is your Foundry requirement. Sources Operations agent (Generally Available), Fabric Updates Blog Improving operational efficiency with operations agents, Fabric Updates Blog Build, deploy, and govern Fabric at scale (FabCon 2026), Fabric Community FabCon Barcelona 2026 announcements, James Serra Retirement of data agent in Copilot in Power BI, Fabric Community What are Fabric data agents: GA status and limits, Atlan Fabric data agent vs operations agent, Daymark One agent, many channels: querying Fabric data agents, Data Science at Microsoft Copilot Studio vs Microsoft Foundry, Nasstar Zava Real-Time Intelligence demo, Microsoft CDXFabric Data Agents Get Smarter on 2 November. Can You Prove Yours Still Answers Correctly?
Microsoft announced Advanced NL2SQL and advanced DAX generation as generally available on 19 October 2026. The core improvements roll out to the standard runtime by default, together with a move to GPT-5.6 Terra. Separately, the old custom-tool way of calling a Fabric data agent from Microsoft Foundry stops being supported on 6 November. Date (2026) What happens What you should have done 19 Oct GA announced Read this post Now to 1 Nov Changes available on the preview runtime Build a ground-truth set, baseline standard, test preview 2 Nov Standard-runtime rollout begins, no opt-in Baseline frozen and stored 6 Nov Foundry custom-connection integration unsupported Foundry agents moved to Fabric IQ By 10 Nov Rollout expected to complete Re-run, diff the queries, tell affected users What Microsoft actually announced The query generators stop writing in one pass and start investigating first. That is why behaviour can change even where accuracy improves. Until now, NL2SQL and NL2DAX generated a query in a single pass, resolving schema, business definitions and filter values at once. As the GA announcement puts it, a wrong guess still runs, and the wrong answer looks like a right one. Advanced NL2SQL explores the schema, tests joins and looks up real filter values before writing SQL. It works in three modes: Example query following: adapts a relevant example query, for instance by swapping a filter value. Clarifying questions: asks the user when it can't answer reliably. Data exploration: infers logic from your examples and checks assumptions against the data. Advanced DAX generation brings the same idea to semantic models: an agentic tool-calling loop plus instance value indexing, so a user's "California" maps to the stored "CA". Microsoft's internal benchmarks Microsoft's 2026 internal evaluations, standard versus preview runtime. Microsoft notes results vary by workload, model and configuration. Benchmark Smart-answer accuracy P50 latency P95 latency NL2SQL, standard 78.8% 28.4 s 46.8 s Advanced NL2SQL, preview 90.9% 26.0 s 32.9 s NL2DAX, standard 89.15% 39.1 s 97.5 s Advanced DAX, preview 91.86% 20.4 s 34.7 s SQL gains 12.1 accuracy points. DAX gains only 2.7 points, but its tail latency drops by about 64%. If your semantic-model agents felt slow, that is your headline. What moves, and what doesn't Moving to standard: Advanced NL2SQL, advanced DAX generation and the GPT-5.6 Terra model upgrade. Staying in preview: schema descriptions and topics for SQL sources. Two details catch people out (runtime docs): A published agent keeps the runtime it was published with. Publish on preview and it stays on preview until you republish. Check every production agent. Runtime choice doesn't pick the model. A preview run today is a strong signal, not a guaranteed copy of standard after 10 November. Re-run after the rollout regardless. Why "better on average" can still surprise you A benchmark counts right answers. Your users notice changed answers. The second one generates support tickets. Workaround instructions start fighting the agent. Rules written to patch single-pass behaviour may no longer be needed, and unneeded guidance can steer an agent away from a path it would have found itself. Example queries get followed more literally. Good when they are current. An example with last year's fiscal logic now gets reproduced more faithfully. Ambiguous questions get a question back. Better for users, but it breaks anything downstream that expects a number, such as a Copilot Studio flow or a Foundry agent chain. A correct new answer can be a business problem. "Active customers in Q3" returned 41,200 because of a wrong date column; now it correctly returns 38,900. Someone may already have put 41,200 in a board pack. The pass count hides the churn. 40 of 50 pass before, 44 after. Looks like +4. Underneath, 6 newly pass and 2 now fail: 8 changed answers, 2 regressions. Question status Count Action Pass both, same query 34 Nothing Pass both, different query 4 Right for the right reason? Failed before, passes now 6 Tell anyone who acted on the old answer Passed before, fails now 2 Fix instructions or examples first Fail both 4 Existing backlog Illustrative numbers. The structure is what your own run should produce. The pre-rollout routine: six steps for this week You need a Fabric notebook, a lakehouse and the Fabric data agent SDK. The SDK and its evaluation features are in preview, so note the version you used. You also need F2 or higher capacity, the data agent tenant settings enabled and read access to the sources. Step 1: Inventory agents and dependencies For each published agent, record the runtime it was published on, its source types, where it is consumed (Fabric, Teams, M365 Copilot, Copilot Studio, Foundry) and its business owner. Anything called from Foundry via the old custom tool also goes on your 6 November list. Step 2: Build a ground-truth set that covers the risky cases The SDK takes a DataFrame or CSV with question and expected_answer. Aim for 30 to 60 questions per agent, ideally from real usage, covering filter values ("California" stored as "CA"), example-adjacent questions, multi-join questions, ambiguous questions and high-stakes KPI figures. %pip install -U fabric-data-agent-sdk import pandas as pd gt = pd.DataFrame(columns=["question", "expected_answer"], data=[ ["Total net sales in California for September 2026", "1,284,550.20"], ["Returns rate by region for FY26", "North 3.1%, South 2.7%, East 4.0%, West 3.4%"], ["What was revenue last year?", "CLARIFY: should ask gross or net, fiscal or calendar"], ["Active customers in Q3 2026", "38,900"], ]) gt.to_csv("/lakehouse/default/Files/agent_eval/ground_truth_sales.csv", index=False) The values are placeholders. Use answers your business owner has signed off. Step 3: Use a critic prompt that accepts clarifying questions The default grader would mark a good clarifying question as wrong. The SDK accepts a custom critic_prompt with {query}, {expected_answer} and {actual_answer} placeholders. critic_prompt = """ Query: {query} Expected Answer: {expected_answer} Actual Answer: {actual_answer} If the expected answer starts with CLARIFY:, respond 'yes' only if the actual answer asks a clarifying question about the same ambiguity. Otherwise respond 'yes' only if the actual answer is numerically and semantically equivalent. Is the actual answer equivalent to the expected answer? """ Step 4: Capture the standard baseline before 2 November Run against the production stage, which is what users hit today. Give each run its own table. from fabric.dataagent.evaluation import ( evaluate_data_agent, get_evaluation_summary, get_evaluation_details) agent_name = "SalesAgent" baseline_table = "eval_sales_standard_baseline_2026_10" baseline_id = evaluate_data_agent( gt, agent_name, table_name=baseline_table, data_agent_stage="production", critic_prompt=critic_prompt) print(get_evaluation_summary(baseline_table)) baseline = get_evaluation_details(baseline_id, baseline_table, get_all_rows=True) print(baseline.columns.tolist()) # check column names in your SDK version This writes <table_name> with per-question results and <table_name>_steps with the detailed reasoning and step-by-step execution. Look there for the generated SQL or DAX, and keep both tables as evidence. For a single question, the Diagnostics button exports the full steps too. Step 5: Record P50 and P95 latency through the MCP endpoint The evaluation tables focus on correctness, so time each question against the published agent's MCP endpoint. This adapts Microsoft's SDK sample: import time, numpy as np from azure.identity import InteractiveBrowserCredential from mcp import ClientSession from mcp.client.streamable_http import streamablehttp_client mcp_url = ("https://api.fabric.microsoft.com/v1/mcp/workspaces/<workspace-id>" "/dataagents/<data-agent-id>/agent") cred = InteractiveBrowserCredential() async def ask(q): tok = cred.get_token("https://api.fabric.microsoft.com/.default").token async with streamablehttp_client(mcp_url, headers={"Authorization": f"Bearer {tok}"}) as (r, w, _): async with ClientSession(r, w) as s: await s.initialize() tool = (await s.list_tools()).tools[0] arg = next(iter(tool.inputSchema["properties"])) t0 = time.perf_counter() await s.call_tool(tool.name, {arg: q}) return time.perf_counter() - t0 secs = [await ask(q) for q in gt.question for _ in range(3)] print(f"P50 {np.percentile(secs, 50):.1f}s P95 {np.percentile(secs, 95):.1f}s") Run each question several times; single timings are noisy. The MCP response returns answer text only, not the query, which is why query evidence comes from the steps table. Step 6: Preview now, standard after 10 November, then diff Preview, today. Switch your draft to preview and evaluate the sandbox stage, leaving production untouched. The switch comes from the runtime docs (the ribbon's Runtime dropdown works too); check the SDK reference for the FabricDataAgentManagement import. data_agent = FabricDataAgentManagement(agent_name) data_agent.update_configuration(enable_preview_features=True) preview_id = evaluate_data_agent(gt, agent_name, table_name="eval_sales_preview_2026_10", data_agent_stage="sandbox", critic_prompt=critic_prompt) data_agent.update_configuration(enable_preview_features=False) # don't publish on preview by accident Standard, after 10 November. Repeat Steps 4 and 5 against production with new table names. The diff. Compare per question, never just the summary. Fill in your SDK's pass-flag and answer column names. def diff_runs(before, after, passed="<pass-col>", answer="<answer-col>"): m = before.merge(after, on="question", suffixes=("_b", "_a")) def status(r): b, a = r[passed + "_b"], r[passed + "_a"] if b and a: return "same" if str(r[answer + "_b"]) == str(r[answer + "_a"]) else "answer changed" return "newly correct" if a else ("REGRESSION" if b else "fail both") m["status"] = m.apply(status, axis=1) return m.sort_values("status") Then compare the generated queries from both _steps tables. A changed query with the same answer deserves a look: it may be right by coincidence on this month's data. When a result changed, check the run steps for: which example query it picked, which join path it chose, which instruction it followed, and whether a clarifying question was the right call. Most fixes will be in your examples and instructions, not the platform. The 6 November deadline: move Foundry agents to Fabric IQ If a Foundry agent calls your data agent through the original custom tool, that path stops being supported on 6 November 2026 (retirement notice). The catch: the replacement, the Fabric IQ (OneLake Catalog) tool, is in preview. Production agents move from a deprecated integration to a preview one, four days into the query rollout. Make that an explicit risk decision, with a fallback plan. What you gain: pick agents by name from the catalog, connect several data agents to one Foundry agent (the old tool allowed one), and an MCP endpoint per agent. Microsoft also says responses will soon include the query used. Portal migration, in Microsoft's order: Publish the data agent with a rich, detailed description. In the Foundry portal, open the agent and add the Fabric IQ (OneLake Catalog) tool, then select your data agent. Update the Foundry agent's instructions to name each data agent and what it handles. Remove the old custom tool, save, and test with representative questions. For code, use the MCP URL from the agent's Settings > Model Context Protocol tab and the FabricIQPreviewTool in azure-ai-projects>=2.2.0. Full samples are in Connect agents to Microsoft Fabric with Fabric IQ. Test the two changes separately. If you migrate on 5 November and answers shift on 8 November, you won't know why. Migrate now and baseline through Foundry before 2 November. The connection calls Fabric as the signed-in user, so test with accounts that match real users' permissions and row-level security. When the new answer is right, who tells people? The hardest part is not technical. Someone acted on the old number. Decide now who owns that conversation: Each agent has a business owner who reviews "newly correct" and "answer changed" rows. High-stakes answers get a change note: old answer, new answer, why, and since when. The note goes where the old answer was used: report owners, Teams channels, Copilot Studio makers. Regressions get fixed, not avoided. Standard is changing anyway, so plan the fix. Also tell users before 2 November that answers may become more precise and the agent may ask follow-up questions. A warned surprise becomes a feature. Your checklist Inventory agents, runtimes, consumers and owners Build ground-truth sets, including ambiguous and high-stakes questions Write a critic prompt that accepts correct clarifying questions Capture the standard baseline, steps and latency before 2 November Run the set on preview in the sandbox stage Migrate Foundry agents to Fabric IQ and baseline them before 6 November Decide explicitly on running production on the preview Fabric IQ tool Re-run after 10 November and diff per question and per query Retire workaround instructions and refresh stale examples Send change notes for high-stakes answers that movedMicrosoft Fabric Security Policies: How They Strengthen Data Security and Governance?
In today’s data-driven world, governance isn’t just about compliance — it’s about enabling trust, consistency, and scale. As organizations adopt Microsoft Fabric to unify their analytics, the challenge of managing who can do what, where, and under which conditions become critical. That’s where Microsoft Fabric policies step in. Think of them as the rulebooks that let administrators define guardrails across workspaces, security groups, and sensitivity labels. Instead of relying on manual oversight, policies provide a centralized, automated way to enforce governance — ensuring that innovation doesn’t come at the cost of control. In this blog will walk you through the key concepts, the entry points for creating and activating policies, and the end-to-end lifecycle that makes governance in Fabric both powerful and practical. Prerequisites: 1) A Valid Fabric Trail / Capacity License 2) A Fabric workspace assigned to a Capacity/Fabric Trail. 3) Proper admin role to activate the policy a) Fabric administrator: Can create and activate policies for tenant-scope policies. b) Capacity administrator: Can create and activate policies for capacity-scope policies. 4) Contributor, Member, or Admin access to the workspace so that you can create a policy item type. 5) The Policies tenant setting is enabled in Admin settings 6) Valid Security groups in AAD. Step 1: Enable the Policies feature in the admin settings Go to fabric admin settings, search for polices(preview) and enable the setting and click on apply. Step 2: Creating Policy item Create a workspace ‘Fabric-Polices-Ws’ which was assigned to ‘Fabric Trail’ and I’m (SI Babu) who’s the capacity admin. Make sure this should be a secured, admin-managed workspace to store policies. Click on ‘New Item’ and select ‘Policy set (preview)’ under ‘Govern data’ In New policy set dialog, enter a unique and descriptive Name ‘Capacity_PolicySet_1’ for the policy set and choose ‘ A capacity’ under policy scope then Click on ‘Create’ Create another policy set ‘Tenent_Policy_set2’ and choose ‘ Fabric Tenant’ under policy scope then Click on ‘Create’ Two policy sets are created. Step 3: Activating the Policy set Open Capacity policy set and select the ‘Active’ from Status Select the capacity and click on ‘Assign + activate’ Policy activated successfully Open Tenant policy set and select the ‘Active’ from Status Click on ‘Activate’ the policy Policy activated successfully Step 4: Create / Add Policy Rules(Capacity Policy) Click on ‘OneLake catalog’ Click on ‘Govern’ à‘Polices(preview)’ àCapacitiesàExpand the ‘Users can create items’ and select ‘Advanced’ Here, Select Allowed, Blocked, or Advanced (Set by user) to change the policy behavior. If you select Advanced, the rules configuration is enabled. Add a new rule with good name, description and Select ‘item type’ in Condition type, operator is ‘Is any of’ the select ‘Lakehouse’ and ‘Dataflow gen2’ then click on ‘+Add condition’ Choose ‘workspace’ in condition type, operator is ‘Is any of’ selected two workspaces gen2’ then click on ‘+Add condition’ Choose ‘Security group’ in condition type, operator is ‘Is any of’ selected my ‘Data Analyst’ AD group. Click on Apply Note: In this Data Analyst group we have three developers are there Click on “Save’ Now Policy successfully added Step 5: Validate policy enforcement (Capacity Policy) Added ‘Data Analyst’ AD group to both the workspaces ‘FabricRTI Training’ & ‘PowerBI-Demo-WS’ Now I was opened another window with account(SP Babu) who’s part of AD group & you can see two workspaces Create a ‘Notebook’ Now when I click on “Create” it’s throwing an error as you seen in below screenshot. Means it enforced a policy Again, When I was trying to create ‘Warehouse’ it’s throwing the same error. I (SP Babu) was able to create ‘DataFlowGen2’. Admin enforced to create only this item I (SP Babu) was able to create ‘Lakehouse’. Admin enforced to create only this item Step 6: Create / Add Policy Rules(Tenant Policy) Create another policy should be at tenant level to restrict workspace Outbound access protection and inbound network rules and click ‘add condition’ Add “Security group’ as I have IT Infra Team and click Apply Note: In this IT Infra Team group we have three resources are there Security policy created successfully Step 7: Validate policy enforcement (Tenant Policy) Now SP Babu unable to edit ‘Outbound access’ it’s showing ‘This setting is managed by a tenant policy.’ SP Babu unable to edit ‘Inbound access’ it’s showing ‘This setting is managed by a tenant policy.’ You can deactivate or replace an active policy set from the workspace where the policy set is stored. Only an administrator of the policy set's admin scope can deactivate or replace it. Best practices for creating a policy: Use a secured, admin-managed workspace to store policies. Store all policies for the admin scopes managed by the same administrator or admin team in the same workspace. This approach provides one location for tenant- and capacity-scoped policies and simplifies access management, discovery, and lifecycle management. Use clear policy set and rule names that identify the admin scope and purpose. Use test users who are members and non-members of the configured security groups to validate both allow and deny outcomes before broader rollout. Limitations as of now: Fabric policies aren't currently supported in the following capacity regions: West Europe, North Europe, and West US. We can currently create policy sets only from the item creation experience in a workspace. Only workspaces assigned to a Fabric capacity can store policy sets. We can't change a policy set's admin scope after creation. Only one policy set can be active for an admin scope at a time. Policy and tenant-setting changes can take time to propagate. Conditions support Any of and None of in the current preview. We can't activate a policy set through Git. So, start small with pilot policies, monitor audit logs for adoption and Communicate changes to workspace owners Conclusion Microsoft Fabric Policies are still in preview, but they signal a future where governance is not a blocker — it’s an enabler of responsible innovation. Happy learning! — Inturi Suparna Babu [LinkedIn]Spark Runtime 2.0 - Practical Overview and Stress Testing (with a built framework)
Link to the notebooks to reproduce: https://github.com/iurii-iurchenko-1/source-materials/tree/main/2026Q3_spark_runtime_2_0 Outline 1. What is Runtime 2.0 and why does it matter? 2. The five features explained 2.1 Vectorized CSV parsing in the Native Execution Engine 2.2 ANSI SQL mode inside the native engine 2.3 Incremental liquid clustering 2.4 Arrow-native Python UDFs 2.5 The VARIANT type in Delta tables 3. What are the main improvements as shown in stress testing 4. How did I test that? – Building the framework – The three notebooks – The tables and what each one measures 5. Main conclusions 5.1 If you are an architect 5.2 If you are an engineer 6. Resources used 7. Notebook results 7.1 Runtime 1.3 metrics 7.2 Runtime 2.0 metrics 8. How to reproduce 9. Links to documentation What is Runtime 2.0 and why does it matter? What is Runtime in Spark notebooks? It is an environment that includes the version of Spark, the default version of Delta tables, and other components that provide the engine to run Spark Notebooks in Fabric. The previous version of the runtime was Runtime 1.3, which included Spark 3.5. The latest Runtime 2.0 includes Spark 4.x and an upgraded version of Delta Tables. I chose 5 particular features introduced in Runtime 2.0 and built a framework to measure the difference in performance between versions 1.3 and 2.0. These features are: Vectorized CSV parsing in the Native Execution Engine ANSI SQL mode inside the native engine Incremental liquid clustering Arrow-native Python UDFs The VARIANT type in the delta tables Let me explain these features in depth. 1 Vectorized CSV parsing in the Native Execution Engine This feature means the engine-level optimization of reading CSV files. It may help if your organization has heavy jobs to ingest CSV files at scale. 2 ANSI SQL mode inside the native engine This feature is powerful, and not a lot of people know about it. If you run Spark queries and a formula fails to calculate some values, Spark quietly returns NULL. If ANSI is enabled, the failing will be loud. It is needed in critical pipelines. Now that feature is supported by the native engine. 3 Incremental liquid clustering This new feature influences the behavior of the OPTIMIZE command on top of a Delta table. In the old version, the behavior was to reprocess all the files. In the new version, only new unclustered files will be reprocessed. Isn't that good? 4 Arrow-native Python UDFs This feature introduces the new optimized algorithm to process Pandas UDF functions. When are they needed? During a migration of the Python/Pandas notebooks. So, instead of rewriting Pandas functions and introducing new bugs, implementing this mechanism may help. IN the new engine, it is optimized to avoid converting to Pandas internally, which saves time and resources. Also, if you know PyArrow functions to proceed with strings, they may also be utilized instead of plain Python. 5 The VARIANT type in the delta tables Variant type promises performance optimization on top of JSON-like data. So, it looks like you decoded the data source as JSON, wrote it once (as a VARIANT column), and it may be processed faster later, when you need to extract some information from it. What are the Main Improvements as Shown in Stress Testing Final Results CSV Scan: I observed ~50% performance improvement on reading CSV files in Runtime 2.0 (9.2 sec vs 6.5 sec) ANSI SQL mode - significant improvement in Runtime 2.0 (see the code and logs at the end of the notebook). 5.3 sec ANSI off →12 sec ANSI ON on Runtime 1.3 vs 3.6→3.7 on Runtime 2.0. Incremental liquid clustering - the new algorithm introduces incremental clustering. So, now, we may have two strategies: incremental clustering or full rebuilding of Z-cubes. So, the introduces incremental clustering gives a significant boost in performance, but still loses the battle compared with full optimization/rebuilding of Z-cubes. Arrow-native Python UDFs give almost a 3x boost on the new runtime due to avoiding converting it to Pandas and back Variant type. Here, I observed that Variant may not give a significant boost in reading performance. Sometimes, it may have even slower performance. But it uses disk space better. How did I test that? To test that, I vibe-coded a framework in Claude to cover all the features mentioned above. It was a back-and-forth process. I carefully selected 5 features recently introduced to Runtime 2.0, built tech requirements for the system to test it, generated the code for three notebooks, ran it, got the first issues and output, improved it, and repeated it a few times till the process was smooth. I tested it in my personal MS Fabric environment with F2 capacity. The solution includes three notebooks: NB_00_GENERATE_DATA NB_10_BENCH_RUNTIME_1_3 NB_20_BENCH_RUNTIME_2_0 The tables are: fact_sales — 20M-row Delta fact table. Feeds the ANSI × NEE arithmetic query and all four UDF variants. Files/bench/raw_csv — the same 20M rows written as uncompressed CSV, 2.1 GB. The only asset that tests vectorized CSV parsing in the native engine. logs_json — 3M nested JSON documents held in a STRING column. The baseline for the VARIANT comparison, and the source for the VARIANT migration. events_lc_r13 — liquid-clustered Delta table, left with unoptimized new data on top. Measures the Delta 3.2 full-rewrite OPTIMIZE on Runtime 1.3. events_lc_r20_inc — identical twin of the above. Measures Delta 4.2's incremental OPTIMIZE on Runtime 2.0. events_lc_r20_full — identical triplet. Measures OPTIMIZE ... FULL on Runtime 2.0, giving a same-session comparison against the incremental arm. logs_variant — not from the generator; created by the Runtime 2.0 notebook itself via parse_json. Its creation time isthe VARIANT migration cost, and it is what the colon-operator query reads. After that, I ran the notebooks NB_10_BENCH_RUNTIME_1_3 and NB_20_BENCH_RUNTIME_2_0 to get the main metrics. You may find these metrics below. They show how new features in Runtime 2.0 differ in performance for different use cases. Main Conclusions If you are an architect The main point with Runtime 2.0 is that it becomes GA. That means the older versions will be deprecated step-by-step, and it is time to plan the modernizational upgrade of the current infrastructure to use Runtime 2.0. If you are an engineer Invest your time in learning about NEE (Native Execution Engine) features, and test how they impact your real data flow. They are efficient, especially on complex logic query Arrow native Python UDFs may be 3x faster compared to ordinary UDFs if you use only the library's functionality and don't use Python variables there. This library is rich. So, it is possible, especially for text operations. If you didn't know what ANSI SQL mode is, now you know. If you want to Spark fails loudly if some formula didn't take place, that's it. Now, it is much more supported by the engine. Resources Used I tested all that on a personal tenant with Fabric Capacity F2. Notebooks Results Runtime 1.3 Metrics Runtime 2.0 Metrics How to Reproduce? Import the notebooks mentioned at the beginning of the article to your Fabric tenant Create a Lakehouse "LH_RUNTIME2_0_TEST Run NB_00_GENERATE_DATA Sequentially, run NB_10 & NB_20, creating before runtimes 1.3 and 2.0 for them. Links to Documentation https://community.fabric.microsoft.com/blog/fbc_fabricupdatesblogs/fabric-august-2026-feature-summary/5325824 https://community.fabric.microsoft.com/t5/Fabric-Updates-Blog/Fabric-July-2026-Feature-Summary/ba-p/5325823 https://learn.microsoft.com/en-us/fabric/data-engineering/runtime-2-0 https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview?tabs=sparksqlI Let Fabric's New Data Engineering Agent Run My ETL: Here's What Happened
Last week I handed Microsoft Fabric's new Data Engineering Agent a messy ETL job that took me 11 hours to build by hand last quarter. It had a working, validated pipeline in front of me in under an hour. I still changed 3 of its 9 steps before I'd let it near anything important. That is the short version. The long version is more interesting, because the places where it went wrong tell you exactly how to use it well. Some context first. At FabCon Europe 2026 in Barcelona (September 28 to October 1), Microsoft announced the Data Engineering Agent in preview. It can plan and run migrations, ETL and performance work under guardrails that engineers define. It builds on technology from Osmos, the agentic data engineering startup Microsoft acquired in January 2026, and was demoed on stage as "Project Osmos". Copilot has been helping us write code for a while now. This is a different idea. You describe the outcome, and the agent does long-running, multi-step work on its own while you supervise. I wanted to know whether that holds up on a real, slightly ugly dataset rather than a keynote demo. So I tried it. What the Data Engineering Agent actually is The agent is an autonomous worker inside your Fabric workspace, not a chat assistant that suggests code. You give it a goal, and it does the multi-step work itself. Its roots are in Osmos, a Seattle startup whose "AI Data Engineer" already generated production-grade PySpark notebooks inside Fabric. Microsoft bought the company in January 2026 and folded the team into Fabric engineering. Based on Microsoft's FabCon session on Project Osmos, a typical run looks like this: Read the request. You describe the outcome, even loosely ("land the orders CSVs in a clean silver table"). Inspect the workspace. It looks at your existing code, Lakehouses and data layout before writing anything. Plan. It breaks the job into steps and can hand parts to sub-agents for long-running work. Build. It creates and updates notebooks and other Fabric items. Validate. It checks its own output so you can trust what was built. Hand back to you. A human reviews and approves. Microsoft is explicit that supervision stays part of the process. The headline use cases are migrations, ETL and performance tuning. I chose ETL, because that is where most of us spend our week. The test: a real pipeline, not a demo I picked a job I had already built by hand, so I could judge the agent's work line by line against my own. The data is sales orders from a mid-sized retail client . Every day their order system drops a CSV export into a Lakehouse folder. The business wants a clean orders table for analysts and a daily sales summary for a Power BI report. Item My setup Source data 14 daily CSV exports, 18.4 million rows, about 2.1 GB, in Files/raw/orders/ Target silver_orders (cleaned Delta table) and gold_daily_sales (aggregate by day, store and category) Known traps Duplicate order IDs from system retries, two date formats, a currency column with nulls, cancelled orders flagged as negative quantities Capacity F64 trial capacity, sandbox workspace Hand-built baseline About 11 hours across two days, including testing The exact prompt I gave it: Ingest the daily order CSVs in Files/raw/orders into a cleaned silver Delta table called silver_orders. Remove duplicates, standardise dates and currency, and exclude cancelled orders. Then build gold_daily_sales with total revenue, order count and average basket size by day, store and category. The pipeline must run incrementally each day. I deliberately kept the prompt a little vague. I didn't say which date formats exist, how cancellations are encoded, or which column identifies a duplicate. In real life, requirements rarely arrive as a perfect spec. I wanted to see whether the agent would investigate, ask, or just guess. What happened, step by step 1. It read the workspace before writing anything The first thing I noticed: it didn't start coding. For the first four minutes it profiled the data. It sampled each CSV, inferred the schema and looked at my existing notebooks in the workspace. It came back with a short summary of what it found. Two findings impressed me: It spotted both date formats (2026-09-14 and 14/09/2026) and flagged that files from 9 September onwards switched format. It noticed my existing notebooks used a nb_<layer>_<entity> naming pattern and said it would follow it. It missed one thing, which matters later: it reported quantity as "integer, some negative values" without asking what negative quantities meant. 2. The plan it proposed Next it produced a 9-step plan and stopped for my approval: Create a staging notebook to read all CSVs with an explicit schema Normalise both date formats to a single order_date column Fill null currency values from the store's default currency Deduplicate on order_id, keeping the latest record Remove cancelled orders Write silver_orders as a Delta table, partitioned by order_date Build gold_daily_sales with the three requested measures Add incremental logic based on the source file date Run validation checks and produce a summary report This is a sensible plan. It is roughly what I wrote on a whiteboard last quarter. But reading it closely, I changed three things before approving: Step 4: it planned to overwrite silver_orders on each run. I changed it to a MERGE on order_id, because late-arriving corrections are common in this data. Step 5: it planned to remove cancellations using a status column. That column exists, but this client never fills it. Cancellations show up as negative quantities. I told it so. Step 8: it wanted to detect new data by file modified time. I asked it to use the date in the file name instead, which is more reliable when files are re-uploaded. It accepted all three changes and rewrote the plan in a few seconds. 3. The build Once approved, it worked for 38 minutes without needing me. It created three notebooks (nb_stg_orders, nb_silver_orders, nb_gold_daily_sales) and a small pipeline to run them in order. The code was clean and readable. Here is its date handling, which is close to what I would write myself: from pyspark.sql import functions as F df = df.withColumn( "order_date", F.coalesce( F.to_date("order_date_raw", "yyyy-MM-dd"), F.to_date("order_date_raw", "dd/MM/yyyy") ) ) And the merge into silver, after my change in the plan: from delta.tables import DeltaTable silver = DeltaTable.forName(spark, "silver_orders") (silver.alias("t") .merge(updates.alias("s"), "t.order_id = s.order_id") .whenMatchedUpdateAll() .whenNotMatchedInsertAll() .execute()) It asked me one question mid-way: whether average basket size should be revenue per order or items per order. That was exactly the right question, and one I had to ask the client myself last time. 4. Self-validation When the build finished, it ran its own checks and produced a short report: Row counts reconciled from raw to staging to silver, with each dropped row explained (duplicates, cancellations) No nulls left in order_date, currency or store_id No duplicate order_id values in silver_orders Daily totals in gold_daily_sales matched a direct aggregate of silver_orders One check failed on the first pass. 41,200 rows had a null order_date after parsing. It traced the cause itself: a third date format (14-Sep-2026) appeared in two of the older files, which its initial sampling had missed. It added a third pattern to the coalesce, re-ran, and the check passed. That moment sold me on the validation step. A human could easily miss 0.2% of rows quietly disappearing. 5. My review against my own version I compared its output with the pipeline I built by hand last quarter, running both on the same 14 files. Measure Agent Me, by hand Time to working pipeline 52 minutes (incl. my review) About 11 hours Plan steps I changed 3 of 9 Not applicable Data issues found 4 of 4, after one self-correction 4 of 4 Full run time on 18.4M rows 6 min 40 s 5 min 55 s Silver row count 17,962,118 17,962,118 The row counts matched exactly. My hand-built version ran about 45 seconds faster, mostly because I had tuned partitioning. More on that below. Would I merge its code as-is? After my three plan changes, yes, with one small performance fix. Guardrails: the part you shouldn't skip An agent that can create and change items in your workspace needs limits before it needs a prompt. Microsoft designed this agent around guardrails that engineers define, and setting them up was the best 15 minutes of the whole test. What I set up before the run: A sandbox workspace. It's a preview feature, so it ran on a copy of the data, nowhere near production. Scoped data. It could read Files/raw/orders/ and write only to new tables. My existing tables were off limits. Approval points. It had to stop for my approval after the plan, and again before the first write to any Delta table. Git in the loop. The workspace was connected to a Git repo, so every notebook it created or changed landed as a diff I could review line by line. In practice, it never tried to step outside these limits. The approval after the plan turned out to be the most valuable one: that is where I caught all three of my changes. If you set only one guardrail, make it that one. Where it struggled No agent review is credible without the failures, so here are the three that mattered. It assumed instead of asking about business rules. It saw negative quantities and planned to use an empty status column for cancellations. Had I approved the plan without reading it, about 310,000 cancelled orders would have flowed into the gold table and inflated revenue. It asked a good question about basket size, but not this one. Its sampling missed a rare date format. The third format appeared in only two files. Its validation caught the problem, but only after the first run. On a much larger dataset, that first run costs real capacity. Correct, but not tuned. It partitioned silver_orders by order_date, which produced many small files for this data volume. Switching to monthly partitions and running OPTIMIZE closed most of the 45-second gap with my version. When I asked it to "make the silver write faster", it suggested the same fix, but it didn't do it unprompted. Three caveats worth knowing before you try it: It's a preview. Features, behaviour and pricing can change before general availability. Don't build production processes on it yet. Watch your capacity. Long-running agent work consumes capacity units on top of the Spark jobs themselves. Check the Capacity Metrics app after your first run so you know what a typical job costs. It doesn't know your business. It can read your schema, not your meeting notes. Every rule I wrote into the prompt was handled correctly. Every rule I left out was a guess. The verdict The Data Engineering Agent is like a fast, tireless junior engineer: excellent at the mechanics, careful about checking its own work, and still in need of a senior reviewing the plan. What I'd tell my team after this test: Spend your time on the prompt and the plan, not the code. Every problem I found was visible in the plan before a single line ran. Write business rules down explicitly. Cancellations, duplicates, currency defaults. If it's tribal knowledge, the agent will guess. Trust the validation step, but read its report. It caught a data loss bug I might have missed myself. Tune afterwards. Treat its first version as correct, then ask it for performance improvements. Try it now if you have a sandbox workspace, a backlog of repetitive ingestion or migration jobs, and someone who can review PySpark. Wait for GA if you need it in production, your pipelines are full of undocumented business logic, or your capacity budget is tight. The bigger shift is in our role. Going from 11 hours to under an hour didn't remove me from the work. It moved my effort from typing code to writing clear requirements, setting guardrails and reviewing output. That is a skill worth building now, preview or not. Have you tried the Data Engineering Agent yet? Tell me in the comments what you threw at it, and where it surprised you. Sources FabCon Europe 2026 announcements (Kanerika) From Prompt to Finished Project: Introducing Project Osmos (FabCon Europe 2026 session) Microsoft Acquires Osmos (Redmond Magazine) Fabric January 2026 Feature Summary (Microsoft Fabric Blog)Privacy by Design in Fabric: Implementing Shift-Left Data Protection
Privacy by Design in Microsoft Fabric: Implementing "Shift-Left" Data Protection for Enterprise Analytics Abstract While data engineers and architects often prioritize pipeline performance, Medallion architectures, and complex transformations, regulatory compliance and data security are frequently treated as afterthoughts. In highly regulated sectors like banking and finance, failing to protect Personally Identifiable Information (PII) can lead to severe penalties under evolving frameworks such as GDPR, PIPEDA, and other international privacy laws. This article explores the concept of "Shift-Left" data protection within Microsoft Fabric—implementing data masking, hashing, and external tokenization to secure sensitive data early in the lifecycle—while distinguishing robust security controls across OneLake, SQL Analytics Endpoint, and Semantic Models. Introduction Modern data professionals excel at writing efficient PySpark scripts, optimizing SQL queries, and designing multi-layered Medallion Lakehouses. However, technical elegance means little if an analytics platform exposes sensitive operational data to unauthorized risks. With the rise of global data privacy regulations and increasing security threats, data governance can no longer be a secondary phase applied right before dashboard deployment; it must be embedded directly into the ingestion and storage strategy. A fundamental best practice in enterprise data architecture is the "Shift-Left" security pattern: masking, pseudonymizing, or hashing sensitive PII attributes at the earliest possible stage rather than relying solely on downstream presentation layers. Within Microsoft Fabric, data teams can execute this strategy across pipelines, PySpark notebooks, and Semantic Models. This article examines practical techniques for implementing early-stage data protection and dynamic security controls in Fabric, helping data engineers build scalable analytical solutions that support strict regulatory mandates. What is "Shift-Left" Data Protection? In traditional data engineering, security controls are frequently applied at the very end of the pipeline—usually at the presentation layer using Power BI dashboard permissions or SQL database views. This legacy approach creates a critical vulnerability: raw, unmasked Personally Identifiable Information (PII) rests in landing zones or intermediate data lake layers, accessible to anyone with direct storage access. "Shift-Left" Data Protection fundamentally changes this dynamic by pushing security, masking, and pseudonymization controls as far upstream as possible. This approach is highly valuable for organizations navigating cross-border data transfers and cloud migrations. Many international regulatory frameworks strongly recommend or mandate that when transferring sensitive data across borders, the information should be pseudonymized or encrypted to support data sovereignty and localization principles. Failing to comply with privacy frameworks carries severe financial and reputational consequences for enterprises worldwide. By applying cryptographic hashing, masking, or tokenization patterns early in the Medallion Architecture: Exposure is Minimized: Sensitive data (such as national IDs, emails, or account numbers) is restricted or neutralized before persisting in downstream analytical environments. Supports Compliance Initiatives: Organizations are better positioned to align with privacy principles established by global frameworks (like GDPR or PIPEDA) by adopting a Privacy by Design architecture. Governance is Streamlined: Instead of managing complex downstream permission rules across dozens of individual reports, core data security is enforced at the storage and engine layers. To determine whether an enterprise data pipeline truly adheres to a "Shift-Left" architecture, data engineering and compliance teams should verify key architectural control points, such as: [ ] Ingestion Gatekeeping: Are raw PII fields masked in transit before landing in the Bronze layer, or if landing raw in Bronze, is direct storage access strictly limited to service principals and administrators? [ ] Pseudonymization (Bronze to Silver): Are identifying attributes transformed into deterministic hashes (like salted SHA-256) before persisting into Silver Delta tables? [ ] Integration with Token Vaults: Are highly sensitive fields passed to external tokenization services during ingestion, ensuring the decryption vaults are isolated outside the analytics workspace? [ ] Identity and Role Verification: Are workspace roles configured correctly so that downstream consumers only access data via secured endpoints rather than having direct underlying OneLake file access? Demystifying the Security Layers in Microsoft Fabric Implementing a "Shift-Left" framework in Fabric requires understanding that security is not monolithic; it is applied across distinct layers. A robust architecture leverages data-layer controls, transformation techniques, and compute-engine security rules. 1. The Storage & Transformation Layer (Upstream) Source-Side Masking: Using Data Factory pipelines to mask data in transit before it ever writes to OneLake. Hashing as Pseudonymization: Using PySpark notebooks to apply SHA-256 hashing to fields like email addresses. Note: Hashing is pseudonymization, not true anonymization. To prevent dictionary or rainbow table attacks, data teams must implement secure "salting" alongside the hashing algorithm. External Tokenization Integration: Fabric pipelines can call external REST APIs (e.g., third-party tokenization services or Azure Key Vault) to replace sensitive strings with non-mathematical tokens during the Bronze-to-Silver transformation. OneLake Security (Data Access Roles): Fabric increasingly supports native OneLake security controls, allowing administrators to define data access roles directly on the OneLake folders and Delta tables, securing the files regardless of which compute engine accesses them. 2. The Engine & Consumption Layer (Downstream) Once data is curated, Fabric provides specific, engine-level security controls: SQL Analytics Endpoint (T-SQL Controls): Row-Level Security (RLS): Filters specific rows (e.g., restricting a regional manager to only see sales from their territory) using standard T-SQL CREATE SECURITY POLICY statements. Column-Level Security (CLS): Uses T-SQL GRANT SELECT statements to prevent specific roles from querying sensitive columns like Salary or SSN, while allowing them to query the rest of the table. Semantic Models (Direct Lake / Power BI Controls): Row-Level Security (RLS): Configured via DAX to filter data rows natively within the semantic model. Object-Level Security (OLS): Configured via Tabular Editor/XMLA to completely hide entire tables or columns from specific users in the semantic model, preventing report visuals from even discovering the fields. High-Level Architecture in Microsoft Fabric Implementing a "Shift-Left" data protection framework within Microsoft Fabric relies on combining two complementary security layers: In-Flight Data Sanitization (Storage Layer): Using Fabric PySpark Notebooks or Data Factory pipelines to hash, mask, or tokenize PII fields during ingestion or before writing into Silver Delta Tables in OneLake. Dynamic Access Controls (Engine Layer): Enforcing Row-Level Security (RLS) and Column-Level Security (CLS) on top of the SQL Analytics Endpoint or Direct Lake Semantic Models for role-based downstream consumption. To visualize how this defense-in-depth strategy comes together, the following reference architecture outlines how "Shift-Left" sanitization and dynamic query controls are orchestrated within Microsoft Fabric: Understanding the Architectural Diagram The architectural diagram above illustrates a simple example of an end-to-end Microsoft Fabric implementation designed around the "Shift-Left" paradigm. To read and interpret this architecture effectively, consider the flow of data through three core operational layers: 1. Data Sources & Boundary Controls (Left): Data flows from operational systems (On-Premises SQL Databases, REST APIs) toward Microsoft Fabric OneLake. Red Locks (Regulatory Blocking & Ingestion Masking): Represent early security controls applied at the connector level. Sensitive operational fields are masked or blocked before landing in cloud storage to prevent cross-border compliance violations. 2. Medallion Lakehouse & In-Flight Sanitization (Center): Bronze Layer (Raw Ingestion): Stores incoming raw landing data. Shift-Left Sanitization (PySpark Notebook): Before moving data into the Silver Layer, PySpark notebooks perform cryptographic transformations (SHA-256 hashing, masking, tokenization). Silver Layer (Sanitized & Standardized): Contains cleaned, enterprise-ready data where PII has been neutralized at the storage level within OneLake Delta files. 3. Consumption & Dynamic Engine Security (Right): Gold Layer (Aggregated & Enriched): Curated dimensional models built for business analytics. SQL Analytics Endpoint & Direct Lake Semantic Models: As queries hit the serving layer, secondary security rules—such as Column-Level Security (CLS) and Row-Level Security (RLS)—dynamically filter rows or restrict specific columns based on the user's Entra ID (Azure AD) identity, ensuring seamless role-based governance across Power BI reports and T-SQL queries. The Critical Role of Workspace Identity and Direct Storage Access The most common pitfall when designing security in Microsoft Fabric is a misunderstanding of Workspace Roles and Identity Modes. SQL RLS/CLS and Semantic Model RLS/OLS are engine-level controls. If a user is granted the Admin, Member, or Contributor role in a Fabric Workspace, they possess inherent direct storage access to the underlying OneLake files. Because they can access the raw Delta files directly via OneLake Explorer or Notebooks, they will completely bypass SQL RLS/CLS and Semantic Model security. To successfully enforce engine-level security (like SQL CLS or Direct Lake RLS), downstream consumers must be assigned the Viewer role in the workspace. This restricts their access to the defined endpoints (SQL Endpoint or Power BI App) where the security policies are evaluated under their Entra ID credentials, effectively blocking direct access to the unmasked storage files. Key Takeaways for Data Professionals Implementing "Shift-Left" data protection within Microsoft Fabric allows organizations to bridge the gap between high-performance analytics and governance requirements. By moving sanitization controls upstream and combining them with dynamic engine-level security, data teams can build a future-proof architecture that meets strict privacy mandates without sacrificing agility. To successfully apply these principles in your enterprise, keep the following core takeaways in mind: Security is an Architecture Feature, Not a Final Step: Embed data protection requirements into your initial pipeline designs rather than treating them as an afterthought. Protect at the Lowest Layer Possible: Hashing, tokenizing, and masking data early in the Medallion Architecture (Bronze-to-Silver transition) protects sensitive attributes across all downstream environments, including Notebooks, SQL queries, and Direct Lake Power BI reports. Align Controls to the Right Layer: Use OneLake security for raw storage, T-SQL CLS/RLS for the SQL Endpoint, and DAX OLS/RLS for Semantic Models. Audit Workspace Roles: Never rely on SQL or Semantic Model security for users who have Admin, Member, or Contributor access to the workspace. Join the Conversation! Designing enterprise-grade security in modern data platforms is an evolving practice, and different organizations adopt unique strategies depending on their regulatory environment. What additional techniques or governance frameworks do you implement in your organization to protect sensitive PII in Microsoft Fabric? Are you primarily leveraging SQL-based CLS, or shifting toward OneLake data access roles? Share your thoughts, experiences, or questions in the comments below!CDC vs Watermark: Choosing an Incremental Load Strategy in Microsoft Fabric
Why incremental loading matters If your source deletes rows or you need history, use CDC; if you only need inserts and updates and have a reliable "last modified" column, a watermark is simpler and works almost everywhere. That is the short answer. The rest of this post explains why. Reloading a whole table every night is fine at 50,000 rows. At 500 million it burns capacity units, hammers the source system, and stretches your refresh window. Incremental loading fixes this by moving only what changed since the last run. In Microsoft Fabric, the main tool for this is the Copy job in Data Factory. It supports both change detection approaches side by side, and can pick one per table. Knowing how each works helps you choose well and avoid silent data gaps. The watermark pattern A watermark load copies only rows whose "incremental column" value is higher than the highest value seen in the last successful run. The first run copies everything; each later run picks up where the last one stopped. The query behind it is simple: SELECT * FROM dbo.Orders WHERE ModifiedAt > @LastWatermark AND ModifiedAt <= @CurrentWatermark; In a classic pipeline you build this yourself: a Lookup for the old watermark, a Copy activity, and a step that saves the new value to a control table. Copy job removes that plumbing. It stores the watermark state for you, and a failed run resumes from the last successful one without data loss (Microsoft Learn). Copy job accepts these column types as a watermark: ROWVERSION – changes automatically on every insert or update; the most reliable choice on SQL Server-family sources. Datetime – columns such as ModifiedAt or LastUpdatedDatetime. Date – date-only columns; Copy job re-reads the last day to avoid gaps. String – values that can be read as datetimes. Integer – an increasing number. A column of another type can still work if you cast it to a supported type in the source query, as long as its values compare in order. File sources use the file's last-modified time as the watermark. The catch: a watermark only sees rows that still exist and whose column actually moved. Deleted rows are invisible, and rows with a NULL watermark are skipped on every incremental run after the first. Change data capture (CDC) CDC reads the source database's own change log instead of querying the table, so it captures inserts, updates and deletes exactly as they happened. You don't pick an incremental column; the database tells Fabric what changed. When a source table has CDC enabled, Copy job detects it, does an initial full load, and then replicates only changes on later runs (Microsoft Learn). Sources that support CDC in Copy job include Azure SQL Database, Azure SQL Managed Instance, on-premises SQL Server, Oracle, Snowflake, Google BigQuery, SAP Datasphere Outbound, and Fabric Lakehouse tables (via Delta Change Data Feed). You then choose how changes land in the destination: SCD Type 1 (Merge) – the default. The destination mirrors the current source: updates overwrite, deletes remove the row. SCD Type 2 – in preview. Copy job keeps every version, adding Valid_From, Valid_To and Is_Current columns. Deletes become soft deletes, so you keep a full audit trail with no custom code. Oracle sources don't support it yet. Copy job isn't the only CDC route in Fabric. Mirroring keeps a near real-time replica of an operational database in OneLake with almost no setup, and Eventstreams offers CDC connectors for streaming changes into Real-Time Intelligence. Microsoft's data movement decision guide compares the three. Side by side The deciding row is usually deletes: only CDC sees them. Consideration Watermark CDC Source prerequisite A column that always increases on insert or update CDC enabled on the source and supported by the connector Inserts Yes Yes Updates Only if the watermark column changes Yes Deletes No Yes History (SCD Type 2) No, unless you build it Built in (preview) Write methods Append or Merge Merge or SCD Type 2 Load on source Range scan on the watermark column each run Reads the change log; lighter on busy tables Setup effort Low; no DBA changes Needs CDC enabled, permissions, log retention Works with files Yes (last-modified time) No Source coverage Most database connectors A defined list of connectors Source: Incremental copy in Copy job, Microsoft Learn. Which one should you use? Three questions settle it for almost every table. If CDC is the right answer but you can't enable it on the source, fall back to a watermark and catch deletes with soft-delete flags or a periodic key comparison. Tips and pitfalls For watermark loads Prefer ROWVERSION over app timestamps. An application can forget to update ModifiedAt; the database never forgets a rowversion. Watch for NULLs. Rows inserted later with a NULL watermark are never picked up. Make the column NOT NULL with a default. Catch deletes another way. Use soft deletes (an IsDeleted flag that also bumps the watermark), or run a periodic key comparison against the source. Use Merge, not Append, for updates. Append writes a second copy of every updated row. For CDC loads Don't mix table types in one job. If a Copy job holds both CDC-enabled and non-CDC tables, it falls back to watermark for all of them. Split them into separate jobs. Size log retention. If the job is paused longer than the source keeps its change data, you will need a full reload. Know the current limits. Copy job captures net changes only (not every intermediate change) and supports only the default capture instance. For Lakehouse sources, enable Change Data Feed yourself; Copy job can't detect it (Microsoft Learn). For both Reset carefully. Resetting incremental state forces a full read on the next run but doesn't clear the destination. With Append, truncate the target first or you'll get duplicates. Plan for schema changes. New source columns aren't synced automatically, and a dropped column that's in your column mapping fails the run. Conclusion Watermark and CDC aren't rivals; most Fabric estates use both. Use a watermark for append-heavy fact tables, files, and sources you can't change. Use CDC wherever deletes or history matter, which is most dimension and master-data tables. Copy job makes the choice cheap to revisit: it detects CDC-enabled tables, handles state for you, and lets you reset per table. Start with the table that hurts most in your nightly full load, pick the method from the guide above, and measure the capacity you save. Sources Incremental copy in Copy job – Microsoft Learn Change data capture (CDC) in Copy job – Microsoft Learn Choose a data movement strategy – Microsoft Learn Copy job enhancements on incremental copy and CDC – Microsoft Fabric BlogFabCon & SQLCon 2026: The Ultimate Roundup of Announcements
While I wasn’t able to catch the action live in Barcelona last week for FabCon and SQLCon 2026, keeping tabs on the announcements has been a wild ride. Microsoft Fabric has officially cleared the 40,000-customer milestone and the event heavily emphasized Frontier Transformation – moving past standard AI productivity hacks and embedding autonomous agents directly into corporate workflows. Yet, among all the flashy Copilot integrations and agentic apps, one specific announcement completely stole the show for me: the introduction of the F0 capacity tier with on-demand billing. For anyone who has ever wrestled with budget approvals just to test out a new data platform, this is a game-changer. Let’s dive into why F0 is my favorite takeaway, alongside a full roundup of everything else that dropped in Barcelona. 1. My #1 Favorite: The F0 Capacity SKU & Pay-As-You-Go Billing Historically, dipping your toes into Microsoft Fabric meant committing upfront to a reserved compute tier (like an F2 or higher). If you just wanted to spin up a quick test environment, build a proof of concept, or play with OneLake shortcuts, you still had to navigate a monthly capacity bill. What’s new: Microsoft introduced F0, a zero-provisioned capacity SKU featuring true on-demand billing. Why it’s a massive deal: It acts as the “serverless” equivalent for Fabric. You can now evaluate features, run lightweight development tasks and test out workloads without locking into an upfront monthly reservation. Compute costs finally align strictly to what actually runs, shattering the barrier to entry for smaller teams and developer sandboxes. 2. Grounding Microsoft Copilot with Fabric IQ AI tools often hallucinate because they lack proper context. Microsoft is tackling this by wiring business context directly into Microsoft Copilot. What’s new: Fabric IQ is generally available as a shared intelligence bridge, connecting OneLake data, Power BI semantic models and operational metrics directly into Copilot Chat and Cowork – with zero additional AI token costs. Why it matters: Instead of guessing, your organization’s Copilot can answer business questions using the exact semantic data definitions your teams already trust and govern. Read more: Check out Arun Ulag’s keynote overview on the official Microsoft Azure Blog. 3. Power BI’s Evolution: Agentic App Creation Power BI is stepping past standard dashboards into operational application building. What’s new: You can use natural language inside Power BI Desktop to build, preview and publish purpose-built data applications straight from a trusted semantic model. These apps can handle inputs, write-back data and power workflows. Why it matters: Power BI Pro and Premium Per User (PPU) users will get access to Fabric Apps and Fabric Database capabilities (up to 1 GB per app) at no additional cost. Read more: Dive into Mohammad Ali’s deep dive on Power BI’s next chapter: The evolution of business intelligence. 4. Fast-Tracking Apps to Production with Fabric Apps & Rayfin Prototyping an app with AI is easy, but getting it to a secure production environment used to mean heavy lifting. Fabric Apps solves this. What’s new: Using the open-source Rayfin SDK and CLI, developers can build app backends and deploy them directly to Fabric with built-in TypeScript functions, a Secret Store, PostgreSQL support and private-by-default security. Why it matters: Apps plug straight into existing Lakehouses and Warehouses without data duplication. Read more: Catch Sachin Patney’s post, From prompt to production: What’s new in Fabric Apps. 5. Expanding the OneLake Ecosystem & IQ Sharing What’s new: Microsoft rolled out IQ sharing (in preview), allowing organizations to securely share governed data, markdown agent instructions and ontologies across different teams, partners and external ecosystems. Read more: Read Dipti Borkar’s breakdown on What’s new in Microsoft OneLake and its rapidly growing ecosystem. 6. Autonomous Data Engineering Agents What’s new: Driven by technology from the Osmos acquisition, Microsoft introduced a preview of the Fabric data engineering agent. Rather than just auto-completing code snippets, this agent handles complex, long-running engineering tasks like migrations, ETL pipeline creation and modernizations based on human guardrails. Read more: Check out Bogdan Crivat’s blog on Bringing governed analytics into the flow of work: Fabric Analytics at FabCon Europe 2026. 7. SQLCon: Scale, Intelligence, and Serverless Pauses Running parallel to FabCon, SQLCon delivered major updates for database administrators balancing traditional relational databases with modern demands. What’s new: A new Database Hub (Public Preview) serves as a single pane of glass to monitor health and security across SQL Server, Azure SQL and PostgreSQL. Plus, DiskANN vector indexes are now generally available to supercharge native semantic search using T-SQL. Read more: Read Shireesh Thota’s update on SQLCon Barcelona 2026: Advancing SQL with greater control, scale and intelligence. Final Thoughts: Looking Ahead Reflecting on everything announced in Barcelona, it’s clear that Microsoft is moving faster than ever to bridge the gap between heavy-duty data engineering and practical, AI-driven applications. While agentic workflows and Copilot enhancements grab the headlines, structural updates like the F0 tier are what truly make this ecosystem accessible to everyone from lone developers to enterprise architects. Honestly, I am still trying to digest all these announcements made and also can’t wait to try them soon! We are officially stepping into an era where building, scaling and operationalizing data isn’t just about managing tables – it’s about powering the intelligence layer that runs the business. Are you as thrilled about the F0 capacity tier as I am? What feature from FabCon are you testing first? Let me know in the comments below!
ADLS Gen2 and OneLake - Choosing the right storage approach
As Microsoft Fabric adoption grows, a practical architecture question follows: how does OneLake relate to Azure Data Lake Storage Gen2 (ADLS Gen2), and do we need to migrate? Both store analytics data in open formats, but they offer different ways to manage and consume it. I would start with the workload and the investments already in place, rather than with a decision to move storage. For an established lake, keeping ADLS Gen2 and exposing selected data to Fabric may be the simplest path. For a new Fabric-centric workload, storing data directly in OneLake may reduce operational work. The useful unit of decision is a layer or data product, not the entire estate. Start with the operating model What ADLS Gen2 is designed for ADLS Gen2 brings analytics capabilities to Azure Blob Storage through a hierarchical namespace. In this PaaS model, Azure runs the storage service while your team manages accounts, containers, networking, lifecycle policies and access through Azure role-based access control and POSIX-style ACLs. It can hold any file format, giving teams flexibility to choose how different engines process the data. That flexibility is valuable when ingestion pipelines and multiple analytics engines already share a lake. Azure Databricks, Azure Machine Learning and other compatible tools can work with the data where it is. The architectural responsibility sits with the team: bringing storage, processing, discovery and governance together into an operating platform. OneLake: Fabric-managed data lake Built on ADLS Gen2, OneLake is created automatically for a Fabric tenant. Microsoft manages the underlying storage infrastructure, while teams organize and secure data through workspaces and items such as Lakehouse’s. This SaaS model shifts the day-to-day focus from storage accounts to analytics data products. OneLake stores physical data and can also reference data elsewhere through shortcuts. The benefit becomes tangible in the way Fabric workloads work together. Spark notebooks prepare Lakehouse data, SQL analytics experiences expose supported tables, and Power BI can consume suitable tables through Direct Lake. A Lakehouse’s Files area can hold any file format, while its Tables area serves supported table formats and engines, including Delta and supported Iceberg tables through metadata virtualization. Shared storage simplifies integration, while each engine still has its own supported capabilities. The architectural relationship The storage hierarchy is straightforward: tenant, workspace, then data item. Within a Lakehouse, Tables and Files support different consumption needs. Domains group workspaces for organizational ownership and governance, rather than adding a folder level to the storage path. A simple example is: Scope Illustrative organization Fabric tenant OneLake namespace Workspace Reporting workspace containing a Sales Lakehouse Lakehouse Tables Local Delta tables, or a shortcut to supported Gold tables in ADLS Gen2 Lakehouse Files Files and folders, including supported external shortcuts What changes for analytics consumers Shortcuts: Bring existing data into reach A plain shortcut references an internal or external storage location without requiring an ingestion copy. If curated Delta tables already live in ADLS Gen2, a supported Lakehouse Tables shortcut can make them available to Fabric consumers. Files shortcuts expose files instead; making those files into queryable tables is a separate step. This provides an incremental adoption path, although downstream workloads may still cache or materialize data. The access design deserves as much attention as the data path. An ADLS shortcut uses the source connection’s configured credentials alongside Fabric permission checks. It does not automatically carry each reader’s source ACLs into Fabric. Before sharing it, I would check least privilege, network reachability and table compatibility with the identities and workloads that will use it. Shortcut transformations address a different need: preparing data for consumption. Supported CSV, Parquet and JSON transformations use Fabric Spark to copy and convert source content into managed Delta output and keep it synchronized. That can simplify preparation, but it introduces compute, output storage and synchronization to operate. Choose a plain shortcut when referencing the existing data is enough; choose a transformation when the output itself adds value. Direct Lake: Simplify the BI data path Direct Lake loads requested Delta-table columns into VertiPaq memory without requiring a full Import-mode refresh. Metadata refresh, also called framing, still determines the data version available to the model. The practical benefit is a simpler BI data path for suitable workloads. Capacity, table optimization, security and model design still determine performance and freshness. The underlying Delta data can remain in ADLS Gen2 when exposed through a supported OneLake Tables shortcut. That is the bridge to Direct Lake, rather than a direct connection from the semantic model to an ADLS path. I would assess the intended Direct Lake mode and security design, then test representative queries and freshness requirements before making performance commitments. Governance: Discovery and enforcement both matter Governance needs both business context and effective access control. ADLS Gen2 can use Microsoft Purview for discovery and governance alongside its storage permissions. In Fabric, the OneLake catalog brings discovery, metadata, lineage and endorsement into the analytics experience, while workspaces and items provide ownership context. Those capabilities help people find and understand data; the permissions on the actual access path determine who can use it. I would compare security and lifecycle requirements against the specific design, rather than assume one platform always offers more control. Fabric supports tenant- and workspace-level private links and customer-managed keys with supported-item constraints. OneLake also supports lifecycle policies and hot, cool and cold tiers. The decision turns on whether the required workload, connector, encryption and network configuration work together, not simply whether a feature appears on a checklist. AI readiness starts with usable data Both approaches can support AI. ADLS Gen2 is a supported Azure Machine Learning datastore, so an established lake may already be a sound foundation. Fabric adds integrated preparation and consumption options that can be useful without relocating that lake. My starting question would be which capability the team needs and where the data is already well governed. For example, AI-powered shortcut transformations can turn supported .txt files into Delta output containing summaries, translations, sentiment, redacted PII or extracted entities. A team could summarize eligible text sourced from ADLS and analyze the results in Fabric. This capability is in public preview, with regional and capacity requirements. Output quality and sensitive-data handling still need evaluation; automated redaction alone cannot establish compliance. Interoperability depends on the integration path Snowflake illustrates why the integration path matters. Fabric Mirroring can replicate Snowflake-managed table data into OneLake. For Iceberg tables in supported object storage, shortcuts and metadata virtualization provide another route; the Snowflake Iceberg mirroring path uses metadata mirroring and storage shortcuts. These are different architectural choices, with different implications for copies, freshness and ownership. Snowflake on Azure can also write Iceberg tables to OneLake in supported configurations. That integration requires public connectivity, does not support private-link workspaces, and has same-region prerequisites for writes. Azure Databricks can read and write OneLake through authenticated ABFS paths, with compute and authentication requirements. Open formats help, but the chosen engine and network design still need to fit the supported integration. Make the choice layer by layer An existing Databricks lake: should Bronze and Silver move? Consider a team with reliable Azure Databricks pipelines and Bronze, Silver and Gold layers in ADLS Gen2 that now wants Fabric reporting. My recommendation would be to keep the working Bronze and Silver layers unless there is a separate business or technical reason to move them. For Gold, there are two useful starting points. Keep physical Gold in ADLS Gen2 and expose supported Delta tables through Lakehouse Tables shortcuts. Gold becomes available in Fabric without physical move. I would start here when existing storage contracts, authoritative writers and non-Fabric consumers need to remain intact. Alternatively, write or materialize selected Gold data directly in OneLake when Fabric ownership or service-level requirements justify changing the pipeline. Databricks can support this pattern through authenticated ABFS access with the appropriate compute configuration. Moving selected Gold products can be a focused decision, independent of any future Bronze or Silver migration. For either pattern, agree on the authoritative writer, table compatibility, effective permissions, freshness targets, monitoring and total cost. A storage shortcut does not transfer Unity Catalog authorization, so test the security model with representative readers. These responsibilities matter more than whether the diagram shows one lake or two. Count the governance and semantic-model work I would also count the work above storage. A team may maintain security rules and semantic logic in Databricks, then recreate them for Fabric reporting. Repeating policies, metrics, relationships and business definitions creates maintenance and testing work, with a risk of drift whenever requirements change. Reducing that duplication can be a stronger reason to adopt a shared consumption layer than saving a data copy. A useful pattern is to use Databricks for data preparation, make governed Gold data available through OneLake, and use Fabric as the shared consumption layer. OneLake security can centralize data-access policies across supported Fabric engines and access paths. Separately, a deliberately shared Power BI semantic model lets reports and compatible clients reuse measures, relationships and business definitions instead of rebuilding them. These are two distinct opportunities to reduce downstream work: common access control and an explicitly reusable semantic layer, not benefits that storage alone creates. That semantic model reuse does not require moving your Gold layer: supported shortcuts can expose ADLS data, and shared Power BI models can also use other supported sources. OneLake does not automatically unify Unity Catalog policies nor give every Databricks or Spark workload the same semantic layer. Where Unity Catalog remains authoritative, I would evaluate supported integrations and ask: where are policies and metrics defined, which engine enforces them under which identity, which consumers reuse the model, and who maintains and tests necessary differences? A practical guide Your situation Starting point What to validate Successful multi-engine lake; established account/API dependencies Keep ADLS Gen2 as an authoritative store Existing contracts, source security and the specific Fabric access path New Fabric-centric analytics and shared Fabric items Use native OneLake storage Ownership, supported workloads, capacity, security and lifecycle requirements Mixed estate; selective Fabric adoption or duplicated governance and model logic Combine ADLS Gen2 and OneLake; assess a shared consumption layer See the mixed-estate checklist below. A required security or storage feature determines the design Compare the exact supported configurations Connector and workload limits, private networking, CMK item coverage, retention and total cost For a mixed estate, validate: Data access: shortcut compatibility versus materialization. Write ownership: the authoritative writer. Operations: freshness targets and monitoring. Security: effective identities and policy enforcement. Semantics: reuse of a shared semantic model. Governance ownership: who maintains and tests necessary cross-platform differences. Coexistence as a practical starting point Use ADLS Gen2 where its customer-managed storage model and existing integrations serve the workload well. Use native OneLake where Fabric’s managed model and shared analytics experiences provide a clear benefit. For an existing estate adopting Fabric, I would usually start by connecting the two and move data only where the benefit is clear. Coexistence can be a transition or a lasting design; consolidation should earn its place through workload requirements and measurable value.Connect Fabric Data Agents to Copilot Studio with MCP (+ Assistants API Migration Fix)
Business users want answers where they already work, in Teams and Microsoft 365 Copilot, and those answers need to come from trusted, governed data. Fabric data agents in Copilot Studio do exactly that, and the integration is now generally available. There is also a deadline you may already have passed. If you connected to Fabric data agents through the OpenAI Assistants API, your integration has likely stopped working. This guide covers both the setup and the fix. What's new Fabric data agents now connect to Copilot Studio through a new tool-based experience: you select Add a tool, search for Fabric, and choose Fabric IQ Data MCP. Your Copilot Studio agent then calls the Fabric data agent like any other tool. Microsoft Community Three points matter for architects: Data stays governed. The Fabric data agent keeps running in Fabric, respects permissions on the underlying data sources, and returns answers grounded in governed enterprise data. Microsoft Community The orchestrator decides. The orchestrator chooses when to use enterprise data and when to use its other knowledge sources and tools. Microsoft Community You can combine agents. You can pair several Fabric data agents with other business systems to build richer solutions. A sales agent could query a sales data agent, a finance data agent and your CRM connector in a single conversation. Microsoft Community Behind this, Copilot Studio also introduced a redesigned authoring surface and runtime powered by the GitHub Copilot harness. Microsoft Community Why MCP matters Model Context Protocol (MCP) is becoming the standard way agents connect to tools and data. By exposing Fabric data agents through MCP, Microsoft makes them a reusable building block instead of a one-off integration. The same move is happening elsewhere: Fabric data agents in Microsoft Foundry are now easier to connect and monitor, and that integration is moving to MCP too. Microsoft Community You build a data agent once in Fabric, and Copilot Studio, Foundry and Microsoft 365 Copilot can all use it. Prerequisites Before you start, check that you have: A Microsoft Fabric workspace on Fabric capacity Data ready to use: a lakehouse, warehouse, semantic model or similar source with clean, well-named tables Permission to create items in the Fabric workspace Access to Copilot Studio with permission to build and publish agents Microsoft 365 Copilot or Teams, depending on where you want to publish Tip: the quality of your data agent depends on the quality of your data. Clear table and column names and descriptions make a big difference to answer accuracy. Step 1: Build and test your Fabric data agent In your Fabric workspace, create a new data agent item, then: Add data sources. Connect the lakehouse, warehouse or semantic model the agent should answer from. Start narrow, since one domain per agent works best. Write agent instructions. Explain the business context, key terms and rules. For example: "Revenue means net revenue after returns. Fiscal year starts in April. Always exclude internal test accounts." Add example questions and queries. Give sample questions with the correct query logic, so the agent learns your patterns. Test thoroughly. Ask the questions your business users actually ask and compare the answers with your certified reports. Publish the agent once the answers are reliable. Step 2: Add the data agent to Copilot Studio This is the new MCP-based flow. Create or open an agent in Copilot Studio, go to Tools, select Add a tool, search for Fabric, and add Fabric IQ Data MCP. Microsoft Community Then: Select the Fabric data agent you published in Step 1. Give the tool a clear description, such as "Answers questions about sales, revenue and pipeline from governed Fabric data." The orchestrator reads this to decide when to call the tool, so be specific. In your Copilot Studio agent's instructions, tell it when to use the tool, for example: "For any question about sales numbers or targets, use the Sales Data tool." Step 3: Test in Copilot Studio Test your agent in Preview before publishing. Check that: Microsoft Community Data questions trigger the Fabric tool, and general questions don't Answers match the results from Fabric directly A user without access to certain data gets no answer from it, since the data agent respects permissions on the underlying sources Follow-up questions ("and what about last quarter?") work as expected Step 4: Publish to Teams or Microsoft 365 Copilot When you're happy with the results, publish to Microsoft Teams or Microsoft 365 Copilot. Business users can now ask data questions in the tools they use every day, without opening a report. Microsoft Community Roll out to a small pilot group first, collect feedback, and improve the data agent's instructions before rolling out widely. The Assistants API retirement: what broke and how to fix it This part matters if you built programmatic integrations. What happened: The OpenAI Assistants API, which powered the orchestration layer of the Fabric data agent, was scheduled to be shut down by OpenAI on August 26, 2026. After that date, direct calls to the Assistants API stop working. Microsoft Community Who is affected: anyone who connects to a Fabric data agent programmatically through the Assistants API needs to migrate to the data agent MCP endpoint. Typical examples are custom apps, scripts, or third-party tools calling the data agent directly. Microsoft Community Who is mostly fine: SDK and Fabric portal users need little or no action, because Microsoft migrates those experiences internally, although conversation history may reset once. Microsoft Community What stays the same: existing agent data sources, instructions and tools remain unchanged. You don't need to rebuild your agents, only change how you connect to them. Microsoft Community Migration checklist Find your integrations. Search your code and configs for Assistants API calls that target Fabric data agents. Check custom apps, automation scripts, Azure Functions and any third-party tools. Check for failures since late August. Errors or silent failures after August 26 are a strong sign an integration is affected. Move to the MCP endpoint. Update each integration to connect through the data agent's MCP endpoint instead. Follow Microsoft's guide, "Prepare your Fabric Data Agent integrations for Assistants API retirement," for the current details. Consider Copilot Studio instead of custom code. If a custom app only exists to put a chat interface on a data agent, the Copilot Studio route above may replace it with less code to maintain. Tell users about history resets. If you use the portal or SDK, let users know earlier conversations may disappear once. Retest. Run your standard question set again after migrating to confirm the answers haven't changed. Best practices for production One domain per data agent. Separate agents for sales, finance and operations are easier to test, govern and improve than one agent that tries to do everything. Write tool descriptions carefully. In Copilot Studio, the tool description is how the orchestrator decides what to call. Vague descriptions lead to wrong tool choices. Treat instructions like code. Version them, review changes, and retest after every edit. Keep a test question set. Keep 20 to 30 real questions with expected answers, and run them after any change to data, instructions or model. Watch permissions. Answers respect underlying permissions, so a mistake in data security becomes a mistake in agent answers. Review access regularly. Start small and grow. Pilot with one team, measure how often answers are right, then expand. The bottom line Connecting Fabric data agents to Copilot Studio through MCP turns your governed Fabric data into a reusable tool that any agent can use, and it puts trusted answers directly into Teams and Microsoft 365 Copilot. The setup takes minutes once your data agent is solid. If you built integrations on the Assistants API, make migration your first job this week. Your agents, data sources and instructions don't need to change, only the connection. After that, you're on the platform Microsoft is building its agent strategy on.
