Fabric platform Community Blog articles

Building an app on governed data with Fabric Apps

Haniftadjik — Thu, 12 Nov 2026 16:41:40 GMT

A hands-on proof of concept with Fabric Apps on governed Fabric data: dashboards, approvals and questions to data agents in one app. The design question is whose access each part uses: dashboards run as the user, writeback relies on rules in code, and agent calls run as the app owner.

After the Zombie Outbreak: Making ArcGIS GeoAnalytics for Microsoft Fabric Easier to Try

Tonyfra91 — Thu, 05 Nov 2026 14:45:08 GMT

Sometimes the best technical scenarios start with a problem you hope never becomes real.

In Patient Zero in Redlands: Tracking a Zombie Outbreak with ArcGIS for Microsoft Fabric, Sean McGinnis from Esri uses a fictional zombie outbreak to demonstrate something very real: how location, movement, population, and time can become part of an analytical problem inside Microsoft Fabric.

Read Sean's original ArcGIS blog: Patient Zero in Redlands

At the center of the scenario is ArcGIS GeoAnalytics for Microsoft Fabric. Running within the Fabric Spark environment, GeoAnalytics brings spatial functions and analysis capabilities to data processing in Fabric, allowing location to become part of the analytical workflow instead of a separate step performed somewhere else.

Sean's scenario makes that tangible. H3 cells, population enrichment, neighborhood relationships, and time-stepped results turn a fictional outbreak into a spatial analytics problem that can be explored, computed, and ultimately communicated through maps and other experiences.

The zombies are fictional. The spatial pattern isn't.

The same approach can be applied to questions involving public safety, disease surveillance, infrastructure disruption, supply chains, customer activity, environmental events, and many other problems where where something happens is just as important as what happened.

Sean highlights several practical use cases that show how the fictional scenario translates to real-world challenges:

Use case	Patient zero becomes	Spread becomes	Vulnerability becomes
Retail product launch	Flagship store opening	Word of mouth and awareness moving between neighborhoods	Income, age, and household size of likely buyers
Utility outage planning	Failed substation	Load shifting to neighboring circuits	Medical-device customers, older residents, no vehicle
Public health	First confirmed case	Contact and commuting between areas	The full CDC SVI, which was built for this
Insurance and catastrophe risk	Wildfire ignition or flood breach	Fire front or water spreading downhill	Housing type, mobile homes, insurance coverage
Emergency management	Chemical spill or plume source	Wind-driven dispersion	People who can’t evacuate on their own
Supply chain disruption	Closed port or warehouse	Delays rippling to stores served by it	Store dependence on that node

From scenario to implementation

After working through Sean's scenario, I became interested in a slightly different question:

How easy can we make it for someone else to get to the point where they can start experimenting with ArcGIS GeoAnalytics for Microsoft Fabric?

Not simply reading about the capability. Not starting with a finished architecture diagram. Actually getting an environment ready, running a spatial workload, and seeing whether the pieces work together.

https://www.esri.com/arcgis-blog/app/uploads/2026/10/ZombieArchitecture.png

That became the goal of my Fabric GeoAnalytics deployment experience.

Explore the Fabric GeoAnalytics deployment repository

The repository takes the spatial pattern demonstrated in the zombie scenario and focuses on the deployment and validation experience around it. The goal is to reduce some of the setup friction between "I'd like to experiment with GeoAnalytics" and "I have a Fabric environment where I can begin testing spatial workloads."

Post Zombie: A Fast Pass to GeoAnalytics

Once the outbreak is under control, it's time to rebuild.

The deployment experience starts with an Azure foundation and guides you through the Fabric-specific setup needed to experiment with GeoAnalytics: a Microsoft Fabric workspace, Lakehouse, GeoAnalytics-compatible environment, parameterized notebook, and the configuration needed to validate a spatial workload.

Prepare → Deploy Azure → Finish Fabric → Configure → Prove → Extend

For those who want the shortest route through the experiment, the repository includes a Copilot Fast Pass, a guided path through prerequisites, Azure deployment, Fabric setup, configuration, and workload validation.

The important word is prove. Rather than assuming a successful deployment means GeoAnalytics is working, the repository runs a small synthetic spatial workload, writes the results to Delta tables, reads them back, and verifies that the expected data reached OneLake.

The repository also maintains a clear boundary with Sean's original work. His notebook is retrieved separately and attributed upstream, while this deployment experience focuses on making the environment setup and validation more repeatable.

Calling a few brave souls

This is where I could use some help.

This repository is a preview and engineering experiment, not a production-certified accelerator. I've built automated checks around deployment, configuration, notebook adaptation, outputs, and the Azure-to-Fabric handoff, but proving the pattern means running it across different tenants, capacities, configurations, and users.

So I'm looking for a few brave souls willing to take the Fast Pass, deploy it, test it, break it, and tell me what happened.

Expect rough edges. There may be bugs, Azure and Fabric resources can incur charges, and different permissions, capacities, tenant configurations, or Esri settings may uncover assumptions that need to be corrected. That's part of the experiment.

Start here:
Fabric GeoAnalytics deployment experience

The repository includes prerequisites, the guided deployment path, validation steps, architecture notes, and an acceptance checklist.

If you try it, issues and feedback are welcome.

Where this goes next

What I like most about Sean's zombie scenario is that the story doesn't really end with zombies. It demonstrates a broader pattern: enterprise and geospatial data can be analyzed together, spatial relationships can become part of large-scale analytics, and the resulting insight can move into maps and experiences that people can actually use.

The next step is making those capabilities easier for architects, developers, data engineers, and GIS professionals to explore themselves.

That's what this deployment experiment is about.

Sean showed us how to survive the outbreak. Now let's see how repeatable we can make the rebuild.

Implement Fabric capacity Pause and Resume operation using Azure Automation Runbook - Part 2

pankaja_ms — Mon, 02 Nov 2026 16:00:00 GMT

In the previous post (found here), I described first 4 steps in the process to implement an Azure automation job to schedule Pause and Refresh functions for Azure capacities. Pausing a Fabric capacity takes the capacity offline which means you do not may for the compute costs. You would still incur costs for storage in OneLake during this time, but given compute costs are typically higher, this process allows you to cost optimize Fabric operational costs for non-production instances.

Following the first part of the process described in Part 1 of this blog series, in this post I will describe the remaining steps.

Step 5 - Add Azure Automation managed identity to new custom RBAC role

Run the following in Cloud Shell after replacing the placeholders with values from Step 1. The assignment scope is the Fabric capacity resource ID, not the entire resource group or subscription.

Powershell

$subscriptionId          = "<SUBSCRIPTION_ID>"
$fabricResourceGroup     = "<FABRIC_RESOURCE_GROUP>"
$capacityName            = "<FABRIC_CAPACITY_NAME>"
$automationResourceGroup = "<AUTOMATION_RESOURCE_GROUP>"
$automationAccountName   = "<AUTOMATION_ACCOUNT_NAME>"
$roleName                = "Fabric Capacity Pause Resume Operator"

Set-AzContext -SubscriptionId $subscriptionId

$principalId = (Get-AzAutomationAccount `
  -ResourceGroupName $automationResourceGroup `
  -Name $automationAccountName).Identity.PrincipalId

$capacityScope = "/subscriptions/$subscriptionId/resourceGroups/" +
  "$fabricResourceGroup/providers/Microsoft.Fabric/capacities/$capacityName"

$role = Get-AzRoleDefinition -Name $roleName
New-AzRoleAssignment ObjectId $principalId -RoleDefinitionId $role.Id -Scope $capacityScope

Step 6 - Create a parameterized Pause/Resume runbook

In the Automation account, select Runbooks > Create a runbook.
Name it Manage_Fabric_Capacity_State, choose PowerShell, and select the PowerShell 7.4 runtime environment created earlier.
Paste the script below in the Cmdlet window. During creation in my environment, I only parameterized the Resume/Start value. I hardcoded other values, but you can parameterize other values too if desirable.

Powershell

param(
  [Parameter(Mandatory = $true)]
  [ValidateSet("Pause", "Resume")]
  [string] $Action,
  [Parameter(Mandatory = $true)]
  [string] $SubscriptionId,
  [Parameter(Mandatory = $true)]
  [string] $ResourceGroupName,
  [Parameter(Mandatory = $true)]
  [string] $CapacityName
)

$ErrorActionPreference = "Stop"
$null = Disable-AzContextAutosave -Scope Process
try {
  $connection = (Connect-AzAccount -Identity).Context
  $null = Set-AzContext -SubscriptionId $SubscriptionId -DefaultProfile $connection
  Write-Output "Action: $Action"
  Write-Output "Capacity: $CapacityName"

  if ($Action -eq "Pause") {
    $result = Suspend-AzFabricCapacity -SubscriptionId $SubscriptionId `
      -ResourceGroupName $ResourceGroupName -CapacityName $CapacityName `
      -PassThru -Confirm:$false
  }
  else {
    $result = Resume-AzFabricCapacity -SubscriptionId $SubscriptionId `
      -ResourceGroupName $ResourceGroupName -CapacityName $CapacityName `
      -PassThru -Confirm:$false
  }

  Write-Output "Completed: $result"
}

catch {
  Write-Error "Fabric capacity $Action failed: $($_.Exception.Message)"
  throw
}

Disable-AzContextAutosave prevents the job from inheriting a saved Azure context. Connect-AzAccount -Identity authenticates with the Automation account identity. The Az.Fabric cmdlets then submit the documented suspend or resume operation by subscription, resource group, and capacity name.

Step 7 - Test, verify and publish the Runbook

Open Test pane from the Runbook editor.
First test Resume against a capacity that is intentionally paused, or Pause against a confirmed non-production or unused capacity. Enter all four parameters exactly.
Start the test and review the job streams. A successful run should show the requested action, capacity name, and a completed result.
Confirm the capacity state in Azure portal. For a pause test, also confirm that the expected Fabric content is unavailable; for resume, confirm that assigned content becomes available again.
Return to the editor and select Publish. Once published, the new runbook can be scheduled to activate automatically.

Step 8 - Create Resume and Pause schedules

Using Azure Automation schedules, a runbook can be initiated automatically. A runbook can have multiple schedules, and each link can supply different parameter values. Schedules support one-time or recurring patterns and daylight-saving-time-aware time zones.

Resume schedule

If the Runbook from previous task is not published, follow Step 5 in previous task to publish it before proceeding to the next step.
Open Runbook Manage_Fabric_Capacity_State > Schedules > Add a schedule.
Create or link a recurring schedule, for example Fabric-Resume-Weekdays-0700. Choose the intended time zone and weekdays.
Configure run settings with Action = Resume plus the subscription ID, Fabric resource group, and capacity name.

Pause schedule

Add a second recurring schedule, for example Fabric-Pause-Weekdays-1900.
Use the same resource parameters, but set Action = Pause.
Verify both schedule links are enabled and show the correct next run time.

Step 9 - Validate operations and troubleshoot

After the first scheduled execution, open Jobs for the runbook and inspect the job status and output. Then verify the capacity state in Azure portal. Azure Automation runs the latest published runbook version with the parameter values stored on the schedule link.

Following table lists some of the common failure and underlying reason why you may be seeing that class of failures. This is not a full list, rather a high-level guide on how to proceed with troubleshooting in case there are errors.

Symptom	Check
AuthorizationFailed	Confirm the role assignment uses the Automation account principal ID, the custom role contains all four actions, and the assignment scope exactly matches the capacity resource ID.
Cmdlet not recognized	Confirm the runbook uses the intended PowerShell 7.4 runtime and that Az.Fabric is present in that runtime environment.
Wrong subscription or capacity	Check the schedule parameters and the explicit Set-AzContext subscription ID.
Test works, schedule fails	Confirm the runbook is published and every mandatory parameter is saved on each schedule link.
Job succeeds, state is unexpected	Review job output, then inspect the target capacity resource in Azure portal to confirm the resource name and current state

Conclusion

Implementing Pause/Resume automation for non-production Fabric capacities can provide an easy cost optimization with little complexity. This pattern separates authentication, authorization, execution, and scheduling. The managed identity removes stored credentials; the custom role limits the identity to the four Fabric capacity actions documented by Microsoft; the resource-level assignment limits reach to the intended capacity; and two schedule links reuse one tested runbook. The result is a repeatable pause/resume implementation that is easier to audit than a broad built-in role or embedded service-principal secret.

References

Pause and Resume Fabric capacity

Using System Managed Identity for an Azure Automation account

Azure Custom RBAC Roles

Tutorial: Create an Azure custom RBAC role using Powershell

Manage schedule in Azure Automation

Part 1 of the blog can be found here: Implement Fabric capacity Pause and Resume operation using Azure Automation Runbook

Keywords: Cost Optimization, Capacity, Pause, Resume

Implement Fabric capacity Pause and Resume operation using Azure Automation Runbook - Part 1

pankaja_ms — Thu, 29 Oct 2026 14:42:32 GMT

Introduction

Microsoft Fabric allows pausing an F SKU capacity when it is not needed and resume it when work should continue. While pausing Fabric capacities will reduce operational cost, it will make Fabric content within these capacities unavailable. When a paused Fabric capacity is resumed, billing resumes and assigned content becomes available again. Pausing Fabric capacity can provide significant operational discounting as the capacity compute charges stop once the capacity is Paused. Hence this approach could be a significant price optimization technique if the capacity does not have to be online 24 hours a day.

This blog will help you implement automated control plane operation with Azure Automation. A system-assigned managed identity authenticates the runbook, a custom Azure role grants only the Fabric capacity suspend and resume operations, and two schedules pass either Pause or Resume to the same runbook.

What will you need to build and enable as part of this process?

Create a custom RBAC role that permits PAUSE and RESUME operations on Fabric capacities within a subscription
Configure an Azure Automation runbook to undertake PAUSE and RESUME operations through a scheduled operation

Following pre-requisites are needed prior to commencing the process described in this Blog.

An existing Microsoft Fabric F SKU capacity. Pause and resume apply to the Azure Fabric capacity resource (not tenant).
Permission to create a custom role and role assignment. Microsoft documents User Access Administrator or equivalent role-definition and role-assignment permissions for these tasks.
An Azure Automation account, or permission to create one. This guide uses its system-assigned managed identity. ^[2] ^[11]
PowerShell in Azure Cloud Shell for the one-time RBAC setup

Step 1 - Record deployment values

Before commencing the process, its important to document the necessary values as they will be needed multiple times during the process.

Value	Example	Where to find it
Subscription ID	00000000-0000-0000-0000-000000000000	Subscriptions > target subscription
Fabric Resource Group	rg-fabric-prod	Fabric Capacity > Overview
Fabric Capacity Name	fab-prod-uswest2	Microsoft Fabric -> Capacity Resource
Azure Automation Resource Group	rg-automation-prod	Azure Automation Account > Resource Group Name
Automation Account Name	aa-fabric-operations	Automation Account -> Account Name
Time zone	Australia/Melbourne	Azure Portal

Step 2 - Create Azure automation account and enable account identity

If an Azure Automation account does not already exist, create one from Automation Accounts > Create and complete the subscription, resource group, name, region, and review steps documented in the Azure Automation quickstart.

Open the Automation account in the Azure portal.
Under Account Settings, select Identity.
On the System assigned tab, set Status to On, select Save, and confirm the change.
Copy the displayed Object (principal) ID for validation. The PowerShell assignment step below also retrieves it directly.

The system-assigned identity is registered in Microsoft Entra ID and is represented by an object ID. The runbook later uses Connect-AzAccount -Identity, so no stored username, password, certificate, or Automation Run As account is required.

Step 3 - Prepare Azure Automation Powershell runtime

From the Automation account Overview, switch to the Runtime Environment experience if it is not already enabled.
Under Process Automation, select Runtime Environments > Create.
Choose PowerShell and runtime version 7.4. Give the environment a descriptive name such as ps74-fabric-operations.
On Packages, retain the default Az PowerShell package and add Az.Fabric from the gallery. Verify that Az.Accounts is available because the script uses Connect-AzAccount, Disable-AzContextAutosave, and Set-AzContext.
Create the runtime environment and wait until its packages are available.

Microsoft documents that PowerShell runtime environments include the Az package by default and can accept additional packages from the gallery. In the older Automation experience, modules are managed under Shared Resources > Modules instead.

Step 4 - Create a new custom RBAC role

Following four Azure RBAC actions are required for Fabric capacity pause and resume operations. Existing built-in privileged (pre-defined) roles do not contain these permissions, hence a custom role is needed.

RBAC Action	Purpose
Microsoft.Fabric/capacities/read	Read the target Fabric capacity
Microsoft.Fabric/capacities/write	Write to the target Fabric capacity
Microsoft.Fabric/capacities/suspend/action	Initiate a pause/suspend action for Fabric capacity
Microsoft.Fabric/capacities/resume/action	Initiate a resume action for Fabric capacity

There are 2 ways to create the custom role -

Using a definition file (in JSON format)
Defining the permissions inline as part of a command issued within a Powershell session.

As part of this blog, I'll show you both ways to create the role. Following examples shows you a functional JSON file that could be leveraged to create the custom role through command line or Azure portal. In the text below, replace SUBSCRIPTION_ID placeholder with actual subscription ID. The role definition is available at the subscription level, while the role assignment in the next step is narrowed to one capacity resource. If its desirable to create this role at resource group level, that can be done by updating the assignableScopes node to the resource group.

JSON

{            
   "properties": {       
      "roleName": "Fabric capacity Pause Operator",       
      "description": "",       
      "assignableScopes": [           
          "/subscriptions/<SUBSCRIPTION_ID>"        ],       
      "permissions": [           
      {               
          "actions": [                   
              "Microsoft.Fabric/capacities/read",                   
              "Microsoft.Fabric/capacities/write",                   
              "Microsoft.Fabric/capacities/suspend/action",                   
              "Microsoft.Fabric/capacities/resume/action"               
          ],               
          "notActions": [],               
          "dataActions": [],               
          "notDataActions": []           
      }]   
   }
}

Now we will use the second method to configure new RBAC custom role. In Azure Cloud Shell (PowerShell), create and inspect the role.

Open a new Cloud Shell Powershell session
Replace the SUBSCRIPTION_ID below with the SubscriptionID in the following commandlet. Copy and paste each command into the Cloud shell session one at a time and execute.

Azure custom roles use Actions for control-plane permissions and AssignableScopes to specify where the role definition is available. Microsoft recommends listing actions explicitly rather than using wildcards

Powershell

Set-AzContext -SubscriptionId "<SUBSCRIPTION_ID>"

$role = New-Object -TypeName Microsoft.Azure.Commands.Resources.Models.Authorization.PSRoleDefinition
$role.Name = 'Fabric Capacity Pause Resume Operator'
$role.Description = 'Pause and Resume Fabric Capacities”
$role.IsCustom = $true
$role.AssignableScopes = @("/subscriptions/<SUBSCRIPTION_ID>")
$permission = New-Object -TypeName Microsoft.Azure.Commands.Resources.Models.Authorization.PSPermission
$permission.Actions = @(
    "Microsoft.Fabric/capacities/read"
    "Microsoft.Fabric/capacities/write"
    "Microsoft.Fabric/capacities/suspend/action"
    "Microsoft.Fabric/capacities/resume/action"
)

$role.Permissions = @($permission)

New-AzRoleDefinition -Role $role

Cloud Shell execution of the script

This blog series continues in the next post. Part 2 can be found here: Implement Fabric capacity Pause and Resume operation using Azure Automation Runbook - Part 2

Copilot Studio, Foundry, Fabric Data Agent or Operations Agent? Which one to pick

FarhanJeelani — Tue, 27 Oct 2026 14:30:58 GMT

Most confusion comes from the word "agent" being used for all four. The fastest way to choose is to ask two questions: who builds it (maker or developer) and what triggers it (a person asking, or data changing).

Meet the four

Copilot Studio is Microsoft's low-code agent builder on Power Platform. You describe the agent in plain language, add knowledge (SharePoint, Dataverse, websites), plug in 1,000+ connectors and Power Automate flows, then publish to Teams, Microsoft 365 Copilot or a website. It is also the easiest place to orchestrate other agents, including Foundry and Fabric agents.

Microsoft Foundry (formerly Azure AI Foundry) is the pro-code platform for engineers. You pick from a large model catalog, write orchestration in code, run evaluations, trace every call and deploy behind private networking. Use it when the agent is a product component, not just a chat window.

Fabric data agent is a read-only, conversational analyst over your Fabric data. It turns a question into SQL, DAX or KQL against up to five sources (lakehouse, warehouse, Power BI semantic model, KQL database, ontology) and respects each user's permissions. It is generally available.

Fabric operations agent lives in Real-Time Intelligence. You give it an Eventhouse, a business goal and allowed actions. It writes its own monitoring rules, wakes up when a condition hits, reasons over the data, and messages people in Teams or runs a Power Automate action. It reached general availability in mid-2026.

Which one should I pick?

Ask the questions in order and stop at the first "yes". A "yes" does not rule the others out: a data agent or Foundry agent is often published through Copilot Studio.

Side by side

	Copilot Studio	Foundry	Fabric data agent	Fabric operations agent
Core job	Converse and automate	Build custom AI apps and agents	Answer questions over data	Monitor live data and act
Triggered by	A person (or event/flow)	Your code	A person or another agent	A data condition
Builder	Maker, IT, business	AI engineer, developer	Data/BI team	Ops and RTI team
Skill level	Low-code	Pro-code	Low-code + good data modelling	Low-code, chat-based setup
Data it reaches	Connectors, SharePoint, Dataverse, other agents	Anything you wire up	Lakehouse, warehouse, semantic model, KQL, ontology	Eventhouse (KQL) streams
Writes or acts?	Yes, via flows and tools	Yes, whatever you code	No, read-only	Yes, Teams messages and Power Automate actions
Where users meet it	Teams, M365 Copilot, web, other channels	Your app, API, Teams	Fabric, Copilot Studio, Foundry, M365 Copilot, MCP	Teams chat or channel
Billing	Copilot Credits	Azure consumption	Fabric capacity (CU)	Fabric capacity (CU)
Pick it when	You need speed and M365 reach	You need model choice, evals, isolation	People

Real-world scenarios

These are the patterns that come up most often in the community. Each names the tool that fits and why.

HR policy helper in Teams → Copilot Studio. A 3,000-person company wants staff to ask "How many leave days do I have?" and "Book Thursday off." Knowledge sits in SharePoint, leave balances in an HR system with a connector. A maker builds it in days and publishes to Teams. No developer needed.
Insurance claims triage inside a customer app → Foundry. An insurer reads uploaded photos and PDFs, extracts fields, scores fraud risk and returns a decision to its claims system through an API. It needs a vision model, evaluation datasets, tracing and a private network. That is a product feature, so it belongs in Foundry.
Sales leaders asking about pipeline → Fabric data agent. "Which regions missed Q3 target, and by how much?" The data agent sits on the sales semantic model, writes the DAX, and only shows rows each person may see. Publish it into Microsoft 365 Copilot or Copilot Studio so leaders never open Fabric.
Cold-chain monitoring for a food distributor → Fabric operations agent. Truck temperature sensors stream into an Eventhouse. The goal: "Keep chilled loads between 2 and 5 °C." When a truck drifts, the agent posts in the fleet Teams channel with the truck, route and trend, and can trigger a Power Automate flow to reroute or open a ticket.
Retail "what should I do?" assistant → all four together. Microsoft's Zava reference demo shows this: an operations agent spots a stock risk, a Fabric data agent pulls the numbers, Foundry agents for sales and product reason over them, and Copilot Studio is the front door in Teams where the manager approves a reallocation.

Better together

The strongest designs use more than one. Copilot Studio meets people where they work, Foundry adds custom reasoning, and the two Fabric agents keep answers and actions grounded in governed data.

The operations agent is the only one that starts on its own: it reads live events and reaches people in Teams without anyone asking first.

Status as of October 2026, and gotchas

Fabric data agent is GA, and its Copilot Studio integration went GA at FabCon Barcelona (October 2026). Data agents in Microsoft Copilot Chat were only a sneak peek there.
The data agent inside Copilot in Power BI was retired on 26 August 2026. Use the data agent directly, or through Foundry, Copilot Studio, Microsoft 365 Copilot or its MCP endpoint.
Operations agent is GA (June to July 2026). Newer abilities, such as root-cause investigations and pre-approved actions, are still in public preview.
Operations agents need a paid Fabric capacity. Trial capacity is not supported, and an admin must turn on the Copilot/AI and operations agent tenant settings.
A data agent is only as good as the model under it. Clear table names, descriptions, relationships and agent instructions matter more than prompt tricks.
An operations agent automates your mistakes too. Start with "recommend" actions that a person approves, then move to automatic actions.
Don't pick Foundry just for governance. Both Copilot Studio and Foundry agents now show up in the same Microsoft governance and security tooling. Pick on build style, model needs and network needs.
Rule of thumb: prototype in Copilot Studio for a week. If you hit a wall (model choice, evals, isolation, scale), that wall is your Foundry requirement.

Sources

Fabric Data Agents Get Smarter on 2 November. Can You Prove Yours Still Answers Correctly?

FarhanJeelani — Mon, 26 Oct 2026 13:49:38 GMT

Microsoft announced Advanced NL2SQL and advanced DAX generation as generally available on 19 October 2026. The core improvements roll out to the standard runtime by default, together with a move to GPT-5.6 Terra. Separately, the old custom-tool way of calling a Fabric data agent from Microsoft Foundry stops being supported on 6 November.

Date (2026)	What happens	What you should have done
19 Oct	GA announced	Read this post
Now to 1 Nov	Changes available on the preview runtime	Build a ground-truth set, baseline standard, test preview
2 Nov	Standard-runtime rollout begins, no opt-in	Baseline frozen and stored
6 Nov	Foundry custom-connection integration unsupported	Foundry agents moved to Fabric IQ
By 10 Nov	Rollout expected to complete	Re-run, diff the queries, tell affected users

What Microsoft actually announced

The query generators stop writing in one pass and start investigating first. That is why behaviour can change even where accuracy improves.

Until now, NL2SQL and NL2DAX generated a query in a single pass, resolving schema, business definitions and filter values at once. As the GA announcement puts it, a wrong guess still runs, and the wrong answer looks like a right one.

Advanced NL2SQL explores the schema, tests joins and looks up real filter values before writing SQL. It works in three modes:

Example query following: adapts a relevant example query, for instance by swapping a filter value.
Clarifying questions: asks the user when it can't answer reliably.
Data exploration: infers logic from your examples and checks assumptions against the data.

Advanced DAX generation brings the same idea to semantic models: an agentic tool-calling loop plus instance value indexing, so a user's "California" maps to the stored "CA".

Microsoft's internal benchmarks

Microsoft's 2026 internal evaluations, standard versus preview runtime. Microsoft notes results vary by workload, model and configuration.

Benchmark	Smart-answer accuracy	P50 latency	P95 latency
NL2SQL, standard	78.8%	28.4 s	46.8 s
Advanced NL2SQL, preview	90.9%	26.0 s	32.9 s
NL2DAX, standard	89.15%	39.1 s	97.5 s
Advanced DAX, preview	91.86%	20.4 s	34.7 s

SQL gains 12.1 accuracy points. DAX gains only 2.7 points, but its tail latency drops by about 64%. If your semantic-model agents felt slow, that is your headline.

What moves, and what doesn't

Moving to standard: Advanced NL2SQL, advanced DAX generation and the GPT-5.6 Terra model upgrade. Staying in preview: schema descriptions and topics for SQL sources.

Two details catch people out (runtime docs):

A published agent keeps the runtime it was published with. Publish on preview and it stays on preview until you republish. Check every production agent.
Runtime choice doesn't pick the model. A preview run today is a strong signal, not a guaranteed copy of standard after 10 November. Re-run after the rollout regardless.

Why "better on average" can still surprise you

A benchmark counts right answers. Your users notice changed answers. The second one generates support tickets.

Workaround instructions start fighting the agent. Rules written to patch single-pass behaviour may no longer be needed, and unneeded guidance can steer an agent away from a path it would have found itself.
Example queries get followed more literally. Good when they are current. An example with last year's fiscal logic now gets reproduced more faithfully.
Ambiguous questions get a question back. Better for users, but it breaks anything downstream that expects a number, such as a Copilot Studio flow or a Foundry agent chain.
A correct new answer can be a business problem. "Active customers in Q3" returned 41,200 because of a wrong date column; now it correctly returns 38,900. Someone may already have put 41,200 in a board pack.
The pass count hides the churn. 40 of 50 pass before, 44 after. Looks like +4. Underneath, 6 newly pass and 2 now fail: 8 changed answers, 2 regressions.

Question status	Count	Action
Pass both, same query	34	Nothing
Pass both, different query	4	Right for the right reason?
Failed before, passes now	6	Tell anyone who acted on the old answer
Passed before, fails now	2	Fix instructions or examples first
Fail both	4	Existing backlog

Illustrative numbers. The structure is what your own run should produce.

The pre-rollout routine: six steps for this week

You need a Fabric notebook, a lakehouse and the Fabric data agent SDK. The SDK and its evaluation features are in preview, so note the version you used. You also need F2 or higher capacity, the data agent tenant settings enabled and read access to the sources.

Step 1: Inventory agents and dependencies

For each published agent, record the runtime it was published on, its source types, where it is consumed (Fabric, Teams, M365 Copilot, Copilot Studio, Foundry) and its business owner. Anything called from Foundry via the old custom tool also goes on your 6 November list.

Step 2: Build a ground-truth set that covers the risky cases

The SDK takes a DataFrame or CSV with question and expected_answer. Aim for 30 to 60 questions per agent, ideally from real usage, covering filter values ("California" stored as "CA"), example-adjacent questions, multi-join questions, ambiguous questions and high-stakes KPI figures.

%pip install -U fabric-data-agent-sdk

import pandas as pd gt = pd.DataFrame(columns=["question", "expected_answer"], data=[ ["Total net sales in California for September 2026", "1,284,550.20"], ["Returns rate by region for FY26", "North 3.1%, South 2.7%, East 4.0%, West 3.4%"], ["What was revenue last year?", "CLARIFY: should ask gross or net, fiscal or calendar"], ["Active customers in Q3 2026", "38,900"], ]) gt.to_csv("/lakehouse/default/Files/agent_eval/ground_truth_sales.csv", index=False)

The values are placeholders. Use answers your business owner has signed off.

Step 3: Use a critic prompt that accepts clarifying questions

The default grader would mark a good clarifying question as wrong. The SDK accepts a custom critic_prompt with {query}, {expected_answer} and {actual_answer} placeholders.

critic_prompt = """ Query: {query} Expected Answer: {expected_answer} Actual Answer: {actual_answer} If the expected answer starts with CLARIFY:, respond 'yes' only if the actual answer asks a clarifying question about the same ambiguity. Otherwise respond 'yes' only if the actual answer is numerically and semantically equivalent. Is the actual answer equivalent to the expected answer? """

Step 4: Capture the standard baseline before 2 November

Run against the production stage, which is what users hit today. Give each run its own table.

from fabric.dataagent.evaluation import ( evaluate_data_agent, get_evaluation_summary, get_evaluation_details) agent_name = "SalesAgent" baseline_table = "eval_sales_standard_baseline_2026_10" baseline_id = evaluate_data_agent( gt, agent_name, table_name=baseline_table, data_agent_stage="production", critic_prompt=critic_prompt) print(get_evaluation_summary(baseline_table)) baseline = get_evaluation_details(baseline_id, baseline_table, get_all_rows=True) print(baseline.columns.tolist()) # check column names in your SDK version

This writes <table_name> with per-question results and <table_name>_steps with the detailed reasoning and step-by-step execution. Look there for the generated SQL or DAX, and keep both tables as evidence. For a single question, the Diagnostics button exports the full steps too.

Step 5: Record P50 and P95 latency through the MCP endpoint

The evaluation tables focus on correctness, so time each question against the published agent's MCP endpoint. This adapts Microsoft's SDK sample:

import time, numpy as np from azure.identity import InteractiveBrowserCredential from mcp import ClientSession from mcp.client.streamable_http import streamablehttp_client mcp_url = ("https://api.fabric.microsoft.com/v1/mcp/workspaces/<workspace-id>" "/dataagents/<data-agent-id>/agent") cred = InteractiveBrowserCredential() async def ask(q): tok = cred.get_token("https://api.fabric.microsoft.com/.default").token async with streamablehttp_client(mcp_url, headers={"Authorization": f"Bearer {tok}"}) as (r, w, _): async with ClientSession(r, w) as s: await s.initialize() tool = (await s.list_tools()).tools[0] arg = next(iter(tool.inputSchema["properties"])) t0 = time.perf_counter() await s.call_tool(tool.name, {arg: q}) return time.perf_counter() - t0 secs = [await ask(q) for q in gt.question for _ in range(3)] print(f"P50 {np.percentile(secs, 50):.1f}s P95 {np.percentile(secs, 95):.1f}s")

Run each question several times; single timings are noisy. The MCP response returns answer text only, not the query, which is why query evidence comes from the steps table.

Step 6: Preview now, standard after 10 November, then diff

Preview, today. Switch your draft to preview and evaluate the sandbox stage, leaving production untouched. The switch comes from the runtime docs (the ribbon's Runtime dropdown works too); check the SDK reference for the FabricDataAgentManagement import.

data_agent = FabricDataAgentManagement(agent_name) data_agent.update_configuration(enable_preview_features=True) preview_id = evaluate_data_agent(gt, agent_name, table_name="eval_sales_preview_2026_10", data_agent_stage="sandbox", critic_prompt=critic_prompt) data_agent.update_configuration(enable_preview_features=False) # don't publish on preview by accident

Standard, after 10 November. Repeat Steps 4 and 5 against production with new table names.

The diff. Compare per question, never just the summary. Fill in your SDK's pass-flag and answer column names.

def diff_runs(before, after, passed="<pass-col>", answer="<answer-col>"): m = before.merge(after, on="question", suffixes=("_b", "_a")) def status(r): b, a = r[passed + "_b"], r[passed + "_a"] if b and a: return "same" if str(r[answer + "_b"]) == str(r[answer + "_a"]) else "answer changed" return "newly correct" if a else ("REGRESSION" if b else "fail both") m["status"] = m.apply(status, axis=1) return m.sort_values("status")

Then compare the generated queries from both _steps tables. A changed query with the same answer deserves a look: it may be right by coincidence on this month's data.

When a result changed, check the run steps for: which example query it picked, which join path it chose, which instruction it followed, and whether a clarifying question was the right call. Most fixes will be in your examples and instructions, not the platform.

The 6 November deadline: move Foundry agents to Fabric IQ

If a Foundry agent calls your data agent through the original custom tool, that path stops being supported on 6 November 2026 (retirement notice).

The catch: the replacement, the Fabric IQ (OneLake Catalog) tool, is in preview. Production agents move from a deprecated integration to a preview one, four days into the query rollout. Make that an explicit risk decision, with a fallback plan.

What you gain: pick agents by name from the catalog, connect several data agents to one Foundry agent (the old tool allowed one), and an MCP endpoint per agent. Microsoft also says responses will soon include the query used.

Portal migration, in Microsoft's order:

Publish the data agent with a rich, detailed description.
In the Foundry portal, open the agent and add the Fabric IQ (OneLake Catalog) tool, then select your data agent.
Update the Foundry agent's instructions to name each data agent and what it handles.
Remove the old custom tool, save, and test with representative questions.

For code, use the MCP URL from the agent's Settings > Model Context Protocol tab and the FabricIQPreviewTool in azure-ai-projects>=2.2.0. Full samples are in Connect agents to Microsoft Fabric with Fabric IQ.

Test the two changes separately. If you migrate on 5 November and answers shift on 8 November, you won't know why. Migrate now and baseline through Foundry before 2 November. The connection calls Fabric as the signed-in user, so test with accounts that match real users' permissions and row-level security.

When the new answer is right, who tells people?

The hardest part is not technical. Someone acted on the old number. Decide now who owns that conversation:

Each agent has a business owner who reviews "newly correct" and "answer changed" rows.
High-stakes answers get a change note: old answer, new answer, why, and since when.
The note goes where the old answer was used: report owners, Teams channels, Copilot Studio makers.
Regressions get fixed, not avoided. Standard is changing anyway, so plan the fix.

Also tell users before 2 November that answers may become more precise and the agent may ask follow-up questions. A warned surprise becomes a feature.

Your checklist

Inventory agents, runtimes, consumers and owners
Build ground-truth sets, including ambiguous and high-stakes questions
Write a critic prompt that accepts correct clarifying questions
Capture the standard baseline, steps and latency before 2 November
Run the set on preview in the sandbox stage
Migrate Foundry agents to Fabric IQ and baseline them before 6 November
Decide explicitly on running production on the preview Fabric IQ tool
Re-run after 10 November and diff per question and per query
Retire workaround instructions and refresh stale examples
Send change notes for high-stakes answers that moved

Microsoft Fabric Security Policies: How They Strengthen Data Security and Governance?

suparnababu8 — Tue, 13 Oct 2026 13:58:44 GMT

In today’s data-driven world, governance isn’t just about compliance — it’s about enabling trust, consistency, and scale. As organizations adopt Microsoft Fabric to unify their analytics, the challenge of managing who can do what, where, and under which conditions become critical. That’s where Microsoft Fabric policies step in.

Think of them as the rulebooks that let administrators define guardrails across workspaces, security groups, and sensitivity labels. Instead of relying on manual oversight, policies provide a centralized, automated way to enforce governance — ensuring that innovation doesn’t come at the cost of control.

In this blog will walk you through the key concepts, the entry points for creating and activating policies, and the end-to-end lifecycle that makes governance in Fabric both powerful and practical.

Prerequisites:

1) A Valid Fabric Trail / Capacity License

2) A Fabric workspace assigned to a Capacity/Fabric Trail.

3) Proper admin role to activate the policy
a) Fabric administrator: Can create and activate policies for tenant-scope policies.
b) Capacity administrator: Can create and activate policies for capacity-scope policies.

4) Contributor, Member, or Admin access to the workspace so that you can create a policy item type.

5) The Policies tenant setting is enabled in Admin settings

6) Valid Security groups in AAD.

Step 1: Enable the Policies feature in the admin settings

Go to fabric admin settings, search for polices(preview) and enable the setting and click on apply.

Step 2: Creating Policy item

Create a workspace ‘Fabric-Polices-Ws’ which was assigned to ‘Fabric Trail’ and I’m (SI Babu) who’s the capacity admin. Make sure this should be a secured, admin-managed workspace to store policies.

Click on ‘New Item’ and select ‘Policy set (preview)’ under ‘Govern data’

In New policy set dialog, enter a unique and descriptive Name ‘Capacity_PolicySet_1’ for the policy set and choose ‘ A capacity’ under policy scope then Click on ‘Create’

Create another policy set ‘Tenent_Policy_set2’ and choose ‘ Fabric Tenant’ under policy scope then Click on ‘Create’

Two policy sets are created.

Step 3: Activating the Policy set

Open Capacity policy set and select the ‘Active’ from Status

Select the capacity and click on ‘Assign + activate’

Policy activated successfully

Open Tenant policy set and select the ‘Active’ from Status

Click on ‘Activate’ the policy

Policy activated successfully

Step 4: Create / Add Policy Rules(Capacity Policy)

Click on ‘OneLake catalog’

Click on ‘Govern’ à‘Polices(preview)’ àCapacitiesàExpand the ‘Users can create items’ and select ‘Advanced’

Here, Select Allowed, Blocked, or Advanced (Set by user) to change the policy behavior. If you select Advanced, the rules configuration is enabled.

Add a new rule with good name, description and Select ‘item type’ in Condition type, operator is ‘Is any of’ the select ‘Lakehouse’ and ‘Dataflow gen2’ then click on ‘+Add condition’

Choose ‘workspace’ in condition type, operator is ‘Is any of’ selected two workspaces gen2’ then click on ‘+Add condition’

Choose ‘Security group’ in condition type, operator is ‘Is any of’ selected my ‘Data Analyst’ AD group. Click on Apply

Note: In this Data Analyst group we have three developers are there

Click on “Save’

Now Policy successfully added

Step 5: Validate policy enforcement (Capacity Policy)

Added ‘Data Analyst’ AD group to both the workspaces ‘FabricRTI Training’ & ‘PowerBI-Demo-WS’

Now I was opened another window with account(SP Babu) who’s part of AD group & you can see two workspaces

Create a ‘Notebook’

Now when I click on “Create” it’s throwing an error as you seen in below screenshot. Means it enforced a policy

Again, When I was trying to create ‘Warehouse’ it’s throwing the same error.

I (SP Babu) was able to create ‘DataFlowGen2’. Admin enforced to create only this item

I (SP Babu) was able to create ‘Lakehouse’. Admin enforced to create only this item

Step 6: Create / Add Policy Rules(Tenant Policy)

Create another policy should be at tenant level to restrict workspace Outbound access protection and inbound network rules and click ‘add condition’

Add “Security group’ as I have IT Infra Team and click Apply

Note: In this IT Infra Team group we have three resources are there

Security policy created successfully

Step 7: Validate policy enforcement (Tenant Policy)

Now SP Babu unable to edit ‘Outbound access’ it’s showing ‘This setting is managed by a tenant policy.’

SP Babu unable to edit ‘Inbound access’ it’s showing ‘This setting is managed by a tenant policy.’

You can deactivate or replace an active policy set from the workspace where the policy set is stored. Only an administrator of the policy set's admin scope can deactivate or replace it.

Best practices for creating a policy:

Use a secured, admin-managed workspace to store policies.
Store all policies for the admin scopes managed by the same administrator or admin team in the same workspace. This approach provides one location for tenant- and capacity-scoped policies and simplifies access management, discovery, and lifecycle management.
Use clear policy set and rule names that identify the admin scope and purpose.
Use test users who are members and non-members of the configured security groups to validate both allow and deny outcomes before broader rollout.

Limitations as of now:

Fabric policies aren't currently supported in the following capacity regions: West Europe, North Europe, and West US.
We can currently create policy sets only from the item creation experience in a workspace.
Only workspaces assigned to a Fabric capacity can store policy sets.
We can't change a policy set's admin scope after creation.
Only one policy set can be active for an admin scope at a time.
Policy and tenant-setting changes can take time to propagate.
Conditions support Any of and None of in the current preview.
We can't activate a policy set through Git.

So, start small with pilot policies, monitor audit logs for adoption and Communicate changes to workspace owners

Conclusion

Microsoft Fabric Policies are still in preview, but they signal a future where governance is not a blocker — it’s an enabler of responsible innovation.

Happy learning!

— Inturi Suparna Babu

[LinkedIn]

I Let Fabric's New Data Engineering Agent Run My ETL: Here's What Happened

FarhanJeelani — Fri, 09 Oct 2026 13:44:42 GMT

Last week I handed Microsoft Fabric's new Data Engineering Agent a messy ETL job that took me 11 hours to build by hand last quarter. It had a working, validated pipeline in front of me in under an hour. I still changed 3 of its 9 steps before I'd let it near anything important.

That is the short version. The long version is more interesting, because the places where it went wrong tell you exactly how to use it well.

Some context first. At FabCon Europe 2026 in Barcelona (September 28 to October 1), Microsoft announced the Data Engineering Agent in preview. It can plan and run migrations, ETL and performance work under guardrails that engineers define. It builds on technology from Osmos, the agentic data engineering startup Microsoft acquired in January 2026, and was demoed on stage as "Project Osmos".

Copilot has been helping us write code for a while now. This is a different idea. You describe the outcome, and the agent does long-running, multi-step work on its own while you supervise. I wanted to know whether that holds up on a real, slightly ugly dataset rather than a keynote demo. So I tried it.

What the Data Engineering Agent actually is

The agent is an autonomous worker inside your Fabric workspace, not a chat assistant that suggests code. You give it a goal, and it does the multi-step work itself.

Its roots are in Osmos, a Seattle startup whose "AI Data Engineer" already generated production-grade PySpark notebooks inside Fabric. Microsoft bought the company in January 2026 and folded the team into Fabric engineering.

Based on Microsoft's FabCon session on Project Osmos, a typical run looks like this:

Read the request. You describe the outcome, even loosely ("land the orders CSVs in a clean silver table").
Inspect the workspace. It looks at your existing code, Lakehouses and data layout before writing anything.
Plan. It breaks the job into steps and can hand parts to sub-agents for long-running work.
Build. It creates and updates notebooks and other Fabric items.
Validate. It checks its own output so you can trust what was built.
Hand back to you. A human reviews and approves. Microsoft is explicit that supervision stays part of the process.

The headline use cases are migrations, ETL and performance tuning. I chose ETL, because that is where most of us spend our week.

The test: a real pipeline, not a demo

I picked a job I had already built by hand, so I could judge the agent's work line by line against my own.

The data is sales orders from a mid-sized retail client . Every day their order system drops a CSV export into a Lakehouse folder. The business wants a clean orders table for analysts and a daily sales summary for a Power BI report.

Item	My setup
Source data	14 daily CSV exports, 18.4 million rows, about 2.1 GB, in Files/raw/orders/
Target	silver_orders (cleaned Delta table) and gold_daily_sales (aggregate by day, store and category)
Known traps	Duplicate order IDs from system retries, two date formats, a currency column with nulls, cancelled orders flagged as negative quantities
Capacity	F64 trial capacity, sandbox workspace
Hand-built baseline	About 11 hours across two days, including testing

The exact prompt I gave it:

Ingest the daily order CSVs in Files/raw/orders into a cleaned silver Delta table called silver_orders. Remove duplicates, standardise dates and currency, and exclude cancelled orders. Then build gold_daily_sales with total revenue, order count and average basket size by day, store and category. The pipeline must run incrementally each day.

I deliberately kept the prompt a little vague. I didn't say which date formats exist, how cancellations are encoded, or which column identifies a duplicate. In real life, requirements rarely arrive as a perfect spec. I wanted to see whether the agent would investigate, ask, or just guess.

What happened, step by step

1. It read the workspace before writing anything

The first thing I noticed: it didn't start coding. For the first four minutes it profiled the data. It sampled each CSV, inferred the schema and looked at my existing notebooks in the workspace.

It came back with a short summary of what it found. Two findings impressed me:

It spotted both date formats (2026-09-14 and 14/09/2026) and flagged that files from 9 September onwards switched format.
It noticed my existing notebooks used a nb_<layer>_<entity> naming pattern and said it would follow it.

It missed one thing, which matters later: it reported quantity as "integer, some negative values" without asking what negative quantities meant.

2. The plan it proposed

Next it produced a 9-step plan and stopped for my approval:

Create a staging notebook to read all CSVs with an explicit schema
Normalise both date formats to a single order_date column
Fill null currency values from the store's default currency
Deduplicate on order_id, keeping the latest record
Remove cancelled orders
Write silver_orders as a Delta table, partitioned by order_date
Build gold_daily_sales with the three requested measures
Add incremental logic based on the source file date
Run validation checks and produce a summary report

This is a sensible plan. It is roughly what I wrote on a whiteboard last quarter. But reading it closely, I changed three things before approving:

Step 4: it planned to overwrite silver_orders on each run. I changed it to a MERGE on order_id, because late-arriving corrections are common in this data.
Step 5: it planned to remove cancellations using a status column. That column exists, but this client never fills it. Cancellations show up as negative quantities. I told it so.
Step 8: it wanted to detect new data by file modified time. I asked it to use the date in the file name instead, which is more reliable when files are re-uploaded.

It accepted all three changes and rewrote the plan in a few seconds.

3. The build

Once approved, it worked for 38 minutes without needing me. It created three notebooks (nb_stg_orders, nb_silver_orders, nb_gold_daily_sales) and a small pipeline to run them in order.

The code was clean and readable. Here is its date handling, which is close to what I would write myself:

from pyspark.sql import functions as F df = df.withColumn( "order_date", F.coalesce( F.to_date("order_date_raw", "yyyy-MM-dd"), F.to_date("order_date_raw", "dd/MM/yyyy") ) )

And the merge into silver, after my change in the plan:

from delta.tables import DeltaTable silver = DeltaTable.forName(spark, "silver_orders") (silver.alias("t") .merge(updates.alias("s"), "t.order_id = s.order_id") .whenMatchedUpdateAll() .whenNotMatchedInsertAll() .execute())

It asked me one question mid-way: whether average basket size should be revenue per order or items per order. That was exactly the right question, and one I had to ask the client myself last time.

4. Self-validation

When the build finished, it ran its own checks and produced a short report:

Row counts reconciled from raw to staging to silver, with each dropped row explained (duplicates, cancellations)
No nulls left in order_date, currency or store_id
No duplicate order_id values in silver_orders
Daily totals in gold_daily_sales matched a direct aggregate of silver_orders

One check failed on the first pass. 41,200 rows had a null order_date after parsing. It traced the cause itself: a third date format (14-Sep-2026) appeared in two of the older files, which its initial sampling had missed. It added a third pattern to the coalesce, re-ran, and the check passed.

That moment sold me on the validation step. A human could easily miss 0.2% of rows quietly disappearing.

5. My review against my own version

I compared its output with the pipeline I built by hand last quarter, running both on the same 14 files.

Measure	Agent	Me, by hand
Time to working pipeline	52 minutes (incl. my review)	About 11 hours
Plan steps I changed	3 of 9	Not applicable
Data issues found	4 of 4, after one self-correction	4 of 4
Full run time on 18.4M rows	6 min 40 s	5 min 55 s
Silver row count	17,962,118	17,962,118

The row counts matched exactly. My hand-built version ran about 45 seconds faster, mostly because I had tuned partitioning. More on that below.

Would I merge its code as-is? After my three plan changes, yes, with one small performance fix.

Guardrails: the part you shouldn't skip

An agent that can create and change items in your workspace needs limits before it needs a prompt. Microsoft designed this agent around guardrails that engineers define, and setting them up was the best 15 minutes of the whole test.

What I set up before the run:

A sandbox workspace. It's a preview feature, so it ran on a copy of the data, nowhere near production.
Scoped data. It could read Files/raw/orders/ and write only to new tables. My existing tables were off limits.
Approval points. It had to stop for my approval after the plan, and again before the first write to any Delta table.
Git in the loop. The workspace was connected to a Git repo, so every notebook it created or changed landed as a diff I could review line by line.

In practice, it never tried to step outside these limits. The approval after the plan turned out to be the most valuable one: that is where I caught all three of my changes. If you set only one guardrail, make it that one.

Where it struggled

No agent review is credible without the failures, so here are the three that mattered.

It assumed instead of asking about business rules. It saw negative quantities and planned to use an empty status column for cancellations. Had I approved the plan without reading it, about 310,000 cancelled orders would have flowed into the gold table and inflated revenue. It asked a good question about basket size, but not this one.
Its sampling missed a rare date format. The third format appeared in only two files. Its validation caught the problem, but only after the first run. On a much larger dataset, that first run costs real capacity.
Correct, but not tuned. It partitioned silver_orders by order_date, which produced many small files for this data volume. Switching to monthly partitions and running OPTIMIZE closed most of the 45-second gap with my version. When I asked it to "make the silver write faster", it suggested the same fix, but it didn't do it unprompted.

Three caveats worth knowing before you try it:

It's a preview. Features, behaviour and pricing can change before general availability. Don't build production processes on it yet.
Watch your capacity. Long-running agent work consumes capacity units on top of the Spark jobs themselves. Check the Capacity Metrics app after your first run so you know what a typical job costs.
It doesn't know your business. It can read your schema, not your meeting notes. Every rule I wrote into the prompt was handled correctly. Every rule I left out was a guess.

The verdict

The Data Engineering Agent is like a fast, tireless junior engineer: excellent at the mechanics, careful about checking its own work, and still in need of a senior reviewing the plan.

What I'd tell my team after this test:

Spend your time on the prompt and the plan, not the code. Every problem I found was visible in the plan before a single line ran.
Write business rules down explicitly. Cancellations, duplicates, currency defaults. If it's tribal knowledge, the agent will guess.
Trust the validation step, but read its report. It caught a data loss bug I might have missed myself.
Tune afterwards. Treat its first version as correct, then ask it for performance improvements.

Try it now if you have a sandbox workspace, a backlog of repetitive ingestion or migration jobs, and someone who can review PySpark.

Wait for GA if you need it in production, your pipelines are full of undocumented business logic, or your capacity budget is tight.

The bigger shift is in our role. Going from 11 hours to under an hour didn't remove me from the work. It moved my effort from typing code to writing clear requirements, setting guardrails and reviewing output. That is a skill worth building now, preview or not.

Have you tried the Data Engineering Agent yet? Tell me in the comments what you threw at it, and where it surprised you.

Sources

ArcGIS Maps for Microsoft Fabric Is Now Generally Available

Tonyfra91 — Thu, 08 Oct 2026 15:22:42 GMT

ArcGIS Maps for Microsoft Fabric is now generally available, bringing advanced mapping and geospatial visualization directly into Microsoft Fabric. Explore how organizations can visualize OneLake data, work with real-time and feature layers, and turn enterprise data into location-aware insights.

Privacy by Design in Fabric: Implementing Shift-Left Data Protection

VLopez03 — Mon, 05 Oct 2026 18:40:42 GMT

Privacy by Design in Microsoft Fabric: Implementing "Shift-Left" Data Protection for Enterprise Analytics

Abstract

While data engineers and architects often prioritize pipeline performance, Medallion architectures, and complex transformations, regulatory compliance and data security are frequently treated as afterthoughts. In highly regulated sectors like banking and finance, failing to protect Personally Identifiable Information (PII) can lead to severe penalties under evolving frameworks such as GDPR, PIPEDA, and other international privacy laws. This article explores the concept of "Shift-Left" data protection within Microsoft Fabric—implementing data masking, hashing, and external tokenization to secure sensitive data early in the lifecycle—while distinguishing robust security controls across OneLake, SQL Analytics Endpoint, and Semantic Models.

Introduction

Modern data professionals excel at writing efficient PySpark scripts, optimizing SQL queries, and designing multi-layered Medallion Lakehouses. However, technical elegance means little if an analytics platform exposes sensitive operational data to unauthorized risks. With the rise of global data privacy regulations and increasing security threats, data governance can no longer be a secondary phase applied right before dashboard deployment; it must be embedded directly into the ingestion and storage strategy.

A fundamental best practice in enterprise data architecture is the "Shift-Left" security pattern: masking, pseudonymizing, or hashing sensitive PII attributes at the earliest possible stage rather than relying solely on downstream presentation layers. Within Microsoft Fabric, data teams can execute this strategy across pipelines, PySpark notebooks, and Semantic Models. This article examines practical techniques for implementing early-stage data protection and dynamic security controls in Fabric, helping data engineers build scalable analytical solutions that support strict regulatory mandates.

What is "Shift-Left" Data Protection?

In traditional data engineering, security controls are frequently applied at the very end of the pipeline—usually at the presentation layer using Power BI dashboard permissions or SQL database views. This legacy approach creates a critical vulnerability: raw, unmasked Personally Identifiable Information (PII) rests in landing zones or intermediate data lake layers, accessible to anyone with direct storage access.

"Shift-Left" Data Protection fundamentally changes this dynamic by pushing security, masking, and pseudonymization controls as far upstream as possible.

This approach is highly valuable for organizations navigating cross-border data transfers and cloud migrations. Many international regulatory frameworks strongly recommend or mandate that when transferring sensitive data across borders, the information should be pseudonymized or encrypted to support data sovereignty and localization principles. Failing to comply with privacy frameworks carries severe financial and reputational consequences for enterprises worldwide.

By applying cryptographic hashing, masking, or tokenization patterns early in the Medallion Architecture:

Exposure is Minimized: Sensitive data (such as national IDs, emails, or account numbers) is restricted or neutralized before persisting in downstream analytical environments.
Supports Compliance Initiatives: Organizations are better positioned to align with privacy principles established by global frameworks (like GDPR or PIPEDA) by adopting a Privacy by Design architecture.
Governance is Streamlined: Instead of managing complex downstream permission rules across dozens of individual reports, core data security is enforced at the storage and engine layers.

To determine whether an enterprise data pipeline truly adheres to a "Shift-Left" architecture, data engineering and compliance teams should verify key architectural control points, such as:

[ ] Ingestion Gatekeeping: Are raw PII fields masked in transit before landing in the Bronze layer, or if landing raw in Bronze, is direct storage access strictly limited to service principals and administrators?
[ ] Pseudonymization (Bronze to Silver): Are identifying attributes transformed into deterministic hashes (like salted SHA-256) before persisting into Silver Delta tables?
[ ] Integration with Token Vaults: Are highly sensitive fields passed to external tokenization services during ingestion, ensuring the decryption vaults are isolated outside the analytics workspace?
[ ] Identity and Role Verification: Are workspace roles configured correctly so that downstream consumers only access data via secured endpoints rather than having direct underlying OneLake file access?

Demystifying the Security Layers in Microsoft Fabric

Implementing a "Shift-Left" framework in Fabric requires understanding that security is not monolithic; it is applied across distinct layers. A robust architecture leverages data-layer controls, transformation techniques, and compute-engine security rules.

1. The Storage & Transformation Layer (Upstream)

Source-Side Masking: Using Data Factory pipelines to mask data in transit before it ever writes to OneLake.
Hashing as Pseudonymization: Using PySpark notebooks to apply SHA-256 hashing to fields like email addresses. Note: Hashing is pseudonymization, not true anonymization. To prevent dictionary or rainbow table attacks, data teams must implement secure "salting" alongside the hashing algorithm.
External Tokenization Integration: Fabric pipelines can call external REST APIs (e.g., third-party tokenization services or Azure Key Vault) to replace sensitive strings with non-mathematical tokens during the Bronze-to-Silver transformation.
OneLake Security (Data Access Roles): Fabric increasingly supports native OneLake security controls, allowing administrators to define data access roles directly on the OneLake folders and Delta tables, securing the files regardless of which compute engine accesses them.

2. The Engine & Consumption Layer (Downstream)

Once data is curated, Fabric provides specific, engine-level security controls:

SQL Analytics Endpoint (T-SQL Controls):
- Row-Level Security (RLS): Filters specific rows (e.g., restricting a regional manager to only see sales from their territory) using standard T-SQL CREATE SECURITY POLICY statements.
- Column-Level Security (CLS): Uses T-SQL GRANT SELECT statements to prevent specific roles from querying sensitive columns like Salary or SSN, while allowing them to query the rest of the table.
Semantic Models (Direct Lake / Power BI Controls):
- Row-Level Security (RLS): Configured via DAX to filter data rows natively within the semantic model.
- Object-Level Security (OLS): Configured via Tabular Editor/XMLA to completely hide entire tables or columns from specific users in the semantic model, preventing report visuals from even discovering the fields.

High-Level Architecture in Microsoft Fabric

Implementing a "Shift-Left" data protection framework within Microsoft Fabric relies on combining two complementary security layers:

In-Flight Data Sanitization (Storage Layer): Using Fabric PySpark Notebooks or Data Factory pipelines to hash, mask, or tokenize PII fields during ingestion or before writing into Silver Delta Tables in OneLake.
Dynamic Access Controls (Engine Layer): Enforcing Row-Level Security (RLS) and Column-Level Security (CLS) on top of the SQL Analytics Endpoint or Direct Lake Semantic Models for role-based downstream consumption.

To visualize how this defense-in-depth strategy comes together, the following reference architecture outlines how "Shift-Left" sanitization and dynamic query controls are orchestrated within Microsoft Fabric:

Understanding the Architectural Diagram

The architectural diagram above illustrates a simple example of an end-to-end Microsoft Fabric implementation designed around the "Shift-Left" paradigm. To read and interpret this architecture effectively, consider the flow of data through three core operational layers:

1. Data Sources & Boundary Controls (Left):

Data flows from operational systems (On-Premises SQL Databases, REST APIs) toward Microsoft Fabric OneLake.
Red Locks (Regulatory Blocking & Ingestion Masking): Represent early security controls applied at the connector level. Sensitive operational fields are masked or blocked before landing in cloud storage to prevent cross-border compliance violations.

2. Medallion Lakehouse & In-Flight Sanitization (Center):

Bronze Layer (Raw Ingestion): Stores incoming raw landing data.
Shift-Left Sanitization (PySpark Notebook): Before moving data into the Silver Layer, PySpark notebooks perform cryptographic transformations (SHA-256 hashing, masking, tokenization).
Silver Layer (Sanitized & Standardized): Contains cleaned, enterprise-ready data where PII has been neutralized at the storage level within OneLake Delta files.

3. Consumption & Dynamic Engine Security (Right):

Gold Layer (Aggregated & Enriched): Curated dimensional models built for business analytics.
SQL Analytics Endpoint & Direct Lake Semantic Models: As queries hit the serving layer, secondary security rules—such as Column-Level Security (CLS) and Row-Level Security (RLS)—dynamically filter rows or restrict specific columns based on the user's Entra ID (Azure AD) identity, ensuring seamless role-based governance across Power BI reports and T-SQL queries.

The Critical Role of Workspace Identity and Direct Storage Access

The most common pitfall when designing security in Microsoft Fabric is a misunderstanding of Workspace Roles and Identity Modes.

SQL RLS/CLS and Semantic Model RLS/OLS are engine-level controls. If a user is granted the Admin, Member, or Contributor role in a Fabric Workspace, they possess inherent direct storage access to the underlying OneLake files. Because they can access the raw Delta files directly via OneLake Explorer or Notebooks, they will completely bypass SQL RLS/CLS and Semantic Model security.

To successfully enforce engine-level security (like SQL CLS or Direct Lake RLS), downstream consumers must be assigned the Viewer role in the workspace. This restricts their access to the defined endpoints (SQL Endpoint or Power BI App) where the security policies are evaluated under their Entra ID credentials, effectively blocking direct access to the unmasked storage files.

Key Takeaways for Data Professionals

Implementing "Shift-Left" data protection within Microsoft Fabric allows organizations to bridge the gap between high-performance analytics and governance requirements. By moving sanitization controls upstream and combining them with dynamic engine-level security, data teams can build a future-proof architecture that meets strict privacy mandates without sacrificing agility.

To successfully apply these principles in your enterprise, keep the following core takeaways in mind:

Security is an Architecture Feature, Not a Final Step: Embed data protection requirements into your initial pipeline designs rather than treating them as an afterthought.
Protect at the Lowest Layer Possible: Hashing, tokenizing, and masking data early in the Medallion Architecture (Bronze-to-Silver transition) protects sensitive attributes across all downstream environments, including Notebooks, SQL queries, and Direct Lake Power BI reports.
Align Controls to the Right Layer: Use OneLake security for raw storage, T-SQL CLS/RLS for the SQL Endpoint, and DAX OLS/RLS for Semantic Models.
Audit Workspace Roles: Never rely on SQL or Semantic Model security for users who have Admin, Member, or Contributor access to the workspace.

Join the Conversation!

Designing enterprise-grade security in modern data platforms is an evolving practice, and different organizations adopt unique strategies depending on their regulatory environment.

What additional techniques or governance frameworks do you implement in your organization to protect sensitive PII in Microsoft Fabric?
Are you primarily leveraging SQL-based CLS, or shifting toward OneLake data access roles?

Share your thoughts, experiences, or questions in the comments below!

Microsoft Fabric Data Agent in M365

uzuntasgokberk — Mon, 05 Oct 2026 16:16:26 GMT

In this short demo, I tested how a Microsoft Fabric Data Agent can be used through Microsoft Teams and Microsoft 365. I also showed how prompt wording can affect an answer when querying Green Trip data.

Key Takeaways

A Data Agent can be published for use in Microsoft 365.
I accessed the published agent through Microsoft Teams.
The sample agent used Green Trip data.
An initial passenger question returned an answer I identified as incorrect.
Refining the prompt produced the expected response in this scenario.
Chat history remained available through the app.

Microsoft Fabric Data Agent publishing to Microsoft 365

I started with a Microsoft Fabric Data Agent that I had created earlier. When publishing it, I selected the option to publish it to Microsoft 365 as well.

After publishing, I could view the agent details, including its URL and MCP information. This makes the Data Agent available beyond its original Fabric context.

Next, I moved to Microsoft Teams. I could open the agent through the app, and I also showed an option to access it through the Microsoft 365 corporate area in Teams.

Opening the sample agent in Teams

The agent in my demo was named Sample Data Agent and was fed by Green Trip data. Once opened in Teams, I could ask questions directly in the chat interface.

This demonstrates a practical Microsoft Fabric Data Agent in Teams workflow: publish the agent, open it through the Microsoft 365-connected experience, and interact with the available data through chat.

I first asked for the highest number of passengers. The agent returned 9 passengers, while also showing the Fabric MCP activity behind the response.

However, I knew that this result did not answer the question I intended to ask. The issue was the wording of my prompt, not a confirmed issue with the integration itself.

I then refined the question to ask: Which year has the highest total passenger count? The Data Agent MCP ran again in the background and returned the response I expected.

This quick test shows why prompt specificity matters. Asking about the “highest number of passengers” can produce a different result than asking for the year with the highest total passenger count.

What this demo did not test

This was a short, single-scenario demonstration. I did not test multiple data sources, alternative permissions, or a wider set of questions. The specific year and passenger total returned by the refined prompt were not stated in the demo.

Can Microsoft Fabric Data Agent chat history remain available?

Yes. In the app experience I showed, the chat history remained available. This allows me to revisit previous Data Agent conversations from the interface.

Frequently Asked Questions

Can I use a Microsoft Fabric Data Agent in Microsoft Teams?

In this demo, yes. I published the Data Agent to Microsoft 365 and opened it through Microsoft Teams.

Why did the first passenger question return an incorrect result?

I identified that the first question was not specific enough for the answer I wanted. After I changed the prompt to ask about the year with the highest total passenger count, the agent returned the expected response.

Conclusion

This Microsoft Fabric Data Agent demo showed a simple path from publishing to Microsoft 365 through asking questions in Teams. I also observed that clearer prompts can help the agent return a more relevant response for the intended question. If you use Data Agents with Microsoft 365, consider testing your prompts carefully and reviewing your chat history as you work.

CDC vs Watermark: Choosing an Incremental Load Strategy in Microsoft Fabric

FarhanJeelani — Mon, 05 Oct 2026 16:15:09 GMT

Why incremental loading matters

If your source deletes rows or you need history, use CDC; if you only need inserts and updates and have a reliable "last modified" column, a watermark is simpler and works almost everywhere. That is the short answer. The rest of this post explains why.

Reloading a whole table every night is fine at 50,000 rows. At 500 million it burns capacity units, hammers the source system, and stretches your refresh window. Incremental loading fixes this by moving only what changed since the last run.

In Microsoft Fabric, the main tool for this is the Copy job in Data Factory. It supports both change detection approaches side by side, and can pick one per table. Knowing how each works helps you choose well and avoid silent data gaps.

The watermark pattern

A watermark load copies only rows whose "incremental column" value is higher than the highest value seen in the last successful run. The first run copies everything; each later run picks up where the last one stopped.

The query behind it is simple:

SELECT * FROM dbo.Orders WHERE ModifiedAt > @LastWatermark AND ModifiedAt <= @CurrentWatermark;

In a classic pipeline you build this yourself: a Lookup for the old watermark, a Copy activity, and a step that saves the new value to a control table. Copy job removes that plumbing. It stores the watermark state for you, and a failed run resumes from the last successful one without data loss (Microsoft Learn).

Copy job accepts these column types as a watermark:

ROWVERSION – changes automatically on every insert or update; the most reliable choice on SQL Server-family sources.
Datetime – columns such as ModifiedAt or LastUpdatedDatetime.
Date – date-only columns; Copy job re-reads the last day to avoid gaps.
String – values that can be read as datetimes.
Integer – an increasing number.

A column of another type can still work if you cast it to a supported type in the source query, as long as its values compare in order. File sources use the file's last-modified time as the watermark.

The catch: a watermark only sees rows that still exist and whose column actually moved. Deleted rows are invisible, and rows with a NULL watermark are skipped on every incremental run after the first.

Change data capture (CDC)

CDC reads the source database's own change log instead of querying the table, so it captures inserts, updates and deletes exactly as they happened. You don't pick an incremental column; the database tells Fabric what changed.

When a source table has CDC enabled, Copy job detects it, does an initial full load, and then replicates only changes on later runs (Microsoft Learn). Sources that support CDC in Copy job include Azure SQL Database, Azure SQL Managed Instance, on-premises SQL Server, Oracle, Snowflake, Google BigQuery, SAP Datasphere Outbound, and Fabric Lakehouse tables (via Delta Change Data Feed).

You then choose how changes land in the destination:

SCD Type 1 (Merge) – the default. The destination mirrors the current source: updates overwrite, deletes remove the row.
SCD Type 2 – in preview. Copy job keeps every version, adding Valid_From, Valid_To and Is_Current columns. Deletes become soft deletes, so you keep a full audit trail with no custom code. Oracle sources don't support it yet.

Copy job isn't the only CDC route in Fabric. Mirroring keeps a near real-time replica of an operational database in OneLake with almost no setup, and Eventstreams offers CDC connectors for streaming changes into Real-Time Intelligence. Microsoft's data movement decision guide compares the three.

Side by side

The deciding row is usually deletes: only CDC sees them.

Consideration	Watermark	CDC
Source prerequisite	A column that always increases on insert or update	CDC enabled on the source and supported by the connector
Inserts	Yes	Yes
Updates	Only if the watermark column changes	Yes
Deletes	No	Yes
History (SCD Type 2)	No, unless you build it	Built in (preview)
Write methods	Append or Merge	Merge or SCD Type 2
Load on source	Range scan on the watermark column each run	Reads the change log; lighter on busy tables
Setup effort	Low; no DBA changes	Needs CDC enabled, permissions, log retention
Works with files	Yes (last-modified time)	No
Source coverage	Most database connectors	A defined list of connectors

Source: Incremental copy in Copy job, Microsoft Learn.

Which one should you use?

Three questions settle it for almost every table.

If CDC is the right answer but you can't enable it on the source, fall back to a watermark and catch deletes with soft-delete flags or a periodic key comparison.

Tips and pitfalls

For watermark loads

Prefer ROWVERSION over app timestamps. An application can forget to update ModifiedAt; the database never forgets a rowversion.
Watch for NULLs. Rows inserted later with a NULL watermark are never picked up. Make the column NOT NULL with a default.
Catch deletes another way. Use soft deletes (an IsDeleted flag that also bumps the watermark), or run a periodic key comparison against the source.
Use Merge, not Append, for updates. Append writes a second copy of every updated row.

For CDC loads

Don't mix table types in one job. If a Copy job holds both CDC-enabled and non-CDC tables, it falls back to watermark for all of them. Split them into separate jobs.
Size log retention. If the job is paused longer than the source keeps its change data, you will need a full reload.
Know the current limits. Copy job captures net changes only (not every intermediate change) and supports only the default capture instance. For Lakehouse sources, enable Change Data Feed yourself; Copy job can't detect it (Microsoft Learn).

For both

Reset carefully. Resetting incremental state forces a full read on the next run but doesn't clear the destination. With Append, truncate the target first or you'll get duplicates.
Plan for schema changes. New source columns aren't synced automatically, and a dropped column that's in your column mapping fails the run.

Conclusion

Watermark and CDC aren't rivals; most Fabric estates use both. Use a watermark for append-heavy fact tables, files, and sources you can't change. Use CDC wherever deletes or history matter, which is most dimension and master-data tables.

Copy job makes the choice cheap to revisit: it detects CDC-enabled tables, handles state for you, and lets you reset per table. Start with the table that hurts most in your nightly full load, pick the method from the guide above, and measure the capacity you save.

Sources

Incremental copy in Copy job – Microsoft Learn
Change data capture (CDC) in Copy job – Microsoft Learn
Choose a data movement strategy – Microsoft Learn
Copy job enhancements on incremental copy and CDC – Microsoft Fabric Blog

FabCon & SQLCon 2026: The Ultimate Roundup of Announcements

Pragati11 — Mon, 05 Oct 2026 16:14:37 GMT

While I wasn’t able to catch the action live in Barcelona last week for FabCon and SQLCon 2026, keeping tabs on the announcements has been a wild ride. Microsoft Fabric has officially cleared the 40,000-customer milestone and the event heavily emphasized Frontier Transformation – moving past standard AI productivity hacks and embedding autonomous agents directly into corporate workflows.

Yet, among all the flashy Copilot integrations and agentic apps, one specific announcement completely stole the show for me: the introduction of the F0 capacity tier with on-demand billing.

For anyone who has ever wrestled with budget approvals just to test out a new data platform, this is a game-changer. Let’s dive into why F0 is my favorite takeaway, alongside a full roundup of everything else that dropped in Barcelona.

1. My #1 Favorite: The F0 Capacity SKU & Pay-As-You-Go Billing

Historically, dipping your toes into Microsoft Fabric meant committing upfront to a reserved compute tier (like an F2 or higher). If you just wanted to spin up a quick test environment, build a proof of concept, or play with OneLake shortcuts, you still had to navigate a monthly capacity bill.

What’s new: Microsoft introduced F0, a zero-provisioned capacity SKU featuring true on-demand billing.
Why it’s a massive deal: It acts as the “serverless” equivalent for Fabric. You can now evaluate features, run lightweight development tasks and test out workloads without locking into an upfront monthly reservation. Compute costs finally align strictly to what actually runs, shattering the barrier to entry for smaller teams and developer sandboxes.

2. Grounding Microsoft Copilot with Fabric IQ

AI tools often hallucinate because they lack proper context. Microsoft is tackling this by wiring business context directly into Microsoft Copilot.

What’s new: Fabric IQ is generally available as a shared intelligence bridge, connecting OneLake data, Power BI semantic models and operational metrics directly into Copilot Chat and Cowork – with zero additional AI token costs.
Why it matters: Instead of guessing, your organization’s Copilot can answer business questions using the exact semantic data definitions your teams already trust and govern.
Read more: Check out Arun Ulag’s keynote overview on the official Microsoft Azure Blog.

3. Power BI’s Evolution: Agentic App Creation

Power BI is stepping past standard dashboards into operational application building.

What’s new: You can use natural language inside Power BI Desktop to build, preview and publish purpose-built data applications straight from a trusted semantic model. These apps can handle inputs, write-back data and power workflows.
Why it matters: Power BI Pro and Premium Per User (PPU) users will get access to Fabric Apps and Fabric Database capabilities (up to 1 GB per app) at no additional cost.
Read more: Dive into Mohammad Ali’s deep dive on Power BI’s next chapter: The evolution of business intelligence.

4. Fast-Tracking Apps to Production with Fabric Apps & Rayfin

Prototyping an app with AI is easy, but getting it to a secure production environment used to mean heavy lifting. Fabric Apps solves this.

What’s new: Using the open-source Rayfin SDK and CLI, developers can build app backends and deploy them directly to Fabric with built-in TypeScript functions, a Secret Store, PostgreSQL support and private-by-default security.
Why it matters: Apps plug straight into existing Lakehouses and Warehouses without data duplication.
Read more: Catch Sachin Patney’s post, From prompt to production: What’s new in Fabric Apps.

5. Expanding the OneLake Ecosystem & IQ Sharing

What’s new: Microsoft rolled out IQ sharing (in preview), allowing organizations to securely share governed data, markdown agent instructions and ontologies across different teams, partners and external ecosystems.
Read more: Read Dipti Borkar’s breakdown on What’s new in Microsoft OneLake and its rapidly growing ecosystem.

6. Autonomous Data Engineering Agents

What’s new: Driven by technology from the Osmos acquisition, Microsoft introduced a preview of the Fabric data engineering agent. Rather than just auto-completing code snippets, this agent handles complex, long-running engineering tasks like migrations, ETL pipeline creation and modernizations based on human guardrails.
Read more: Check out Bogdan Crivat’s blog on Bringing governed analytics into the flow of work: Fabric Analytics at FabCon Europe 2026.

7. SQLCon: Scale, Intelligence, and Serverless Pauses

Running parallel to FabCon, SQLCon delivered major updates for database administrators balancing traditional relational databases with modern demands.

What’s new: A new Database Hub (Public Preview) serves as a single pane of glass to monitor health and security across SQL Server, Azure SQL and PostgreSQL. Plus, DiskANN vector indexes are now generally available to supercharge native semantic search using T-SQL.
Read more: Read Shireesh Thota’s update on SQLCon Barcelona 2026: Advancing SQL with greater control, scale and intelligence.

Final Thoughts: Looking Ahead

Reflecting on everything announced in Barcelona, it’s clear that Microsoft is moving faster than ever to bridge the gap between heavy-duty data engineering and practical, AI-driven applications. While agentic workflows and Copilot enhancements grab the headlines, structural updates like the F0 tier are what truly make this ecosystem accessible to everyone from lone developers to enterprise architects.

Honestly, I am still trying to digest all these announcements made and also can’t wait to try them soon! We are officially stepping into an era where building, scaling and operationalizing data isn’t just about managing tables – it’s about powering the intelligence layer that runs the business.

Are you as thrilled about the F0 capacity tier as I am? What feature from FabCon are you testing first? Let me know in the comments below!

ADLS Gen2 and OneLake - Choosing the right storage approach

PrabhKaur — Wed, 30 Sep 2026 13:45:53 GMT

As Microsoft Fabric adoption grows, a practical architecture question follows: how does OneLake relate to Azure Data Lake Storage Gen2 (ADLS Gen2), and do we need to migrate? Both store analytics data in open formats, but they offer different ways to manage and consume it. I would start with the workload and the investments already in place, rather than with a decision to move storage.

For an established lake, keeping ADLS Gen2 and exposing selected data to Fabric may be the simplest path. For a new Fabric-centric workload, storing data directly in OneLake may reduce operational work. The useful unit of decision is a layer or data product, not the entire estate.

Start with the operating model

What ADLS Gen2 is designed for

ADLS Gen2 brings analytics capabilities to Azure Blob Storage through a hierarchical namespace. In this PaaS model, Azure runs the storage service while your team manages accounts, containers, networking, lifecycle policies and access through Azure role-based access control and POSIX-style ACLs. It can hold any file format, giving teams flexibility to choose how different engines process the data.

That flexibility is valuable when ingestion pipelines and multiple analytics engines already share a lake. Azure Databricks, Azure Machine Learning and other compatible tools can work with the data where it is. The architectural responsibility sits with the team: bringing storage, processing, discovery and governance together into an operating platform.

OneLake: Fabric-managed data lake

Built on ADLS Gen2, OneLake is created automatically for a Fabric tenant. Microsoft manages the underlying storage infrastructure, while teams organize and secure data through workspaces and items such as Lakehouse’s. This SaaS model shifts the day-to-day focus from storage accounts to analytics data products. OneLake stores physical data and can also reference data elsewhere through shortcuts.

The benefit becomes tangible in the way Fabric workloads work together. Spark notebooks prepare Lakehouse data, SQL analytics experiences expose supported tables, and Power BI can consume suitable tables through Direct Lake. A Lakehouse’s Files area can hold any file format, while its Tables area serves supported table formats and engines, including Delta and supported Iceberg tables through metadata virtualization. Shared storage simplifies integration, while each engine still has its own supported capabilities.

The architectural relationship

The storage hierarchy is straightforward: tenant, workspace, then data item. Within a Lakehouse, Tables and Files support different consumption needs. Domains group workspaces for organizational ownership and governance, rather than adding a folder level to the storage path. A simple example is:

Scope	Illustrative organization
Fabric tenant	OneLake namespace
Workspace	Reporting workspace containing a Sales Lakehouse
Lakehouse Tables	Local Delta tables, or a shortcut to supported Gold tables in ADLS Gen2
Lakehouse Files	Files and folders, including supported external shortcuts

What changes for analytics consumers

Shortcuts: Bring existing data into reach

A plain shortcut references an internal or external storage location without requiring an ingestion copy. If curated Delta tables already live in ADLS Gen2, a supported Lakehouse Tables shortcut can make them available to Fabric consumers. Files shortcuts expose files instead; making those files into queryable tables is a separate step. This provides an incremental adoption path, although downstream workloads may still cache or materialize data.

The access design deserves as much attention as the data path. An ADLS shortcut uses the source connection’s configured credentials alongside Fabric permission checks. It does not automatically carry each reader’s source ACLs into Fabric. Before sharing it, I would check least privilege, network reachability and table compatibility with the identities and workloads that will use it.

Shortcut transformations address a different need: preparing data for consumption. Supported CSV, Parquet and JSON transformations use Fabric Spark to copy and convert source content into managed Delta output and keep it synchronized. That can simplify preparation, but it introduces compute, output storage and synchronization to operate. Choose a plain shortcut when referencing the existing data is enough; choose a transformation when the output itself adds value.

Direct Lake: Simplify the BI data path

Direct Lake loads requested Delta-table columns into VertiPaq memory without requiring a full Import-mode refresh. Metadata refresh, also called framing, still determines the data version available to the model. The practical benefit is a simpler BI data path for suitable workloads. Capacity, table optimization, security and model design still determine performance and freshness.

The underlying Delta data can remain in ADLS Gen2 when exposed through a supported OneLake Tables shortcut. That is the bridge to Direct Lake, rather than a direct connection from the semantic model to an ADLS path. I would assess the intended Direct Lake mode and security design, then test representative queries and freshness requirements before making performance commitments.

Governance: Discovery and enforcement both matter

Governance needs both business context and effective access control. ADLS Gen2 can use Microsoft Purview for discovery and governance alongside its storage permissions. In Fabric, the OneLake catalog brings discovery, metadata, lineage and endorsement into the analytics experience, while workspaces and items provide ownership context. Those capabilities help people find and understand data; the permissions on the actual access path determine who can use it.

I would compare security and lifecycle requirements against the specific design, rather than assume one platform always offers more control. Fabric supports tenant- and workspace-level private links and customer-managed keys with supported-item constraints. OneLake also supports lifecycle policies and hot, cool and cold tiers. The decision turns on whether the required workload, connector, encryption and network configuration work together, not simply whether a feature appears on a checklist.

AI readiness starts with usable data

Both approaches can support AI. ADLS Gen2 is a supported Azure Machine Learning datastore, so an established lake may already be a sound foundation. Fabric adds integrated preparation and consumption options that can be useful without relocating that lake. My starting question would be which capability the team needs and where the data is already well governed.

For example, AI-powered shortcut transformations can turn supported .txt files into Delta output containing summaries, translations, sentiment, redacted PII or extracted entities. A team could summarize eligible text sourced from ADLS and analyze the results in Fabric. This capability is in public preview, with regional and capacity requirements. Output quality and sensitive-data handling still need evaluation; automated redaction alone cannot establish compliance.

Interoperability depends on the integration path

Snowflake illustrates why the integration path matters. Fabric Mirroring can replicate Snowflake-managed table data into OneLake. For Iceberg tables in supported object storage, shortcuts and metadata virtualization provide another route; the Snowflake Iceberg mirroring path uses metadata mirroring and storage shortcuts. These are different architectural choices, with different implications for copies, freshness and ownership.

Snowflake on Azure can also write Iceberg tables to OneLake in supported configurations. That integration requires public connectivity, does not support private-link workspaces, and has same-region prerequisites for writes. Azure Databricks can read and write OneLake through authenticated ABFS paths, with compute and authentication requirements. Open formats help, but the chosen engine and network design still need to fit the supported integration.

Make the choice layer by layer

An existing Databricks lake: should Bronze and Silver move?

Consider a team with reliable Azure Databricks pipelines and Bronze, Silver and Gold layers in ADLS Gen2 that now wants Fabric reporting. My recommendation would be to keep the working Bronze and Silver layers unless there is a separate business or technical reason to move them. For Gold, there are two useful starting points.

Keep physical Gold in ADLS Gen2 and expose supported Delta tables through Lakehouse Tables shortcuts. Gold becomes available in Fabric without physical move. I would start here when existing storage contracts, authoritative writers and non-Fabric consumers need to remain intact.

Alternatively, write or materialize selected Gold data directly in OneLake when Fabric ownership or service-level requirements justify changing the pipeline. Databricks can support this pattern through authenticated ABFS access with the appropriate compute configuration. Moving selected Gold products can be a focused decision, independent of any future Bronze or Silver migration.

For either pattern, agree on the authoritative writer, table compatibility, effective permissions, freshness targets, monitoring and total cost. A storage shortcut does not transfer Unity Catalog authorization, so test the security model with representative readers. These responsibilities matter more than whether the diagram shows one lake or two.

Count the governance and semantic-model work

I would also count the work above storage. A team may maintain security rules and semantic logic in Databricks, then recreate them for Fabric reporting. Repeating policies, metrics, relationships and business definitions creates maintenance and testing work, with a risk of drift whenever requirements change. Reducing that duplication can be a stronger reason to adopt a shared consumption layer than saving a data copy.

A useful pattern is to use Databricks for data preparation, make governed Gold data available through OneLake, and use Fabric as the shared consumption layer. OneLake security can centralize data-access policies across supported Fabric engines and access paths. Separately, a deliberately shared Power BI semantic model lets reports and compatible clients reuse measures, relationships and business definitions instead of rebuilding them. These are two distinct opportunities to reduce downstream work: common access control and an explicitly reusable semantic layer, not benefits that storage alone creates.

That semantic model reuse does not require moving your Gold layer: supported shortcuts can expose ADLS data, and shared Power BI models can also use other supported sources. OneLake does not automatically unify Unity Catalog policies nor give every Databricks or Spark workload the same semantic layer. Where Unity Catalog remains authoritative, I would evaluate supported integrations and ask: where are policies and metrics defined, which engine enforces them under which identity, which consumers reuse the model, and who maintains and tests necessary differences?

A practical guide

Your situation	Starting point	What to validate
Successful multi-engine lake; established account/API dependencies	Keep ADLS Gen2 as an authoritative store	Existing contracts, source security and the specific Fabric access path
New Fabric-centric analytics and shared Fabric items	Use native OneLake storage	Ownership, supported workloads, capacity, security and lifecycle requirements
Mixed estate; selective Fabric adoption or duplicated governance and model logic	Combine ADLS Gen2 and OneLake; assess a shared consumption layer	See the mixed-estate checklist below.
A required security or storage feature determines the design	Compare the exact supported configurations	Connector and workload limits, private networking, CMK item coverage, retention and total cost

For a mixed estate, validate:

Data access: shortcut compatibility versus materialization.
Write ownership: the authoritative writer.
Operations: freshness targets and monitoring.
Security: effective identities and policy enforcement.
Semantics: reuse of a shared semantic model.
Governance ownership: who maintains and tests necessary cross-platform differences.

Coexistence as a practical starting point

Use ADLS Gen2 where its customer-managed storage model and existing integrations serve the workload well. Use native OneLake where Fabric’s managed model and shared analytics experiences provide a clear benefit. For an existing estate adopting Fabric, I would usually start by connecting the two and move data only where the benefit is clear. Coexistence can be a transition or a lasting design; consolidation should earn its place through workload requirements and measurable value.

Testing Fabric Runtime 2.0 Before It Becomes the Default

FarhanJeelani — Tue, 29 Sep 2026 14:40:03 GMT

Why you should test this week, not next month

Runtime 2.0 has been GA since August, but it is still opt-in. According to the Runtime 2.0 docs, Microsoft plans to make it the default for new workspaces and new Environment items in late September 2026. That is right now.

At the same time, Runtime 1.3 (Spark 3.5) is listed with an end-of-support date of 30 September 2026. It moves into Long Term Support from 1 October for six months, through March 2027 (lifecycle page). So you have a runway, but it is not long.

In my trainings, the question I get most is: "Will my notebooks break when this flips?" My honest answer: most will run fine, a few will fail loudly, and a small number will return different results without failing. That last group is the reason to test. This post walks through the exact method I use to test in isolation, without touching production.

What you need: a Fabric capacity (F2 or trial is fine), Contributor or higher on a workspace, and 30–45 minutes.

What actually changes between 1.3 and 2.0

Every core component moves a major or minor version. Versions below are from the official Runtime 1.3 and Runtime 2.0 pages.

Component	Runtime 1.3	Runtime 2.0	What to watch
Apache Spark	3.5	4.1	ANSI SQL mode is on by default in Spark 4
Delta Lake	3.2	4.2	New 4.x table features can break reads from other Fabric engines
Python	3.11	3.13	Pinned pip packages may have no 3.13 wheel
Java	11	21	Custom JARs built for old JDKs
Scala	2.12.17	2.13	Any Scala JAR compiled for 2.12 will not load
OS	Azure Linux 2.0	Azure Linux 3.0	Native libraries in custom packages
R	4.4.1	4.5.2	SparkR is deprecated in Spark 4.x

The four areas that cause real trouble in my experience:

ANSI mode. In Spark 3.5, CAST('abc' AS INT) quietly returns NULL. In Spark 4.x it throws an error. The same applies to integer overflow and invalid dates. Pipelines that relied on silent nulls now fail, which is actually good, but you need to know before a 2 a.m. run does.
Scala 2.13 / Java 21. If your Environment has custom JARs, they must be rebuilt. There is no workaround.
Python library pins. A requirements.txt or Environment library list pinned for Python 3.11 is the most common failure at publish time.
Delta 4.x features. The docs are explicit: Delta Lake 4.x-specific features are experimental and only work in Spark experiences. If the SQL analytics endpoint, Power BI Direct Lake or the Warehouse read the same table, do not enable them. Check Delta Lake interoperability first.

The good news: Runtime 2.0 ships with the Native Execution Engine, which Microsoft benchmarks at up to 6x faster than open-source Spark on TPC-DS, with no code change.

The safe test method: a side-by-side Environment

The trick is simple. Never change the runtime at workspace level first. An Environment item pinned to 2.0 overrides the workspace default, but only for the notebooks you attach to it. Production keeps running on 1.3 while you test the same code on 2.0.

Step 1: Create a test workspace

Create a workspace such as WS-Runtime2-Test on the same capacity. If your production workspace is Git-connected, branch out to it, or simply export the 3–5 notebooks that matter most. Add a shortcut to the production lakehouse tables you read from, so you test against real data without copying it.

Step 2: Create two Environment items

Create env_rt13 and env_rt20 in the test workspace. Two environments, same libraries, only the runtime differs. That is what makes the comparison fair.

New item → Environment, name it env_rt20.
In the Runtime dropdown, select 2.0 (Spark 4.1, Delta 4.2).
Under Spark compute → Acceleration, turn on Native Execution Engine.
Copy your production library list into Public libraries (and custom JARs/wheels if you use them).
Save, then Publish. Publishing is where library conflicts surface, so watch this step.
Repeat for env_rt13 with 1.3 (Spark 3.5, Delta 3.2) and the same libraries.

Step 3: Attach and run

Open a copy of each notebook, and in the environment dropdown on the notebook ribbon, pick env_rt20. Run it. Then run the same notebook attached to env_rt13. A Spark job definition works the same way through its settings.

Optional: the early access release channel

There is a second mechanism worth knowing about for later. Release channels (preview) let you test the next set of updates within a runtime before they become default. You set it in the Environment's Spark properties:

spark.fabric.pools.skipStarterPools=true spark.computeConf.runtime.releaseChannel=earlyAccess

Early access does not use the Starter Pool, so session start is slower, and the setting is fixed for the lifetime of a session. Billing is the same. Use it after your 2.0 migration to catch library updates early.

The test notebook I use in class

Four cells. Run the whole notebook once on env_rt13 and once on env_rt20. It takes about 10 minutes, most of it session start.

Cell 1: Prove which runtime you are on

Don't trust the dropdown; print it. This also gives you the VHD image name that support will ask for if you raise a ticket.

import sys from importlib.metadata import version, PackageNotFoundError jvm = spark.sparkContext._jvm def pkg(name): try: return version(name) except PackageNotFoundError: return "not installed" info = { "spark": spark.version, "python": sys.version.split()[0], "java": jvm.System.getProperty("java.version"), "scala": jvm.scala.util.Properties.versionNumberString(), "pandas": pkg("pandas"), "native engine": spark.conf.get("spark.native.enabled", "not set"), "ansi mode": spark.conf.get("spark.sql.ansi.enabled"), "vhd image": spark.conf.get("spark.synapse.vhd.name", ""), } for k, v in info.items(): print(f"{k:<15} {v}")

Cell 2: The ANSI probe

This is the cell that makes the room go quiet in a workshop. Same SQL, different behaviour.

probes = { "bad cast": "SELECT CAST('abc' AS INT) AS v", "int overflow": "SELECT CAST(2147483647 AS INT) + 1 AS v", "bad date": "SELECT CAST('2026-02-30' AS DATE) AS v", "div by zero": "SELECT 1/0 AS v", } for name, sql in probes.items(): try: print(f"{name:<13} -> {spark.sql(sql).first()['v']}") except Exception as e: print(f"{name:<13} -> ERROR: {type(e).__name__}")

On 1.3 you get None, -2147483648, None, None. Four silent problems. On 2.0 with default settings, all four raise errors. Now search your code for CAST(, arithmetic on IDs and date parsing from strings.

Cell 3: Regression check on your real logic

Paste one production transformation, write the result from each runtime, and compare row by row.

tag = "rt20" if spark.version.startswith("4") else "rt13" result_df = spark.sql(""" -- paste one real production query here SELECT ... """) result_df.write.mode("overwrite").format("delta").saveAsTable(f"regress_{tag}")

After both runs, compare from either environment:

a = spark.table("regress_rt13") b = spark.table("regress_rt20") print("row counts :", a.count(), b.count()) print("only in 1.3 :", a.exceptAll(b).count()) print("only in 2.0 :", b.exceptAll(a).count())

Zero and zero is what you want. If you get differences on float columns, round them before comparing; tiny floating point drift is not a bug.

Cell 4: Make sure other engines can still read the table

A table written from Runtime 2.0 must stay readable by the SQL analytics endpoint and Direct Lake. Check the protocol:

(spark.sql("DESCRIBE DETAIL regress_rt20") .select("minReaderVersion", "minWriterVersion", "tableFeatures") .show(truncate=False))

Compare it with regress_rt13. They should match. Then open the lakehouse's SQL analytics endpoint and run SELECT TOP 10 * FROM regress_rt20. If that works, your downstream reports are safe.

Finally, note the duration of each run from the Monitor hub. With the Native Execution Engine on, 2.0 is often faster, but measure it on your workload rather than quoting benchmarks.

Fixes, rollout plan and checklist

These are the fixes I reach for when the test fails.

Symptom	Likely cause	Fix
Environment publish fails	Library pinned for Python 3.11	Upgrade the pin to a version with a 3.13 wheel, or remove the pin
NoClassDefFoundError / scala. errors	JAR compiled for Scala 2.12	Rebuild the JAR for Scala 2.13 and Java 21
CAST_INVALID_INPUT, ARITHMETIC_OVERFLOW, DIVIDE_BY_ZERO	ANSI mode	Use try_cast, try_divide, try_add; clean the input
SQL endpoint can't read a table	Delta 4.x feature enabled on a shared table	Don't enable 4.x-only features on tables other engines read
Results differ, no error	Changed default behaviour	Diff with Cell 3, then read the Spark SQL migration guide

On ANSI mode: you can set spark.sql.ansi.enabled=false in the Environment to get the old behaviour back. I use it only as a temporary bridge, never as the fix. Silent nulls are exactly the data quality issue ANSI mode exists to catch.

My rollout plan

test workspace, two Environments, run the four cells on your top 5 notebooks.
fix libraries and ANSI failures; rerun until Cell 3 shows zero differences.
attach env_rt20 to production notebooks one pipeline at a time. Keep env_rt13 so rollback is one dropdown change.
once everything runs on 2.0, change the workspace default in Workspace settings → Data Engineering/Science → Spark settings → Environment → Runtime version.
Before March 2027: nothing left on 1.3 when LTS ends.

Checklist

Test workspace created, production untouched
env_rt13 and env_rt20 published with identical libraries
Cell 1 confirms Spark 4.1 / Python 3.13 / Java 21
ANSI probe run and code searched for risky casts
Cell 3 regression shows zero differences
SQL analytics endpoint reads tables written by 2.0
Custom JARs rebuilt for Scala 2.13
Workspace default switched last

If you do only one thing from this post, run Cell 2 on your busiest notebook today.

Sources

Microsoft Fabric Data Agents: Setup, Testing, and Publishing

uzuntasgokberk — Mon, 28 Sep 2026 18:55:42 GMT

In this walkthrough, I configured Microsoft Fabric Data Agents to answer questions against business data using natural-language prompts. The focus was on selecting data sources, guiding agent behavior with instructions and example queries, testing responses, and publishing the completed agent.

You can watch the video in English with English subtitles. Please check the subtitle settings on YouTube.

A data agent can be useful when company data is distributed across different sources and users need more consistent answers to recurring questions, such as questions based on sales data.

Key Takeaways

I added data sources such as Eventhouse, Semantic Models, and Lakehouses to a Fabric Data Agent.

I selected the tables the agent could read and observed a consistency warning when selecting more tables.

The agent translated natural-language questions into the query language appropriate for the connected source.

I used data source interactions, example queries, and agent instructions to influence responses.

I tested custom instructions before publishing the agent.

I also showed a Power BI Copilot scenario where a Data Agent could be referenced without opening the agent directly.

What I Configured in Microsoft Fabric Data Agents

I started in a workspace with a Fabric license and opened a Data Agent I had previously created as an example. The interface shown at the beginning was the interface from an earlier video, before I later shared a short excerpt of updates.

The first important step was opening the Data Source option. From there, I could add an Eventhouse, Semantic Models, or a Lakehouse. In my example, I added four Lakehouses.

After adding the sources, I selected the tables that I wanted the Data Agent to read. When I selected more tables, the agent displayed a warning that results might not be consistent. This is an important configuration point: expanding the number of available tables can give the agent more data to work with, but the interface warns that consistency may be affected.

Once the sources were available, I could either start with sample questions or enter my own prompt. The purpose was to ask questions in natural language and receive an answer based on the selected business data.

How Microsoft Fabric Data Agents Translate Questions

One of the key behaviors I demonstrated was how Microsoft Fabric Data Agents convert a natural-language request into a query language that matches the connected data source.

For the sources discussed in the video, the agent uses different query approaches:

Eventhouse: KQL
Semantic Models: DAX
Lakehouse or Warehouse: SQL

This means I could begin with a plain-language prompt while the Data Agent generated the source-specific query needed to retrieve a response. The exact query language depends on the type of data source attached to the agent.

Using Data Source Interactions and Example Queries

Next, I moved into the configuration areas that help shape how a Data Agent responds. In the Data Source Interactions section, I entered instructions related to a specific data source. This was where I could provide guidance on how the agent should work with that source.

I also used Example Queries, which are sample questions connected to a desired query behavior. My goal was to ensure that, when a particular question was asked, the specified SQL query would run consistently.

I compared this behavior to the Verified Answers concept for a Power BI visual. In this scenario, the sample question acts as a controlled way to steer the agent toward the expected query and output.

The available setup differs by source type:

For a Lakehouse or Warehouse, SQL appears in this configuration flow.
For an Eventhouse, the agent responds through KQL.
For the Power BI side, I mentioned preparing the data for AI so that the Data Agent can automatically detect it.

This is where example queries can be useful for recurring business questions. Rather than relying only on an open-ended prompt, I can add known questions and define the behavior I want for them.

Testing Agent Instructions Before Publishing

In addition to source-level settings, I configured Agent Instructions. These are broader instructions that tell the agent how to approach its responses.

To test this behavior, I added a simple instruction asking the agent to always address me as “boss.” I then entered a sample prompt into the agent. In this test, the response included the requested word while also returning an answer based on the prompt.

I entered another prompt that matched one of the example queries I had written. The agent used that example-query path and returned the output in the same format as the sample query. I repeated this with another example to check whether the configured behavior appeared consistently in these tests.

This was a short functional check rather than a comprehensive evaluation. I tested the instruction and example-query behavior shown in the agent interface, but I did not cover broader accuracy measurement, performance testing, or governance controls.

Publishing a Microsoft Fabric Data Agent

After I was satisfied with the agent, I selected Publish. Before publishing, I could add a description to help explain what the Data Agent was for.

Once published, the interface displayed the implementation and output for the agent. Publishing is the point where the configured Data Agent becomes ready to use beyond the editing and test flow I demonstrated.

I also included a short September 2026 excerpt showing an updated agent experience. In that excerpt, I showed options to view existing data sources through Add Data, add tools, and work with visualizations. I also mentioned a capability that can help generate instructions more quickly, rather than manually writing every example query, data-source instruction, and agent instruction.

The updated experience also included a Test Data Agent area for checking the agent before publishing. I referenced Runtime as part of working with the newer updates shown in the interface.

Using a Data Agent from Standalone Power BI Copilot

Finally, I showed a scenario involving Standalone Power BI Copilot. The Data Agent was offline at the time of the recording, but I included the previously available scenario to demonstrate the intended interaction.

In this flow, I used the @ symbol to reference one of my Data Agents. I had created an agent named Copilot Data Agent. The goal was to ask a question in Standalone Power BI Copilot and receive an answer from the Data Agent without first navigating directly into the agent itself.

This illustrates how Microsoft Fabric Data Agents can be part of a broader Copilot-based experience, where users ask questions from a conversational interface while the agent works with its configured data sources.

Frequently Asked Questions

Which data sources can I add to a Microsoft Fabric Data Agent?

In the demonstrated interface, I could add an Eventhouse, Semantic Models, and Lakehouses. I then selected the tables that the agent could read.

Which languages can a Fabric Data Agent use for queries?

The video showed KQL for Eventhouse, DAX for Semantic Models, and SQL for Lakehouse or Warehouse sources.

Conclusion

In this walkthrough, I configured Microsoft Fabric Data Agents by connecting data sources, selecting tables, adding source instructions, defining example queries, and testing agent instructions. I then published the agent and showed how it could be referenced from Standalone Power BI Copilot.

The main takeaway is that Microsoft Fabric Data Agents provide a configuration-based way to connect natural-language questions with business data sources and source-specific query languages. If you are working with Fabric data sources, start by defining a focused set of tables and recurring questions, then test the responses before publishing.

Connect Fabric Data Agents to Copilot Studio with MCP (+ Assistants API Migration Fix)

FarhanJeelani — Thu, 24 Sep 2026 16:01:53 GMT

Business users want answers where they already work, in Teams and Microsoft 365 Copilot, and those answers need to come from trusted, governed data. Fabric data agents in Copilot Studio do exactly that, and the integration is now generally available.

There is also a deadline you may already have passed. If you connected to Fabric data agents through the OpenAI Assistants API, your integration has likely stopped working. This guide covers both the setup and the fix.

What's new

Fabric data agents now connect to Copilot Studio through a new tool-based experience: you select Add a tool, search for Fabric, and choose Fabric IQ Data MCP. Your Copilot Studio agent then calls the Fabric data agent like any other tool. Microsoft Community

Three points matter for architects:

Data stays governed. The Fabric data agent keeps running in Fabric, respects permissions on the underlying data sources, and returns answers grounded in governed enterprise data. Microsoft Community
The orchestrator decides. The orchestrator chooses when to use enterprise data and when to use its other knowledge sources and tools. Microsoft Community
You can combine agents. You can pair several Fabric data agents with other business systems to build richer solutions. A sales agent could query a sales data agent, a finance data agent and your CRM connector in a single conversation. Microsoft Community

Behind this, Copilot Studio also introduced a redesigned authoring surface and runtime powered by the GitHub Copilot harness. Microsoft Community

Why MCP matters

Model Context Protocol (MCP) is becoming the standard way agents connect to tools and data. By exposing Fabric data agents through MCP, Microsoft makes them a reusable building block instead of a one-off integration. The same move is happening elsewhere: Fabric data agents in Microsoft Foundry are now easier to connect and monitor, and that integration is moving to MCP too. Microsoft Community

You build a data agent once in Fabric, and Copilot Studio, Foundry and Microsoft 365 Copilot can all use it.

Prerequisites

Before you start, check that you have:

A Microsoft Fabric workspace on Fabric capacity
Data ready to use: a lakehouse, warehouse, semantic model or similar source with clean, well-named tables
Permission to create items in the Fabric workspace
Access to Copilot Studio with permission to build and publish agents
Microsoft 365 Copilot or Teams, depending on where you want to publish

Tip: the quality of your data agent depends on the quality of your data. Clear table and column names and descriptions make a big difference to answer accuracy.

Step 1: Build and test your Fabric data agent

In your Fabric workspace, create a new data agent item, then:

Add data sources. Connect the lakehouse, warehouse or semantic model the agent should answer from. Start narrow, since one domain per agent works best.
Write agent instructions. Explain the business context, key terms and rules. For example: "Revenue means net revenue after returns. Fiscal year starts in April. Always exclude internal test accounts."
Add example questions and queries. Give sample questions with the correct query logic, so the agent learns your patterns.
Test thoroughly. Ask the questions your business users actually ask and compare the answers with your certified reports.
Publish the agent once the answers are reliable.

Step 2: Add the data agent to Copilot Studio

This is the new MCP-based flow. Create or open an agent in Copilot Studio, go to Tools, select Add a tool, search for Fabric, and add Fabric IQ Data MCP. Microsoft Community

Then:

Select the Fabric data agent you published in Step 1.
Give the tool a clear description, such as "Answers questions about sales, revenue and pipeline from governed Fabric data." The orchestrator reads this to decide when to call the tool, so be specific.
In your Copilot Studio agent's instructions, tell it when to use the tool, for example: "For any question about sales numbers or targets, use the Sales Data tool."

Step 3: Test in Copilot Studio

Test your agent in Preview before publishing. Check that: Microsoft Community

Data questions trigger the Fabric tool, and general questions don't
Answers match the results from Fabric directly
A user without access to certain data gets no answer from it, since the data agent respects permissions on the underlying sources
Follow-up questions ("and what about last quarter?") work as expected

Step 4: Publish to Teams or Microsoft 365 Copilot

When you're happy with the results, publish to Microsoft Teams or Microsoft 365 Copilot. Business users can now ask data questions in the tools they use every day, without opening a report. Microsoft Community

Roll out to a small pilot group first, collect feedback, and improve the data agent's instructions before rolling out widely.

The Assistants API retirement: what broke and how to fix it

This part matters if you built programmatic integrations.

What happened: The OpenAI Assistants API, which powered the orchestration layer of the Fabric data agent, was scheduled to be shut down by OpenAI on August 26, 2026. After that date, direct calls to the Assistants API stop working. Microsoft Community

Who is affected: anyone who connects to a Fabric data agent programmatically through the Assistants API needs to migrate to the data agent MCP endpoint. Typical examples are custom apps, scripts, or third-party tools calling the data agent directly. Microsoft Community

Who is mostly fine: SDK and Fabric portal users need little or no action, because Microsoft migrates those experiences internally, although conversation history may reset once. Microsoft Community

What stays the same: existing agent data sources, instructions and tools remain unchanged. You don't need to rebuild your agents, only change how you connect to them. Microsoft Community

Migration checklist

Find your integrations. Search your code and configs for Assistants API calls that target Fabric data agents. Check custom apps, automation scripts, Azure Functions and any third-party tools.
Check for failures since late August. Errors or silent failures after August 26 are a strong sign an integration is affected.
Move to the MCP endpoint. Update each integration to connect through the data agent's MCP endpoint instead. Follow Microsoft's guide, "Prepare your Fabric Data Agent integrations for Assistants API retirement," for the current details.
Consider Copilot Studio instead of custom code. If a custom app only exists to put a chat interface on a data agent, the Copilot Studio route above may replace it with less code to maintain.
Tell users about history resets. If you use the portal or SDK, let users know earlier conversations may disappear once.
Retest. Run your standard question set again after migrating to confirm the answers haven't changed.

Best practices for production

One domain per data agent. Separate agents for sales, finance and operations are easier to test, govern and improve than one agent that tries to do everything.
Write tool descriptions carefully. In Copilot Studio, the tool description is how the orchestrator decides what to call. Vague descriptions lead to wrong tool choices.
Treat instructions like code. Version them, review changes, and retest after every edit.
Keep a test question set. Keep 20 to 30 real questions with expected answers, and run them after any change to data, instructions or model.
Watch permissions. Answers respect underlying permissions, so a mistake in data security becomes a mistake in agent answers. Review access regularly.
Start small and grow. Pilot with one team, measure how often answers are right, then expand.

The bottom line

Connecting Fabric data agents to Copilot Studio through MCP turns your governed Fabric data into a reusable tool that any agent can use, and it puts trusted answers directly into Teams and Microsoft 365 Copilot. The setup takes minutes once your data agent is solid.

If you built integrations on the Assistants API, make migration your first job this week. Your agents, data sources and instructions don't need to change, only the connection. After that, you're on the platform Microsoft is building its agent strategy on.

Fabric Governance Patterns: The First Step to Trusted Scale

rawatsudhirfabr — Thu, 24 Sep 2026 15:59:32 GMT

A large enterprise had already embraced self-service analytics. Business teams were building reports, connecting to data, publishing dashboards, and moving fast.

At first, this looked like success. More people had access to data. More teams could answer their own questions. More reports were being created without waiting for a central team. But over time, a different pattern started to appear.

The same metric had more than one definition. Reports were difficult to find. Some workspaces had too many administrators. Critical assets did not always have clear owners. Development and production were not consistently separated. Data lineage was hard to trace. Capacity usage needed closer oversight. And as AI and Copilot entered the conversation, the question became even sharper:

Can we scale innovation without scaling risk?

That is where Fabric governance begins. Not as a blocker. Not as another layer of bureaucracy. But as the first step toward trusted, reusable, secure, and AI-ready data.

The first lesson: governance is not a document, it is an operating model

One thing I have learned is that governance does not succeed because a framework exists. It succeeds when people know how to use it.

A governance document can describe roles, standards, and controls, but the operating model is what makes those ideas practical. It defines who decides, who approves, who owns, who monitors, and who acts when something is not working.

For Fabric, this usually means moving toward a hub-and-spoke model. The hub defines enterprise guardrails, platform standards, monitoring, training, and governance consistency. The spokes own business context, local adoption, workspace execution, semantic model quality, and day-to-day compliance.

That shift matters because Fabric is not only a reporting platform. It brings together data engineering, Power BI, OneLake, semantic models, pipelines, security, metadata, AI, and governance capabilities. If the operating model is unclear, the platform can grow faster than accountability.

The first step is therefore simple to state but important to execute:

Define who owns what before scaling Fabric further.

What happens if the first step is skipped

Without governance at the start, self-service becomes self-risk.

I have seen this pattern many times: teams do not intend to create risk. They are trying to solve real business problems quickly. A report is needed urgently. A dataset is copied because it is faster. A semantic model is rebuilt because the existing one is hard to find. A workspace is created because the team needs somewhere to collaborate. Each decision makes sense locally. But collectively, the platform becomes harder to govern.

Current pattern	Risk created	Governance direction
Decentralized citizen development with limited oversight	Report proliferation and reduced trust	Hub-and-spoke governance model
Users self-manage workspaces, datasets, refreshes, and models	Source-of-truth confusion	Asset lifecycle ownership
Too many workspace admins and inconsistent access approval	Privacy, exposure, and compliance risk	Least-privilege access controls
Reports and metrics are hard to discover	Duplication and poor reuse	Metadata, catalog, and marketplace discovery
Semantic models are rebuilt by each analyst	Conflicting business logic	Shared, reusable semantic models
Reports are published directly to production	Quality and change risk	Dev/Test/Prod deployment path
Unoptimized refreshes and transformations	Capacity pressure and performance issues	Central reusable pipelines and monitoring
AI features are enabled without readiness checks	Misuse, poor prompts, weak auditability	Responsible AI, metadata, training, and approval controls

The biggest risk is not that people build too much. The biggest risk is that no one can confidently say which assets are trusted, who owns them, and whether they are safe to reuse.

The story of the first step

Imagine a business user needs a new dashboard. In the old world, they might connect Power BI Desktop directly to Excel, SQL, SharePoint, or an API. They build a model, define their own business rules, publish to a workspace, and share the report. It works.

Until another team builds the same metric differently. Until the source changes. Until access becomes unclear. Until refreshes start consuming capacity. Until a critical executive report depends on an undocumented model.

In the governed Fabric world, the story changes. The user still gets the insight. But the path is safer:

The data is ingested through governed pipelines.
Data is curated in OneLake or an approved warehouse/lakehouse pattern.
Certified or strategic data products are reused where available.
Semantic models are owned, documented, and version controlled.
Reports move through development and production controls.
Metadata, lineage, access, and quality checks are captured.
Monitoring shows whether the platform is healthy and compliant.

The user experience should feel easier, not harder. Governance should make the trusted path the default path.

Personas: the people who make governance real

Fabric governance only works when roles are explicit. In my experience, this is where many governance initiatives either become practical or remain theoretical. If the role definitions are vague, everyone agrees governance is important, but no one knows who should approve, remediate, certify, or retire an asset.

Persona	Story role	Responsibility
Fabric Governance Lead	Sets the rules of the road	Defines policies, guardrails, governance forums, standards, and adoption KPIs
Fabric Technical Product Owner	Owns the platform roadmap	Manages feature enablement, Fabric roadmap, platform health, and Microsoft relationship
Fabric Architect	Designs the pattern	Defines architecture blueprints, workspace structure, integration patterns, and semantic model guidance
Fabric Product Owner / Spoke Owner	Owns business outcomes	Prioritizes BI demand, funds initiatives, approves adoption, and ensures business value
Steward	Runs governance day to day	Ensures standards, metadata, access, certification, lifecycle, and workspace guardrails are followed
Capacity Manager	Protects performance and cost	Monitors capacity usage, manages allocation, forecasts demand, and resolves overages
Monitoring Analyst	Turns governance into evidence	Builds governance scorecards, compliance metrics, adoption reporting, and exception triggers
Developers and report creators	Build governed assets	Use certified data, follow standards, document metadata, validate quality, and publish through controlled pipelines
End users	Consume trusted insights	Use certified reports and datasets, provide feedback, and avoid unmanaged duplication
Security, privacy, and responsible AI teams	Protect the enterprise	Advise on sensitive data, AI use, privacy, compliance, auditability, and risk controls

The important point is not the title. The important point is accountability. Every workspace, semantic model, data product, report, capacity, and AI-enabled experience needs an owner.

Governance patterns that turn intent into action

A practical Fabric governance framework should define patterns across ten areas:

Governance pattern	What it controls
Operating model pattern	Hub-and-spoke governance, community of practice, training, enablement
Capacity pattern	Dedicated capacity ownership, funding, sizing, production/development tagging
Workspace pattern	Workspace approval, ownership, production isolation, review and decommissioning
Copilot and AI pattern	Metadata readiness, training, responsible AI, logging, approval, monitoring
Data product pattern	Certified data products, shared lakehouses, curated ingestion, retention, exfiltration control
Semantic model pattern	Reusable models, ownership, glossary integration, lineage, change management
Access pattern	Least privilege, RBAC groups, RLS/OLS/CLS, masking, periodic access reviews
Monitoring pattern	Capacity health, platform alerts, governance scorecards, compliance KPIs
DevOps pattern	Dev/Test/Prod separation, deployment pipelines, validation, version control
Onboarding pattern	Role-based access, training, appropriate workspace roles, production read-only controls

This is where governance becomes operational. It moves from “we should govern Fabric” to “this is how every team works.”

A lightweight Power Platform intake can help

One of the most practical lessons I have learned is that governance needs an easy front door. If the process is unclear, people bypass it. If the process is too heavy, people avoid it. If approvals live only in email, governance becomes hard to track. A simple Power Platform intake can make the first step easier:

Request a new capacity.
Request a workspace.
Request access.
Request Copilot enablement.
Request certification.
Request exception approval.
Capture business owner, technical owner, data classification, lifecycle, and approval history.
Route the request to the right steward, product owner, platform owner, or security reviewer.

Closing thought

Fabric governance is not about creating more process. It is about creating confidence.

Confidence that business users are working with trusted data.
Confidence that platform teams can scale without losing control.
Confidence that security and compliance teams have visibility.
Confidence that AI and Copilot experiences are grounded in governed, reusable, well-understood data

The organizations that will get the most value from Microsoft Fabric will not be the ones that simply adopt it fastest. They will be the ones that create the right patterns early, so speed and trust can grow together.

Start small, define ownership, make the governed path easy, and let governance become the foundation for trusted scale.

Fabric Capacity Cockpit: pause, resume and track the cost of your F-SKU from one window

Hans-Georg_Puls — Thu, 24 Sep 2026 15:59:07 GMT

Why I built it

When my Fabric trial capacity expired, I moved my workspaces to a small pay-as-you-go F2. I use it for experiments and demos, not production, so the obvious way to keep costs down is to pause it whenever I'm not working with it.

In practice, the information I need for that is spread across several places:

Azure portal – capacity overview: status and pause/resume, but no cost
Azure Cost Management: cost, but no status
Fabric Capacity Metrics app: CU and storage usage, but neither cost nor status
Workspace settings: which workspace is assigned to which capacity

I wanted one small window that answers three questions at a glance (Is it running? What has it cost this month? Can I pause it now?) and does the pause/resume for me. The result is the Fabric Capacity Cockpit, now open source under the MIT license:

👉 https://github.com/Siebzehnundvier/FabricCockpit-GUI

What it does

Capacity picker: lists all Fabric capacities across all enabled subscriptions of the signed-in account. The list is cached, so the window is usable immediately while a fresh list loads in the background.
Capacity info: name, subscription, resource group, region, SKU, state, provisioning state and month-to-date cost of the selected capacity.
Cost split: the cost figure shows the OneLake storage share separately (e.g. 12.34 € (of which storage 0.87 €)), because that part keeps accruing while the capacity is paused.
Pause / Resume: each action asks for confirmation, then shows live progress with an elapsed-time counter until the target state is reached.
Auto-pause timer: choose 15, 30, 60 or 120 minutes. Two minutes before the deadline, a banner offers Pause now, Extend or Cancel.
Tray icon: its colour follows the capacity state (green active, amber paused, blue transitioning). The right-click menu offers Open cockpit / Resume / Pause / Exit. Closing the window only hides it to the tray.
Links: Azure capacity overview, cost analysis for the resource group, and your Capacity Metrics app.
Log: every action and error is logged with a timestamp and can be copied to the clipboard.

Under the hood

The cockpit is plain Windows PowerShell 5.1 with WinForms, so it needs no installer, no compiled binaries and no admin rights. All Azure access goes through the Azure CLI:

Capacity list, state, suspend and resume use the microsoft-fabric CLI extension (az fabric capacity ...). The cockpit installs the extension automatically on first start.
Pause and resume are sent with --no-wait, and the state is then polled every 5 seconds. Without --no-wait, the CLI blocks until the operation finishes, which would freeze the UI.
Cost comes from a Cost Management REST query at resource level, grouped by resource ID and meter. Stored-data meters count as storage; everything billed in CU counts as compute. Results are cached for 10 minutes, and throttling (HTTP 429) is handled with a retry.
Sign-in and tokens are handled entirely by the Azure CLI. The tool stores no credentials. Its only local data is a settings file in %APPDATA%\FabricCockpit plus cost cache files in %TEMP%. There is no telemetry and no update check.

I tested the state transitions (Paused → Resuming → Active and Active → Pausing → Paused) against Azure CLI 2.90.0 with microsoft-fabric 1.0.0b1.

Getting started

Install the Azure CLI and check that az --version works.
Download the release zip from GitHub and unblock it (right-click → Properties → Unblock), then extract it anywhere.
Double-click Start-Cockpit.cmd, click Sign in, and pick your capacity.

Permissions:

Viewing only: Reader on the capacity or resource group.
Pause / Resume: additionally suspend/action and resume/action on the capacity, e.g. Contributor.
Cost figure: Cost Management Reader. Without it, the field shows n/a and everything else still works.

Before you pause anything

The cockpit issues the same calls as the Azure portal, so the consequences are the same:

Pausing makes every workspace on the capacity unavailable. Scheduled refreshes and pipeline runs fail while it is paused.
Running work is aborted. The auto-pause timer does not check for running refreshes, notebooks or Spark jobs. Use it only on personal, dev or test capacities.
Resuming starts billing immediately.
OneLake storage is billed while paused.
Reservations are not detected. If your capacity is covered by a reservation, pausing saves nothing.
The cost figure is approximate. Azure cost data lags behind, typically can lag by a day or more. For invoices, use the portal.

Trial capacities and Power BI Premium P-SKUs are not Azure resources and therefore don't appear in the list.

How it was built

I built the cockpit together with an AI assistant and verified the behaviour on my own capacity. The code is plain text and deliberately simple, so please review it before you run it in an environment with stricter policies.

Feedback welcome

This is version 0.52, a tool for my own daily use that I think others with small pay-as-you-go capacities might find handy. Issues, ideas and pull requests on GitHub are very welcome. I'd also be interested to hear how you manage pause/resume and cost tracking for your own capacities: automation runbooks, Logic Apps, or something else entirely?

The Hidden Cost of the Data Swamp: Fabric Capacity Guardrails

Pragati11 — Wed, 23 Sep 2026 14:41:14 GMT

Introduction: The Double-Edged Sword of Fabric Flexibility

Microsoft Fabric has fundamentally revolutionized how enterprise data teams operate. By unifying data engineering, data science, data warehousing and Power BI into a single Software-as-a-Service (SaaS) platform, it breaks down silos and lets business users move at lightning speed.

However, this democratization of data comes with a massive hidden risk. When workspace creation is unchecked and pipelines run wild, speed quickly transforms into a costly “data swamp.” Unlike traditional infrastructure where compute and storage are rigidly partitioned, Fabric pools your processing power into Capacity Units (CUs). If you don’t govern those capacities, runaway queries, poorly optimized Spark jobs and unmanaged workspace sprawl will quietly drain your cloud budget before finance even sees the bill.

1. The Anatomy of a Fabric Cost Explosion

Cause-and-effect visualization demonstrating how unmonitored workspace sprawl and orphaned background pipelines drive sudden spikes in Capacity Unit (CU) consumption.

How Cloud Costs Spiral Out of Control

In a consumption-based SaaS model, expenses aren’t driven just by storage – they are driven by active compute. Without strict guardrails, organizations frequently run into three major financial traps:

The “Noisy Neighbor” Problem: A single inefficient, ad-hoc DAX query or an unoptimized PySpark notebook running in a shared capacity can consume 100% of the processing power, throttling critical executive dashboards and slowing down the entire organization.
Orphaned Pipelines & Infinite Loops: Automated data factories and pipelines left running by developers who forgot to turn them off after testing, continuously burning CUs over weekends and holidays.
Unmonitored Sprawl: Business units spinning up independent workspaces without designated capacity assignments, causing unpredictable spikes when heavy reporting cycles (like month-end financial closing) collide.

2. Real-World Challenges: Why Monitoring is Hard

Real-world operational challenges illustrated through capacity metrics app warnings and automated performance bottleneck notifications.

Blind Spots in the Enterprise

Managing compute costs in Fabric is uniquely challenging because responsibility is decentralized.

Decentralized Ownership: When business analysts build and manage their own workspaces, they rarely think about the underlying cluster compute footprint. They care about insights, not core optimization.
The Delayed Billing Shock: Because usage metrics update dynamically, teams often don’t realize a workload has choked the capacity until after the performance degradation has already impacted end users.
Lack of Throttling Transparency: Without proactive alert mechanisms, teams are caught entirely off guard when Fabric automatically pauses or throttles operations due to capacity exhaustion.

3. Best Practices for Fabric FinOps and Capacity Governance

Actionable three-step checklist highlighting core FinOps best practices for establishing sustainable capacity guardrails in Fabric.

Taking Back Control of Your Compute Budget

To keep your cloud bill predictable and your performance lightning-fast, you need to embed FinOps principles directly into your governance strategy.

Isolate Workloads by Capacity Assignment: Never mix heavy data engineering Spark jobs with executive Power BI semantic models on the exact same capacity tier. Separate your development, test and production workloads to protect critical business assets.
Master the Fabric Capacity Metrics App: Make it a non-negotiable weekly habit for administrators to review the Microsoft Fabric Capacity Metrics app. Look closely at your Interactive vs. Background CU consumption spikes to pinpoint which specific semantic models or items are dragging down performance.
Set Up Proactive Alerting and Guardrails: Use Azure Monitor or built-in notification hooks to alert your administrators when capacity utilization hits dangerous thresholds (e.g., crossing 80% sustained usage), allowing you to intervene before automatic throttling kicks in.
Enforce Workspace Lifecycle Policies: Automatically archive or clean up abandoned workspaces and semantic models that haven’t been queried in over 90 days, saving both storage and compute metadata overhead.

4. Conclusion: Turning FinOps into a Competitive Advantage

The ultimate goal of Fabric FinOps and Governance – transforming an unpredictable data swamp into a secure, high-performance and economically sustainable analytics powerhouse.

Governance in Microsoft Fabric isn’t just about security clearance lists and data privacy – it is about economic sustainability. By treating your capacity units as a finite, precious corporate asset, you protect your cloud budget from unexpected waste.

When you combine robust workspace structuring, active Purview data labeling and diligent FinOps capacity monitoring, you turn your Fabric tenant from an unpredictable financial black hole into a predictable, high-performance enterprise analytics powerhouse.

What about you? How is your organization currently keeping runaway Capacity Unit (CU) spikes and “noisy neighbor” workloads under control in Microsoft Fabric? Drop your strategies or biggest cost-management challenges in the comments below!