Microsoft Fabric Security Policies: How They Strengthen Data Security and Governance?
In today’s data-driven world, governance isn’t just about compliance — it’s about enabling trust, consistency, and scale. As organizations adopt Microsoft Fabric to unify their analytics, the challenge of managing who can do what, where, and under which conditions become critical. That’s where Microsoft Fabric policies step in. Think of them as the rulebooks that let administrators define guardrails across workspaces, security groups, and sensitivity labels. Instead of relying on manual oversight, policies provide a centralized, automated way to enforce governance — ensuring that innovation doesn’t come at the cost of control. In this blog will walk you through the key concepts, the entry points for creating and activating policies, and the end-to-end lifecycle that makes governance in Fabric both powerful and practical. Prerequisites: 1) A Valid Fabric Trail / Capacity License 2) A Fabric workspace assigned to a Capacity/Fabric Trail. 3) Proper admin role to activate the policy a) Fabric administrator: Can create and activate policies for tenant-scope policies. b) Capacity administrator: Can create and activate policies for capacity-scope policies. 4) Contributor, Member, or Admin access to the workspace so that you can create a policy item type. 5) The Policies tenant setting is enabled in Admin settings 6) Valid Security groups in AAD. Step 1: Enable the Policies feature in the admin settings Go to fabric admin settings, search for polices(preview) and enable the setting and click on apply. Step 2: Creating Policy item Create a workspace ‘Fabric-Polices-Ws’ which was assigned to ‘Fabric Trail’ and I’m (SI Babu) who’s the capacity admin. Make sure this should be a secured, admin-managed workspace to store policies. Click on ‘New Item’ and select ‘Policy set (preview)’ under ‘Govern data’ In New policy set dialog, enter a unique and descriptive Name ‘Capacity_PolicySet_1’ for the policy set and choose ‘ A capacity’ under policy scope then Click on ‘Create’ Create another policy set ‘Tenent_Policy_set2’ and choose ‘ Fabric Tenant’ under policy scope then Click on ‘Create’ Two policy sets are created. Step 3: Activating the Policy set Open Capacity policy set and select the ‘Active’ from Status Select the capacity and click on ‘Assign + activate’ Policy activated successfully Open Tenant policy set and select the ‘Active’ from Status Click on ‘Activate’ the policy Policy activated successfully Step 4: Create / Add Policy Rules(Capacity Policy) Click on ‘OneLake catalog’ Click on ‘Govern’ à‘Polices(preview)’ àCapacitiesàExpand the ‘Users can create items’ and select ‘Advanced’ Here, Select Allowed, Blocked, or Advanced (Set by user) to change the policy behavior. If you select Advanced, the rules configuration is enabled. Add a new rule with good name, description and Select ‘item type’ in Condition type, operator is ‘Is any of’ the select ‘Lakehouse’ and ‘Dataflow gen2’ then click on ‘+Add condition’ Choose ‘workspace’ in condition type, operator is ‘Is any of’ selected two workspaces gen2’ then click on ‘+Add condition’ Choose ‘Security group’ in condition type, operator is ‘Is any of’ selected my ‘Data Analyst’ AD group. Click on Apply Note: In this Data Analyst group we have three developers are there Click on “Save’ Now Policy successfully added Step 5: Validate policy enforcement (Capacity Policy) Added ‘Data Analyst’ AD group to both the workspaces ‘FabricRTI Training’ & ‘PowerBI-Demo-WS’ Now I was opened another window with account(SP Babu) who’s part of AD group & you can see two workspaces Create a ‘Notebook’ Now when I click on “Create” it’s throwing an error as you seen in below screenshot. Means it enforced a policy Again, When I was trying to create ‘Warehouse’ it’s throwing the same error. I (SP Babu) was able to create ‘DataFlowGen2’. Admin enforced to create only this item I (SP Babu) was able to create ‘Lakehouse’. Admin enforced to create only this item Step 6: Create / Add Policy Rules(Tenant Policy) Create another policy should be at tenant level to restrict workspace Outbound access protection and inbound network rules and click ‘add condition’ Add “Security group’ as I have IT Infra Team and click Apply Note: In this IT Infra Team group we have three resources are there Security policy created successfully Step 7: Validate policy enforcement (Tenant Policy) Now SP Babu unable to edit ‘Outbound access’ it’s showing ‘This setting is managed by a tenant policy.’ SP Babu unable to edit ‘Inbound access’ it’s showing ‘This setting is managed by a tenant policy.’ You can deactivate or replace an active policy set from the workspace where the policy set is stored. Only an administrator of the policy set's admin scope can deactivate or replace it. Best practices for creating a policy: Use a secured, admin-managed workspace to store policies. Store all policies for the admin scopes managed by the same administrator or admin team in the same workspace. This approach provides one location for tenant- and capacity-scoped policies and simplifies access management, discovery, and lifecycle management. Use clear policy set and rule names that identify the admin scope and purpose. Use test users who are members and non-members of the configured security groups to validate both allow and deny outcomes before broader rollout. Limitations as of now: Fabric policies aren't currently supported in the following capacity regions: West Europe, North Europe, and West US. We can currently create policy sets only from the item creation experience in a workspace. Only workspaces assigned to a Fabric capacity can store policy sets. We can't change a policy set's admin scope after creation. Only one policy set can be active for an admin scope at a time. Policy and tenant-setting changes can take time to propagate. Conditions support Any of and None of in the current preview. We can't activate a policy set through Git. So, start small with pilot policies, monitor audit logs for adoption and Communicate changes to workspace owners Conclusion Microsoft Fabric Policies are still in preview, but they signal a future where governance is not a blocker — it’s an enabler of responsible innovation. Happy learning! — Inturi Suparna Babu [LinkedIn]I Let Fabric's New Data Engineering Agent Run My ETL: Here's What Happened
Last week I handed Microsoft Fabric's new Data Engineering Agent a messy ETL job that took me 11 hours to build by hand last quarter. It had a working, validated pipeline in front of me in under an hour. I still changed 3 of its 9 steps before I'd let it near anything important. That is the short version. The long version is more interesting, because the places where it went wrong tell you exactly how to use it well. Some context first. At FabCon Europe 2026 in Barcelona (September 28 to October 1), Microsoft announced the Data Engineering Agent in preview. It can plan and run migrations, ETL and performance work under guardrails that engineers define. It builds on technology from Osmos, the agentic data engineering startup Microsoft acquired in January 2026, and was demoed on stage as "Project Osmos". Copilot has been helping us write code for a while now. This is a different idea. You describe the outcome, and the agent does long-running, multi-step work on its own while you supervise. I wanted to know whether that holds up on a real, slightly ugly dataset rather than a keynote demo. So I tried it. What the Data Engineering Agent actually is The agent is an autonomous worker inside your Fabric workspace, not a chat assistant that suggests code. You give it a goal, and it does the multi-step work itself. Its roots are in Osmos, a Seattle startup whose "AI Data Engineer" already generated production-grade PySpark notebooks inside Fabric. Microsoft bought the company in January 2026 and folded the team into Fabric engineering. Based on Microsoft's FabCon session on Project Osmos, a typical run looks like this: Read the request. You describe the outcome, even loosely ("land the orders CSVs in a clean silver table"). Inspect the workspace. It looks at your existing code, Lakehouses and data layout before writing anything. Plan. It breaks the job into steps and can hand parts to sub-agents for long-running work. Build. It creates and updates notebooks and other Fabric items. Validate. It checks its own output so you can trust what was built. Hand back to you. A human reviews and approves. Microsoft is explicit that supervision stays part of the process. The headline use cases are migrations, ETL and performance tuning. I chose ETL, because that is where most of us spend our week. The test: a real pipeline, not a demo I picked a job I had already built by hand, so I could judge the agent's work line by line against my own. The data is sales orders from a mid-sized retail client . Every day their order system drops a CSV export into a Lakehouse folder. The business wants a clean orders table for analysts and a daily sales summary for a Power BI report. Item My setup Source data 14 daily CSV exports, 18.4 million rows, about 2.1 GB, in Files/raw/orders/ Target silver_orders (cleaned Delta table) and gold_daily_sales (aggregate by day, store and category) Known traps Duplicate order IDs from system retries, two date formats, a currency column with nulls, cancelled orders flagged as negative quantities Capacity F64 trial capacity, sandbox workspace Hand-built baseline About 11 hours across two days, including testing The exact prompt I gave it: Ingest the daily order CSVs in Files/raw/orders into a cleaned silver Delta table called silver_orders. Remove duplicates, standardise dates and currency, and exclude cancelled orders. Then build gold_daily_sales with total revenue, order count and average basket size by day, store and category. The pipeline must run incrementally each day. I deliberately kept the prompt a little vague. I didn't say which date formats exist, how cancellations are encoded, or which column identifies a duplicate. In real life, requirements rarely arrive as a perfect spec. I wanted to see whether the agent would investigate, ask, or just guess. What happened, step by step 1. It read the workspace before writing anything The first thing I noticed: it didn't start coding. For the first four minutes it profiled the data. It sampled each CSV, inferred the schema and looked at my existing notebooks in the workspace. It came back with a short summary of what it found. Two findings impressed me: It spotted both date formats (2026-09-14 and 14/09/2026) and flagged that files from 9 September onwards switched format. It noticed my existing notebooks used a nb_<layer>_<entity> naming pattern and said it would follow it. It missed one thing, which matters later: it reported quantity as "integer, some negative values" without asking what negative quantities meant. 2. The plan it proposed Next it produced a 9-step plan and stopped for my approval: Create a staging notebook to read all CSVs with an explicit schema Normalise both date formats to a single order_date column Fill null currency values from the store's default currency Deduplicate on order_id, keeping the latest record Remove cancelled orders Write silver_orders as a Delta table, partitioned by order_date Build gold_daily_sales with the three requested measures Add incremental logic based on the source file date Run validation checks and produce a summary report This is a sensible plan. It is roughly what I wrote on a whiteboard last quarter. But reading it closely, I changed three things before approving: Step 4: it planned to overwrite silver_orders on each run. I changed it to a MERGE on order_id, because late-arriving corrections are common in this data. Step 5: it planned to remove cancellations using a status column. That column exists, but this client never fills it. Cancellations show up as negative quantities. I told it so. Step 8: it wanted to detect new data by file modified time. I asked it to use the date in the file name instead, which is more reliable when files are re-uploaded. It accepted all three changes and rewrote the plan in a few seconds. 3. The build Once approved, it worked for 38 minutes without needing me. It created three notebooks (nb_stg_orders, nb_silver_orders, nb_gold_daily_sales) and a small pipeline to run them in order. The code was clean and readable. Here is its date handling, which is close to what I would write myself: from pyspark.sql import functions as F df = df.withColumn( "order_date", F.coalesce( F.to_date("order_date_raw", "yyyy-MM-dd"), F.to_date("order_date_raw", "dd/MM/yyyy") ) ) And the merge into silver, after my change in the plan: from delta.tables import DeltaTable silver = DeltaTable.forName(spark, "silver_orders") (silver.alias("t") .merge(updates.alias("s"), "t.order_id = s.order_id") .whenMatchedUpdateAll() .whenNotMatchedInsertAll() .execute()) It asked me one question mid-way: whether average basket size should be revenue per order or items per order. That was exactly the right question, and one I had to ask the client myself last time. 4. Self-validation When the build finished, it ran its own checks and produced a short report: Row counts reconciled from raw to staging to silver, with each dropped row explained (duplicates, cancellations) No nulls left in order_date, currency or store_id No duplicate order_id values in silver_orders Daily totals in gold_daily_sales matched a direct aggregate of silver_orders One check failed on the first pass. 41,200 rows had a null order_date after parsing. It traced the cause itself: a third date format (14-Sep-2026) appeared in two of the older files, which its initial sampling had missed. It added a third pattern to the coalesce, re-ran, and the check passed. That moment sold me on the validation step. A human could easily miss 0.2% of rows quietly disappearing. 5. My review against my own version I compared its output with the pipeline I built by hand last quarter, running both on the same 14 files. Measure Agent Me, by hand Time to working pipeline 52 minutes (incl. my review) About 11 hours Plan steps I changed 3 of 9 Not applicable Data issues found 4 of 4, after one self-correction 4 of 4 Full run time on 18.4M rows 6 min 40 s 5 min 55 s Silver row count 17,962,118 17,962,118 The row counts matched exactly. My hand-built version ran about 45 seconds faster, mostly because I had tuned partitioning. More on that below. Would I merge its code as-is? After my three plan changes, yes, with one small performance fix. Guardrails: the part you shouldn't skip An agent that can create and change items in your workspace needs limits before it needs a prompt. Microsoft designed this agent around guardrails that engineers define, and setting them up was the best 15 minutes of the whole test. What I set up before the run: A sandbox workspace. It's a preview feature, so it ran on a copy of the data, nowhere near production. Scoped data. It could read Files/raw/orders/ and write only to new tables. My existing tables were off limits. Approval points. It had to stop for my approval after the plan, and again before the first write to any Delta table. Git in the loop. The workspace was connected to a Git repo, so every notebook it created or changed landed as a diff I could review line by line. In practice, it never tried to step outside these limits. The approval after the plan turned out to be the most valuable one: that is where I caught all three of my changes. If you set only one guardrail, make it that one. Where it struggled No agent review is credible without the failures, so here are the three that mattered. It assumed instead of asking about business rules. It saw negative quantities and planned to use an empty status column for cancellations. Had I approved the plan without reading it, about 310,000 cancelled orders would have flowed into the gold table and inflated revenue. It asked a good question about basket size, but not this one. Its sampling missed a rare date format. The third format appeared in only two files. Its validation caught the problem, but only after the first run. On a much larger dataset, that first run costs real capacity. Correct, but not tuned. It partitioned silver_orders by order_date, which produced many small files for this data volume. Switching to monthly partitions and running OPTIMIZE closed most of the 45-second gap with my version. When I asked it to "make the silver write faster", it suggested the same fix, but it didn't do it unprompted. Three caveats worth knowing before you try it: It's a preview. Features, behaviour and pricing can change before general availability. Don't build production processes on it yet. Watch your capacity. Long-running agent work consumes capacity units on top of the Spark jobs themselves. Check the Capacity Metrics app after your first run so you know what a typical job costs. It doesn't know your business. It can read your schema, not your meeting notes. Every rule I wrote into the prompt was handled correctly. Every rule I left out was a guess. The verdict The Data Engineering Agent is like a fast, tireless junior engineer: excellent at the mechanics, careful about checking its own work, and still in need of a senior reviewing the plan. What I'd tell my team after this test: Spend your time on the prompt and the plan, not the code. Every problem I found was visible in the plan before a single line ran. Write business rules down explicitly. Cancellations, duplicates, currency defaults. If it's tribal knowledge, the agent will guess. Trust the validation step, but read its report. It caught a data loss bug I might have missed myself. Tune afterwards. Treat its first version as correct, then ask it for performance improvements. Try it now if you have a sandbox workspace, a backlog of repetitive ingestion or migration jobs, and someone who can review PySpark. Wait for GA if you need it in production, your pipelines are full of undocumented business logic, or your capacity budget is tight. The bigger shift is in our role. Going from 11 hours to under an hour didn't remove me from the work. It moved my effort from typing code to writing clear requirements, setting guardrails and reviewing output. That is a skill worth building now, preview or not. Have you tried the Data Engineering Agent yet? Tell me in the comments what you threw at it, and where it surprised you. Sources FabCon Europe 2026 announcements (Kanerika) From Prompt to Finished Project: Introducing Project Osmos (FabCon Europe 2026 session) Microsoft Acquires Osmos (Redmond Magazine) Fabric January 2026 Feature Summary (Microsoft Fabric Blog)Privacy by Design in Fabric: Implementing Shift-Left Data Protection
Privacy by Design in Microsoft Fabric: Implementing "Shift-Left" Data Protection for Enterprise Analytics Abstract While data engineers and architects often prioritize pipeline performance, Medallion architectures, and complex transformations, regulatory compliance and data security are frequently treated as afterthoughts. In highly regulated sectors like banking and finance, failing to protect Personally Identifiable Information (PII) can lead to severe penalties under evolving frameworks such as GDPR, PIPEDA, and other international privacy laws. This article explores the concept of "Shift-Left" data protection within Microsoft Fabric—implementing data masking, hashing, and external tokenization to secure sensitive data early in the lifecycle—while distinguishing robust security controls across OneLake, SQL Analytics Endpoint, and Semantic Models. Introduction Modern data professionals excel at writing efficient PySpark scripts, optimizing SQL queries, and designing multi-layered Medallion Lakehouses. However, technical elegance means little if an analytics platform exposes sensitive operational data to unauthorized risks. With the rise of global data privacy regulations and increasing security threats, data governance can no longer be a secondary phase applied right before dashboard deployment; it must be embedded directly into the ingestion and storage strategy. A fundamental best practice in enterprise data architecture is the "Shift-Left" security pattern: masking, pseudonymizing, or hashing sensitive PII attributes at the earliest possible stage rather than relying solely on downstream presentation layers. Within Microsoft Fabric, data teams can execute this strategy across pipelines, PySpark notebooks, and Semantic Models. This article examines practical techniques for implementing early-stage data protection and dynamic security controls in Fabric, helping data engineers build scalable analytical solutions that support strict regulatory mandates. What is "Shift-Left" Data Protection? In traditional data engineering, security controls are frequently applied at the very end of the pipeline—usually at the presentation layer using Power BI dashboard permissions or SQL database views. This legacy approach creates a critical vulnerability: raw, unmasked Personally Identifiable Information (PII) rests in landing zones or intermediate data lake layers, accessible to anyone with direct storage access. "Shift-Left" Data Protection fundamentally changes this dynamic by pushing security, masking, and pseudonymization controls as far upstream as possible. This approach is highly valuable for organizations navigating cross-border data transfers and cloud migrations. Many international regulatory frameworks strongly recommend or mandate that when transferring sensitive data across borders, the information should be pseudonymized or encrypted to support data sovereignty and localization principles. Failing to comply with privacy frameworks carries severe financial and reputational consequences for enterprises worldwide. By applying cryptographic hashing, masking, or tokenization patterns early in the Medallion Architecture: Exposure is Minimized: Sensitive data (such as national IDs, emails, or account numbers) is restricted or neutralized before persisting in downstream analytical environments. Supports Compliance Initiatives: Organizations are better positioned to align with privacy principles established by global frameworks (like GDPR or PIPEDA) by adopting a Privacy by Design architecture. Governance is Streamlined: Instead of managing complex downstream permission rules across dozens of individual reports, core data security is enforced at the storage and engine layers. To determine whether an enterprise data pipeline truly adheres to a "Shift-Left" architecture, data engineering and compliance teams should verify key architectural control points, such as: [ ] Ingestion Gatekeeping: Are raw PII fields masked in transit before landing in the Bronze layer, or if landing raw in Bronze, is direct storage access strictly limited to service principals and administrators? [ ] Pseudonymization (Bronze to Silver): Are identifying attributes transformed into deterministic hashes (like salted SHA-256) before persisting into Silver Delta tables? [ ] Integration with Token Vaults: Are highly sensitive fields passed to external tokenization services during ingestion, ensuring the decryption vaults are isolated outside the analytics workspace? [ ] Identity and Role Verification: Are workspace roles configured correctly so that downstream consumers only access data via secured endpoints rather than having direct underlying OneLake file access? Demystifying the Security Layers in Microsoft Fabric Implementing a "Shift-Left" framework in Fabric requires understanding that security is not monolithic; it is applied across distinct layers. A robust architecture leverages data-layer controls, transformation techniques, and compute-engine security rules. 1. The Storage & Transformation Layer (Upstream) Source-Side Masking: Using Data Factory pipelines to mask data in transit before it ever writes to OneLake. Hashing as Pseudonymization: Using PySpark notebooks to apply SHA-256 hashing to fields like email addresses. Note: Hashing is pseudonymization, not true anonymization. To prevent dictionary or rainbow table attacks, data teams must implement secure "salting" alongside the hashing algorithm. External Tokenization Integration: Fabric pipelines can call external REST APIs (e.g., third-party tokenization services or Azure Key Vault) to replace sensitive strings with non-mathematical tokens during the Bronze-to-Silver transformation. OneLake Security (Data Access Roles): Fabric increasingly supports native OneLake security controls, allowing administrators to define data access roles directly on the OneLake folders and Delta tables, securing the files regardless of which compute engine accesses them. 2. The Engine & Consumption Layer (Downstream) Once data is curated, Fabric provides specific, engine-level security controls: SQL Analytics Endpoint (T-SQL Controls): Row-Level Security (RLS): Filters specific rows (e.g., restricting a regional manager to only see sales from their territory) using standard T-SQL CREATE SECURITY POLICY statements. Column-Level Security (CLS): Uses T-SQL GRANT SELECT statements to prevent specific roles from querying sensitive columns like Salary or SSN, while allowing them to query the rest of the table. Semantic Models (Direct Lake / Power BI Controls): Row-Level Security (RLS): Configured via DAX to filter data rows natively within the semantic model. Object-Level Security (OLS): Configured via Tabular Editor/XMLA to completely hide entire tables or columns from specific users in the semantic model, preventing report visuals from even discovering the fields. High-Level Architecture in Microsoft Fabric Implementing a "Shift-Left" data protection framework within Microsoft Fabric relies on combining two complementary security layers: In-Flight Data Sanitization (Storage Layer): Using Fabric PySpark Notebooks or Data Factory pipelines to hash, mask, or tokenize PII fields during ingestion or before writing into Silver Delta Tables in OneLake. Dynamic Access Controls (Engine Layer): Enforcing Row-Level Security (RLS) and Column-Level Security (CLS) on top of the SQL Analytics Endpoint or Direct Lake Semantic Models for role-based downstream consumption. To visualize how this defense-in-depth strategy comes together, the following reference architecture outlines how "Shift-Left" sanitization and dynamic query controls are orchestrated within Microsoft Fabric: Understanding the Architectural Diagram The architectural diagram above illustrates a simple example of an end-to-end Microsoft Fabric implementation designed around the "Shift-Left" paradigm. To read and interpret this architecture effectively, consider the flow of data through three core operational layers: 1. Data Sources & Boundary Controls (Left): Data flows from operational systems (On-Premises SQL Databases, REST APIs) toward Microsoft Fabric OneLake. Red Locks (Regulatory Blocking & Ingestion Masking): Represent early security controls applied at the connector level. Sensitive operational fields are masked or blocked before landing in cloud storage to prevent cross-border compliance violations. 2. Medallion Lakehouse & In-Flight Sanitization (Center): Bronze Layer (Raw Ingestion): Stores incoming raw landing data. Shift-Left Sanitization (PySpark Notebook): Before moving data into the Silver Layer, PySpark notebooks perform cryptographic transformations (SHA-256 hashing, masking, tokenization). Silver Layer (Sanitized & Standardized): Contains cleaned, enterprise-ready data where PII has been neutralized at the storage level within OneLake Delta files. 3. Consumption & Dynamic Engine Security (Right): Gold Layer (Aggregated & Enriched): Curated dimensional models built for business analytics. SQL Analytics Endpoint & Direct Lake Semantic Models: As queries hit the serving layer, secondary security rules—such as Column-Level Security (CLS) and Row-Level Security (RLS)—dynamically filter rows or restrict specific columns based on the user's Entra ID (Azure AD) identity, ensuring seamless role-based governance across Power BI reports and T-SQL queries. The Critical Role of Workspace Identity and Direct Storage Access The most common pitfall when designing security in Microsoft Fabric is a misunderstanding of Workspace Roles and Identity Modes. SQL RLS/CLS and Semantic Model RLS/OLS are engine-level controls. If a user is granted the Admin, Member, or Contributor role in a Fabric Workspace, they possess inherent direct storage access to the underlying OneLake files. Because they can access the raw Delta files directly via OneLake Explorer or Notebooks, they will completely bypass SQL RLS/CLS and Semantic Model security. To successfully enforce engine-level security (like SQL CLS or Direct Lake RLS), downstream consumers must be assigned the Viewer role in the workspace. This restricts their access to the defined endpoints (SQL Endpoint or Power BI App) where the security policies are evaluated under their Entra ID credentials, effectively blocking direct access to the unmasked storage files. Key Takeaways for Data Professionals Implementing "Shift-Left" data protection within Microsoft Fabric allows organizations to bridge the gap between high-performance analytics and governance requirements. By moving sanitization controls upstream and combining them with dynamic engine-level security, data teams can build a future-proof architecture that meets strict privacy mandates without sacrificing agility. To successfully apply these principles in your enterprise, keep the following core takeaways in mind: Security is an Architecture Feature, Not a Final Step: Embed data protection requirements into your initial pipeline designs rather than treating them as an afterthought. Protect at the Lowest Layer Possible: Hashing, tokenizing, and masking data early in the Medallion Architecture (Bronze-to-Silver transition) protects sensitive attributes across all downstream environments, including Notebooks, SQL queries, and Direct Lake Power BI reports. Align Controls to the Right Layer: Use OneLake security for raw storage, T-SQL CLS/RLS for the SQL Endpoint, and DAX OLS/RLS for Semantic Models. Audit Workspace Roles: Never rely on SQL or Semantic Model security for users who have Admin, Member, or Contributor access to the workspace. Join the Conversation! Designing enterprise-grade security in modern data platforms is an evolving practice, and different organizations adopt unique strategies depending on their regulatory environment. What additional techniques or governance frameworks do you implement in your organization to protect sensitive PII in Microsoft Fabric? Are you primarily leveraging SQL-based CLS, or shifting toward OneLake data access roles? Share your thoughts, experiences, or questions in the comments below!
ADLS Gen2 and OneLake - Choosing the right storage approach
As Microsoft Fabric adoption grows, a practical architecture question follows: how does OneLake relate to Azure Data Lake Storage Gen2 (ADLS Gen2), and do we need to migrate? Both store analytics data in open formats, but they offer different ways to manage and consume it. I would start with the workload and the investments already in place, rather than with a decision to move storage. For an established lake, keeping ADLS Gen2 and exposing selected data to Fabric may be the simplest path. For a new Fabric-centric workload, storing data directly in OneLake may reduce operational work. The useful unit of decision is a layer or data product, not the entire estate. Start with the operating model What ADLS Gen2 is designed for ADLS Gen2 brings analytics capabilities to Azure Blob Storage through a hierarchical namespace. In this PaaS model, Azure runs the storage service while your team manages accounts, containers, networking, lifecycle policies and access through Azure role-based access control and POSIX-style ACLs. It can hold any file format, giving teams flexibility to choose how different engines process the data. That flexibility is valuable when ingestion pipelines and multiple analytics engines already share a lake. Azure Databricks, Azure Machine Learning and other compatible tools can work with the data where it is. The architectural responsibility sits with the team: bringing storage, processing, discovery and governance together into an operating platform. OneLake: Fabric-managed data lake Built on ADLS Gen2, OneLake is created automatically for a Fabric tenant. Microsoft manages the underlying storage infrastructure, while teams organize and secure data through workspaces and items such as Lakehouse’s. This SaaS model shifts the day-to-day focus from storage accounts to analytics data products. OneLake stores physical data and can also reference data elsewhere through shortcuts. The benefit becomes tangible in the way Fabric workloads work together. Spark notebooks prepare Lakehouse data, SQL analytics experiences expose supported tables, and Power BI can consume suitable tables through Direct Lake. A Lakehouse’s Files area can hold any file format, while its Tables area serves supported table formats and engines, including Delta and supported Iceberg tables through metadata virtualization. Shared storage simplifies integration, while each engine still has its own supported capabilities. The architectural relationship The storage hierarchy is straightforward: tenant, workspace, then data item. Within a Lakehouse, Tables and Files support different consumption needs. Domains group workspaces for organizational ownership and governance, rather than adding a folder level to the storage path. A simple example is: Scope Illustrative organization Fabric tenant OneLake namespace Workspace Reporting workspace containing a Sales Lakehouse Lakehouse Tables Local Delta tables, or a shortcut to supported Gold tables in ADLS Gen2 Lakehouse Files Files and folders, including supported external shortcuts What changes for analytics consumers Shortcuts: Bring existing data into reach A plain shortcut references an internal or external storage location without requiring an ingestion copy. If curated Delta tables already live in ADLS Gen2, a supported Lakehouse Tables shortcut can make them available to Fabric consumers. Files shortcuts expose files instead; making those files into queryable tables is a separate step. This provides an incremental adoption path, although downstream workloads may still cache or materialize data. The access design deserves as much attention as the data path. An ADLS shortcut uses the source connection’s configured credentials alongside Fabric permission checks. It does not automatically carry each reader’s source ACLs into Fabric. Before sharing it, I would check least privilege, network reachability and table compatibility with the identities and workloads that will use it. Shortcut transformations address a different need: preparing data for consumption. Supported CSV, Parquet and JSON transformations use Fabric Spark to copy and convert source content into managed Delta output and keep it synchronized. That can simplify preparation, but it introduces compute, output storage and synchronization to operate. Choose a plain shortcut when referencing the existing data is enough; choose a transformation when the output itself adds value. Direct Lake: Simplify the BI data path Direct Lake loads requested Delta-table columns into VertiPaq memory without requiring a full Import-mode refresh. Metadata refresh, also called framing, still determines the data version available to the model. The practical benefit is a simpler BI data path for suitable workloads. Capacity, table optimization, security and model design still determine performance and freshness. The underlying Delta data can remain in ADLS Gen2 when exposed through a supported OneLake Tables shortcut. That is the bridge to Direct Lake, rather than a direct connection from the semantic model to an ADLS path. I would assess the intended Direct Lake mode and security design, then test representative queries and freshness requirements before making performance commitments. Governance: Discovery and enforcement both matter Governance needs both business context and effective access control. ADLS Gen2 can use Microsoft Purview for discovery and governance alongside its storage permissions. In Fabric, the OneLake catalog brings discovery, metadata, lineage and endorsement into the analytics experience, while workspaces and items provide ownership context. Those capabilities help people find and understand data; the permissions on the actual access path determine who can use it. I would compare security and lifecycle requirements against the specific design, rather than assume one platform always offers more control. Fabric supports tenant- and workspace-level private links and customer-managed keys with supported-item constraints. OneLake also supports lifecycle policies and hot, cool and cold tiers. The decision turns on whether the required workload, connector, encryption and network configuration work together, not simply whether a feature appears on a checklist. AI readiness starts with usable data Both approaches can support AI. ADLS Gen2 is a supported Azure Machine Learning datastore, so an established lake may already be a sound foundation. Fabric adds integrated preparation and consumption options that can be useful without relocating that lake. My starting question would be which capability the team needs and where the data is already well governed. For example, AI-powered shortcut transformations can turn supported .txt files into Delta output containing summaries, translations, sentiment, redacted PII or extracted entities. A team could summarize eligible text sourced from ADLS and analyze the results in Fabric. This capability is in public preview, with regional and capacity requirements. Output quality and sensitive-data handling still need evaluation; automated redaction alone cannot establish compliance. Interoperability depends on the integration path Snowflake illustrates why the integration path matters. Fabric Mirroring can replicate Snowflake-managed table data into OneLake. For Iceberg tables in supported object storage, shortcuts and metadata virtualization provide another route; the Snowflake Iceberg mirroring path uses metadata mirroring and storage shortcuts. These are different architectural choices, with different implications for copies, freshness and ownership. Snowflake on Azure can also write Iceberg tables to OneLake in supported configurations. That integration requires public connectivity, does not support private-link workspaces, and has same-region prerequisites for writes. Azure Databricks can read and write OneLake through authenticated ABFS paths, with compute and authentication requirements. Open formats help, but the chosen engine and network design still need to fit the supported integration. Make the choice layer by layer An existing Databricks lake: should Bronze and Silver move? Consider a team with reliable Azure Databricks pipelines and Bronze, Silver and Gold layers in ADLS Gen2 that now wants Fabric reporting. My recommendation would be to keep the working Bronze and Silver layers unless there is a separate business or technical reason to move them. For Gold, there are two useful starting points. Keep physical Gold in ADLS Gen2 and expose supported Delta tables through Lakehouse Tables shortcuts. Gold becomes available in Fabric without physical move. I would start here when existing storage contracts, authoritative writers and non-Fabric consumers need to remain intact. Alternatively, write or materialize selected Gold data directly in OneLake when Fabric ownership or service-level requirements justify changing the pipeline. Databricks can support this pattern through authenticated ABFS access with the appropriate compute configuration. Moving selected Gold products can be a focused decision, independent of any future Bronze or Silver migration. For either pattern, agree on the authoritative writer, table compatibility, effective permissions, freshness targets, monitoring and total cost. A storage shortcut does not transfer Unity Catalog authorization, so test the security model with representative readers. These responsibilities matter more than whether the diagram shows one lake or two. Count the governance and semantic-model work I would also count the work above storage. A team may maintain security rules and semantic logic in Databricks, then recreate them for Fabric reporting. Repeating policies, metrics, relationships and business definitions creates maintenance and testing work, with a risk of drift whenever requirements change. Reducing that duplication can be a stronger reason to adopt a shared consumption layer than saving a data copy. A useful pattern is to use Databricks for data preparation, make governed Gold data available through OneLake, and use Fabric as the shared consumption layer. OneLake security can centralize data-access policies across supported Fabric engines and access paths. Separately, a deliberately shared Power BI semantic model lets reports and compatible clients reuse measures, relationships and business definitions instead of rebuilding them. These are two distinct opportunities to reduce downstream work: common access control and an explicitly reusable semantic layer, not benefits that storage alone creates. That semantic model reuse does not require moving your Gold layer: supported shortcuts can expose ADLS data, and shared Power BI models can also use other supported sources. OneLake does not automatically unify Unity Catalog policies nor give every Databricks or Spark workload the same semantic layer. Where Unity Catalog remains authoritative, I would evaluate supported integrations and ask: where are policies and metrics defined, which engine enforces them under which identity, which consumers reuse the model, and who maintains and tests necessary differences? A practical guide Your situation Starting point What to validate Successful multi-engine lake; established account/API dependencies Keep ADLS Gen2 as an authoritative store Existing contracts, source security and the specific Fabric access path New Fabric-centric analytics and shared Fabric items Use native OneLake storage Ownership, supported workloads, capacity, security and lifecycle requirements Mixed estate; selective Fabric adoption or duplicated governance and model logic Combine ADLS Gen2 and OneLake; assess a shared consumption layer See the mixed-estate checklist below. A required security or storage feature determines the design Compare the exact supported configurations Connector and workload limits, private networking, CMK item coverage, retention and total cost For a mixed estate, validate: Data access: shortcut compatibility versus materialization. Write ownership: the authoritative writer. Operations: freshness targets and monitoring. Security: effective identities and policy enforcement. Semantics: reuse of a shared semantic model. Governance ownership: who maintains and tests necessary cross-platform differences. Coexistence as a practical starting point Use ADLS Gen2 where its customer-managed storage model and existing integrations serve the workload well. Use native OneLake where Fabric’s managed model and shared analytics experiences provide a clear benefit. For an existing estate adopting Fabric, I would usually start by connecting the two and move data only where the benefit is clear. Coexistence can be a transition or a lasting design; consolidation should earn its place through workload requirements and measurable value.Connect Fabric Data Agents to Copilot Studio with MCP (+ Assistants API Migration Fix)
Business users want answers where they already work, in Teams and Microsoft 365 Copilot, and those answers need to come from trusted, governed data. Fabric data agents in Copilot Studio do exactly that, and the integration is now generally available. There is also a deadline you may already have passed. If you connected to Fabric data agents through the OpenAI Assistants API, your integration has likely stopped working. This guide covers both the setup and the fix. What's new Fabric data agents now connect to Copilot Studio through a new tool-based experience: you select Add a tool, search for Fabric, and choose Fabric IQ Data MCP. Your Copilot Studio agent then calls the Fabric data agent like any other tool. Microsoft Community Three points matter for architects: Data stays governed. The Fabric data agent keeps running in Fabric, respects permissions on the underlying data sources, and returns answers grounded in governed enterprise data. Microsoft Community The orchestrator decides. The orchestrator chooses when to use enterprise data and when to use its other knowledge sources and tools. Microsoft Community You can combine agents. You can pair several Fabric data agents with other business systems to build richer solutions. A sales agent could query a sales data agent, a finance data agent and your CRM connector in a single conversation. Microsoft Community Behind this, Copilot Studio also introduced a redesigned authoring surface and runtime powered by the GitHub Copilot harness. Microsoft Community Why MCP matters Model Context Protocol (MCP) is becoming the standard way agents connect to tools and data. By exposing Fabric data agents through MCP, Microsoft makes them a reusable building block instead of a one-off integration. The same move is happening elsewhere: Fabric data agents in Microsoft Foundry are now easier to connect and monitor, and that integration is moving to MCP too. Microsoft Community You build a data agent once in Fabric, and Copilot Studio, Foundry and Microsoft 365 Copilot can all use it. Prerequisites Before you start, check that you have: A Microsoft Fabric workspace on Fabric capacity Data ready to use: a lakehouse, warehouse, semantic model or similar source with clean, well-named tables Permission to create items in the Fabric workspace Access to Copilot Studio with permission to build and publish agents Microsoft 365 Copilot or Teams, depending on where you want to publish Tip: the quality of your data agent depends on the quality of your data. Clear table and column names and descriptions make a big difference to answer accuracy. Step 1: Build and test your Fabric data agent In your Fabric workspace, create a new data agent item, then: Add data sources. Connect the lakehouse, warehouse or semantic model the agent should answer from. Start narrow, since one domain per agent works best. Write agent instructions. Explain the business context, key terms and rules. For example: "Revenue means net revenue after returns. Fiscal year starts in April. Always exclude internal test accounts." Add example questions and queries. Give sample questions with the correct query logic, so the agent learns your patterns. Test thoroughly. Ask the questions your business users actually ask and compare the answers with your certified reports. Publish the agent once the answers are reliable. Step 2: Add the data agent to Copilot Studio This is the new MCP-based flow. Create or open an agent in Copilot Studio, go to Tools, select Add a tool, search for Fabric, and add Fabric IQ Data MCP. Microsoft Community Then: Select the Fabric data agent you published in Step 1. Give the tool a clear description, such as "Answers questions about sales, revenue and pipeline from governed Fabric data." The orchestrator reads this to decide when to call the tool, so be specific. In your Copilot Studio agent's instructions, tell it when to use the tool, for example: "For any question about sales numbers or targets, use the Sales Data tool." Step 3: Test in Copilot Studio Test your agent in Preview before publishing. Check that: Microsoft Community Data questions trigger the Fabric tool, and general questions don't Answers match the results from Fabric directly A user without access to certain data gets no answer from it, since the data agent respects permissions on the underlying sources Follow-up questions ("and what about last quarter?") work as expected Step 4: Publish to Teams or Microsoft 365 Copilot When you're happy with the results, publish to Microsoft Teams or Microsoft 365 Copilot. Business users can now ask data questions in the tools they use every day, without opening a report. Microsoft Community Roll out to a small pilot group first, collect feedback, and improve the data agent's instructions before rolling out widely. The Assistants API retirement: what broke and how to fix it This part matters if you built programmatic integrations. What happened: The OpenAI Assistants API, which powered the orchestration layer of the Fabric data agent, was scheduled to be shut down by OpenAI on August 26, 2026. After that date, direct calls to the Assistants API stop working. Microsoft Community Who is affected: anyone who connects to a Fabric data agent programmatically through the Assistants API needs to migrate to the data agent MCP endpoint. Typical examples are custom apps, scripts, or third-party tools calling the data agent directly. Microsoft Community Who is mostly fine: SDK and Fabric portal users need little or no action, because Microsoft migrates those experiences internally, although conversation history may reset once. Microsoft Community What stays the same: existing agent data sources, instructions and tools remain unchanged. You don't need to rebuild your agents, only change how you connect to them. Microsoft Community Migration checklist Find your integrations. Search your code and configs for Assistants API calls that target Fabric data agents. Check custom apps, automation scripts, Azure Functions and any third-party tools. Check for failures since late August. Errors or silent failures after August 26 are a strong sign an integration is affected. Move to the MCP endpoint. Update each integration to connect through the data agent's MCP endpoint instead. Follow Microsoft's guide, "Prepare your Fabric Data Agent integrations for Assistants API retirement," for the current details. Consider Copilot Studio instead of custom code. If a custom app only exists to put a chat interface on a data agent, the Copilot Studio route above may replace it with less code to maintain. Tell users about history resets. If you use the portal or SDK, let users know earlier conversations may disappear once. Retest. Run your standard question set again after migrating to confirm the answers haven't changed. Best practices for production One domain per data agent. Separate agents for sales, finance and operations are easier to test, govern and improve than one agent that tries to do everything. Write tool descriptions carefully. In Copilot Studio, the tool description is how the orchestrator decides what to call. Vague descriptions lead to wrong tool choices. Treat instructions like code. Version them, review changes, and retest after every edit. Keep a test question set. Keep 20 to 30 real questions with expected answers, and run them after any change to data, instructions or model. Watch permissions. Answers respect underlying permissions, so a mistake in data security becomes a mistake in agent answers. Review access regularly. Start small and grow. Pilot with one team, measure how often answers are right, then expand. The bottom line Connecting Fabric data agents to Copilot Studio through MCP turns your governed Fabric data into a reusable tool that any agent can use, and it puts trusted answers directly into Teams and Microsoft 365 Copilot. The setup takes minutes once your data agent is solid. If you built integrations on the Assistants API, make migration your first job this week. Your agents, data sources and instructions don't need to change, only the connection. After that, you're on the platform Microsoft is building its agent strategy on.Fabric Governance Patterns: The First Step to Trusted Scale
A large enterprise had already embraced self-service analytics. Business teams were building reports, connecting to data, publishing dashboards, and moving fast. At first, this looked like success. More people had access to data. More teams could answer their own questions. More reports were being created without waiting for a central team. But over time, a different pattern started to appear. The same metric had more than one definition. Reports were difficult to find. Some workspaces had too many administrators. Critical assets did not always have clear owners. Development and production were not consistently separated. Data lineage was hard to trace. Capacity usage needed closer oversight. And as AI and Copilot entered the conversation, the question became even sharper: Can we scale innovation without scaling risk? That is where Fabric governance begins. Not as a blocker. Not as another layer of bureaucracy. But as the first step toward trusted, reusable, secure, and AI-ready data. The first lesson: governance is not a document, it is an operating model One thing I have learned is that governance does not succeed because a framework exists. It succeeds when people know how to use it. A governance document can describe roles, standards, and controls, but the operating model is what makes those ideas practical. It defines who decides, who approves, who owns, who monitors, and who acts when something is not working. For Fabric, this usually means moving toward a hub-and-spoke model. The hub defines enterprise guardrails, platform standards, monitoring, training, and governance consistency. The spokes own business context, local adoption, workspace execution, semantic model quality, and day-to-day compliance. That shift matters because Fabric is not only a reporting platform. It brings together data engineering, Power BI, OneLake, semantic models, pipelines, security, metadata, AI, and governance capabilities. If the operating model is unclear, the platform can grow faster than accountability. The first step is therefore simple to state but important to execute: Define who owns what before scaling Fabric further. What happens if the first step is skipped Without governance at the start, self-service becomes self-risk. I have seen this pattern many times: teams do not intend to create risk. They are trying to solve real business problems quickly. A report is needed urgently. A dataset is copied because it is faster. A semantic model is rebuilt because the existing one is hard to find. A workspace is created because the team needs somewhere to collaborate. Each decision makes sense locally. But collectively, the platform becomes harder to govern. Current pattern Risk created Governance direction Decentralized citizen development with limited oversight Report proliferation and reduced trust Hub-and-spoke governance model Users self-manage workspaces, datasets, refreshes, and models Source-of-truth confusion Asset lifecycle ownership Too many workspace admins and inconsistent access approval Privacy, exposure, and compliance risk Least-privilege access controls Reports and metrics are hard to discover Duplication and poor reuse Metadata, catalog, and marketplace discovery Semantic models are rebuilt by each analyst Conflicting business logic Shared, reusable semantic models Reports are published directly to production Quality and change risk Dev/Test/Prod deployment path Unoptimized refreshes and transformations Capacity pressure and performance issues Central reusable pipelines and monitoring AI features are enabled without readiness checks Misuse, poor prompts, weak auditability Responsible AI, metadata, training, and approval controls The biggest risk is not that people build too much. The biggest risk is that no one can confidently say which assets are trusted, who owns them, and whether they are safe to reuse. The story of the first step Imagine a business user needs a new dashboard. In the old world, they might connect Power BI Desktop directly to Excel, SQL, SharePoint, or an API. They build a model, define their own business rules, publish to a workspace, and share the report. It works. Until another team builds the same metric differently. Until the source changes. Until access becomes unclear. Until refreshes start consuming capacity. Until a critical executive report depends on an undocumented model. In the governed Fabric world, the story changes. The user still gets the insight. But the path is safer: The data is ingested through governed pipelines. Data is curated in OneLake or an approved warehouse/lakehouse pattern. Certified or strategic data products are reused where available. Semantic models are owned, documented, and version controlled. Reports move through development and production controls. Metadata, lineage, access, and quality checks are captured. Monitoring shows whether the platform is healthy and compliant. The user experience should feel easier, not harder. Governance should make the trusted path the default path. Personas: the people who make governance real Fabric governance only works when roles are explicit. In my experience, this is where many governance initiatives either become practical or remain theoretical. If the role definitions are vague, everyone agrees governance is important, but no one knows who should approve, remediate, certify, or retire an asset. Persona Story role Responsibility Fabric Governance Lead Sets the rules of the road Defines policies, guardrails, governance forums, standards, and adoption KPIs Fabric Technical Product Owner Owns the platform roadmap Manages feature enablement, Fabric roadmap, platform health, and Microsoft relationship Fabric Architect Designs the pattern Defines architecture blueprints, workspace structure, integration patterns, and semantic model guidance Fabric Product Owner / Spoke Owner Owns business outcomes Prioritizes BI demand, funds initiatives, approves adoption, and ensures business value Steward Runs governance day to day Ensures standards, metadata, access, certification, lifecycle, and workspace guardrails are followed Capacity Manager Protects performance and cost Monitors capacity usage, manages allocation, forecasts demand, and resolves overages Monitoring Analyst Turns governance into evidence Builds governance scorecards, compliance metrics, adoption reporting, and exception triggers Developers and report creators Build governed assets Use certified data, follow standards, document metadata, validate quality, and publish through controlled pipelines End users Consume trusted insights Use certified reports and datasets, provide feedback, and avoid unmanaged duplication Security, privacy, and responsible AI teams Protect the enterprise Advise on sensitive data, AI use, privacy, compliance, auditability, and risk controls The important point is not the title. The important point is accountability. Every workspace, semantic model, data product, report, capacity, and AI-enabled experience needs an owner. Governance patterns that turn intent into action A practical Fabric governance framework should define patterns across ten areas: Governance pattern What it controls Operating model pattern Hub-and-spoke governance, community of practice, training, enablement Capacity pattern Dedicated capacity ownership, funding, sizing, production/development tagging Workspace pattern Workspace approval, ownership, production isolation, review and decommissioning Copilot and AI pattern Metadata readiness, training, responsible AI, logging, approval, monitoring Data product pattern Certified data products, shared lakehouses, curated ingestion, retention, exfiltration control Semantic model pattern Reusable models, ownership, glossary integration, lineage, change management Access pattern Least privilege, RBAC groups, RLS/OLS/CLS, masking, periodic access reviews Monitoring pattern Capacity health, platform alerts, governance scorecards, compliance KPIs DevOps pattern Dev/Test/Prod separation, deployment pipelines, validation, version control Onboarding pattern Role-based access, training, appropriate workspace roles, production read-only controls This is where governance becomes operational. It moves from “we should govern Fabric” to “this is how every team works.” A lightweight Power Platform intake can help One of the most practical lessons I have learned is that governance needs an easy front door. If the process is unclear, people bypass it. If the process is too heavy, people avoid it. If approvals live only in email, governance becomes hard to track. A simple Power Platform intake can make the first step easier: Request a new capacity. Request a workspace. Request access. Request Copilot enablement. Request certification. Request exception approval. Capture business owner, technical owner, data classification, lifecycle, and approval history. Route the request to the right steward, product owner, platform owner, or security reviewer. Closing thought Fabric governance is not about creating more process. It is about creating confidence. Confidence that business users are working with trusted data. Confidence that platform teams can scale without losing control. Confidence that security and compliance teams have visibility. Confidence that AI and Copilot experiences are grounded in governed, reusable, well-understood data The organizations that will get the most value from Microsoft Fabric will not be the ones that simply adopt it fastest. They will be the ones that create the right patterns early, so speed and trust can grow together. Start small, define ownership, make the governed path easy, and let governance become the foundation for trusted scale.The Hidden Cost of the Data Swamp: Fabric Capacity Guardrails
Introduction: The Double-Edged Sword of Fabric Flexibility Microsoft Fabric has fundamentally revolutionized how enterprise data teams operate. By unifying data engineering, data science, data warehousing and Power BI into a single Software-as-a-Service (SaaS) platform, it breaks down silos and lets business users move at lightning speed. However, this democratization of data comes with a massive hidden risk. When workspace creation is unchecked and pipelines run wild, speed quickly transforms into a costly “data swamp.” Unlike traditional infrastructure where compute and storage are rigidly partitioned, Fabric pools your processing power into Capacity Units (CUs). If you don’t govern those capacities, runaway queries, poorly optimized Spark jobs and unmanaged workspace sprawl will quietly drain your cloud budget before finance even sees the bill. 1. The Anatomy of a Fabric Cost Explosion Cause-and-effect visualization demonstrating how unmonitored workspace sprawl and orphaned background pipelines drive sudden spikes in Capacity Unit (CU) consumption. How Cloud Costs Spiral Out of Control In a consumption-based SaaS model, expenses aren’t driven just by storage – they are driven by active compute. Without strict guardrails, organizations frequently run into three major financial traps: The “Noisy Neighbor” Problem: A single inefficient, ad-hoc DAX query or an unoptimized PySpark notebook running in a shared capacity can consume 100% of the processing power, throttling critical executive dashboards and slowing down the entire organization. Orphaned Pipelines & Infinite Loops: Automated data factories and pipelines left running by developers who forgot to turn them off after testing, continuously burning CUs over weekends and holidays. Unmonitored Sprawl: Business units spinning up independent workspaces without designated capacity assignments, causing unpredictable spikes when heavy reporting cycles (like month-end financial closing) collide. 2. Real-World Challenges: Why Monitoring is Hard Real-world operational challenges illustrated through capacity metrics app warnings and automated performance bottleneck notifications. Blind Spots in the Enterprise Managing compute costs in Fabric is uniquely challenging because responsibility is decentralized. Decentralized Ownership: When business analysts build and manage their own workspaces, they rarely think about the underlying cluster compute footprint. They care about insights, not core optimization. The Delayed Billing Shock: Because usage metrics update dynamically, teams often don’t realize a workload has choked the capacity until after the performance degradation has already impacted end users. Lack of Throttling Transparency: Without proactive alert mechanisms, teams are caught entirely off guard when Fabric automatically pauses or throttles operations due to capacity exhaustion. 3. Best Practices for Fabric FinOps and Capacity Governance Actionable three-step checklist highlighting core FinOps best practices for establishing sustainable capacity guardrails in Fabric. Taking Back Control of Your Compute Budget To keep your cloud bill predictable and your performance lightning-fast, you need to embed FinOps principles directly into your governance strategy. Isolate Workloads by Capacity Assignment: Never mix heavy data engineering Spark jobs with executive Power BI semantic models on the exact same capacity tier. Separate your development, test and production workloads to protect critical business assets. Master the Fabric Capacity Metrics App: Make it a non-negotiable weekly habit for administrators to review the Microsoft Fabric Capacity Metrics app. Look closely at your Interactive vs. Background CU consumption spikes to pinpoint which specific semantic models or items are dragging down performance. Set Up Proactive Alerting and Guardrails: Use Azure Monitor or built-in notification hooks to alert your administrators when capacity utilization hits dangerous thresholds (e.g., crossing 80% sustained usage), allowing you to intervene before automatic throttling kicks in. Enforce Workspace Lifecycle Policies: Automatically archive or clean up abandoned workspaces and semantic models that haven’t been queried in over 90 days, saving both storage and compute metadata overhead. 4. Conclusion: Turning FinOps into a Competitive Advantage The ultimate goal of Fabric FinOps and Governance – transforming an unpredictable data swamp into a secure, high-performance and economically sustainable analytics powerhouse. Governance in Microsoft Fabric isn’t just about security clearance lists and data privacy – it is about economic sustainability. By treating your capacity units as a finite, precious corporate asset, you protect your cloud budget from unexpected waste. When you combine robust workspace structuring, active Purview data labeling and diligent FinOps capacity monitoring, you turn your Fabric tenant from an unpredictable financial black hole into a predictable, high-performance enterprise analytics powerhouse. What about you? How is your organization currently keeping runaway Capacity Unit (CU) spikes and “noisy neighbor” workloads under control in Microsoft Fabric? Drop your strategies or biggest cost-management challenges in the comments below!Row-Level Security in Direct Lake Models: The Undocumented Gotchas
When we moved our sales reporting from Import mode to Direct Lake, RLS looked like the easy part. We already had dynamic RLS working in the old Import model: a security table, a USERPRINCIPALNAME() filter, and a couple of relationships. We expected to copy it over and be done in an afternoon. It took two weeks. The RLS logic itself was fine. What caused the trouble was how Direct Lake interacts with workspace permissions, the SQL analytics endpoint, framing, and fallback. Most of it is technically documented, but spread across a dozen pages, and a few behaviors we only found by testing. This post covers what we hit, how we diagnosed it, and what we'd do differently next time. The setup Here's a simplified version of what we built: Lakehouse: LH_Sales with fact_sales, dim_region, dim_product, dim_date Security table: sec_user_region (UserEmail, RegionKey), one row per user per region Semantic model: Direct Lake on the SQL analytics endpoint Users: about 400 sales reps and managers, each seeing only their regions The RLS role was the standard dynamic pattern: // Role: RegionSecurity // Table: sec_user_region [UserEmail] = USERPRINCIPALNAME() The relationship sec_user_region[RegionKey] → dim_region[RegionKey] had "Apply security filter in both directions" enabled, and dim_region filtered fact_sales. It worked when I tested it as myself, and it failed in several different ways once real users got access. Gotcha #1: Workspace roles silently bypass your RLS This is the most common one, and it matters more in Direct Lake. Semantic model RLS only applies to users who have Read permission on the model. Anyone with Admin, Member, or Contributor in the workspace has write permission, so RLS does not apply to them. They see everything. That's the same as Import mode. The Direct Lake-specific problem is the next part. The Viewer role has a hole too. A workspace Viewer is subject to your semantic model RLS, but Viewers can also connect to the lakehouse's SQL analytics endpoint and query fact_sales directly from SSMS, Excel, or a notebook. Your DAX roles don't exist there, so they can read every row. We found this when a regional manager emailed us a pivot table with national numbers. He had connected Excel to the SQL endpoint because he found it faster. The relationship sec_user_region[RegionKey] → dim_region[RegionKey] had "Apply security filter in both directions" enabled, and dim_region filtered fact_sales. It worked when I tested it as myself, and it failed in several different ways once real users got access. What we changed: Removed every business user from workspace roles. Shared the report and semantic model through an App (or item-level sharing) with Read permission only, without "Build" unless it was genuinely needed. Did not grant lakehouse access to end users at all (Gotcha #2 explains how that still works). Rule of thumb: If a user can see the lakehouse, assume they can see all of it unless you've also secured it at the SQL/OneLake layer. Gotcha #2: SSO vs. fixed identity changes who needs lakehouse access By default, a Direct Lake model on the SQL endpoint uses single sign-on (SSO): the viewer's own identity is used to access the underlying data. That means every report viewer needs permission to read the lakehouse, which brings back the problem from Gotcha #1. The fix is to configure the model's data source connection to use a fixed identity through a shareable cloud connection (a service principal or a dedicated account). Then: The fixed identity reads the Delta tables. End users only need Read on the semantic model. Your DAX RLS still applies per user, because USERPRINCIPALNAME() still returns the viewer, not the fixed identity. The catch: once you switch to fixed identity, any RLS you defined on the SQL analytics endpoint (T-SQL security policies) is evaluated against the fixed identity, not the end user. Warehouse-level RLS effectively stops being per-user for this model. So pick one layer as the source of truth: Approach Security lives in Connection Users need lakehouse access? A Semantic model (DAX roles) Fixed identity No B SQL endpoint (T-SQL RLS) SSO Yes We chose A. Mixing both led to confusing results where the same user saw different numbers depending on the path the query took. Gotcha #3: SQL endpoint RLS quietly forces DirectQuery fallback We initially tried approach B, because the data engineering team liked having security defined once in T-SQL: CREATE FUNCTION sec.fn_region_filter(@RegionKey INT) RETURNS TABLE WITH SCHEMABINDING AS RETURN SELECT 1 AS allowed FROM dbo.sec_user_region s WHERE s.RegionKey = @RegionKey AND s.UserEmail = USER_NAME(); CREATE SECURITY POLICY sec.RegionPolicy ADD FILTER PREDICATE sec.fn_region_filter(RegionKey) ON dbo.fact_sales WITH (STATE = ON); Security worked. Performance got much worse. With Direct Lake on the SQL endpoint, any table with RLS (or OLS) defined at the SQL endpoint can't be read directly from Delta, because VertiPaq can't enforce the T-SQL policy. Queries against it fall back to DirectQuery. Our main visuals went from about 300 ms to 4–8 seconds. It fails quietly. Nothing errors. Reports just get slow. How to detect it: Run a trace in DAX Studio or SQL Profiler against the XMLA endpoint and look for DirectQuery Begin/End events. In pure Direct Lake you should see only VertiPaq scan events. In Performance Analyzer in Desktop, a "Direct query" duration on a visual is a clear sign of fallback. How to make it fail loudly instead: set the model's DirectLakeBehavior property (via Tabular Editor or TMDL) to DirectLakeOnly: model Model directLakeBehavior: directLakeOnly Now fallback-triggering queries error instead of silently degrading. We set this in dev and test so we find these issues before users do. In production you may prefer Automatic so reports keep working, but you should make that choice deliberately. Note: Direct Lake on OneLake (the newer flavour) reads Delta directly and doesn't use the SQL endpoint for queries, so SQL endpoint RLS isn't applied there at all. That's another reason not to rely on T-SQL RLS for Direct Lake models. Check the current docs on how OneLake security interacts with it, because that area is still changing. Gotcha #4: Your security table can't be a view or a calculated table In Import mode, our security table was a calculated table that unpivoted a hierarchy of managers and regions using DAX. In Direct Lake (on SQL endpoint), calculated tables and calculated columns over Direct Lake tables aren't supported. The obvious next step is a SQL view. But SQL views in a Direct Lake on SQL endpoint model always fall back to DirectQuery, because they aren't Delta tables. Your security table sits in the filter path of every query, so it pulls every query into DirectQuery with it. What worked: materialise the security table as a real Delta table in the lakehouse, rebuilt by a notebook in the same pipeline that loads the facts: from pyspark.sql import functions as F sec = ( spark.table("stg_user_access") .withColumn("UserEmail", F.lower(F.trim("UserEmail"))) .select("UserEmail", "RegionKey") .dropDuplicates() ) sec.write.mode("overwrite").format("delta").saveAsTable("sec_user_region") Keep it narrow (two columns) and deduplicated. The next gotcha explains why the lower() matters. Gotcha #5: Case sensitivity changes between Direct Lake and fallback This one took us a day to track down. A user could see data in one report page but got blanks on another. Same model, same role. The cause: USERPRINCIPALNAME() returned [email protected]. The security table had [email protected]. In Direct Lake (VertiPaq), string comparison is case-insensitive, so it matched. One page had a visual that fell back to DirectQuery (it hit a guardrail). In DirectQuery, the RLS filter is pushed down as SQL, and the Fabric SQL endpoint's default collation is case-sensitive, so it didn't match. Same user, same role, different result depending on the query path. Fix: normalise on both sides. // Role filter on sec_user_region [UserEmail] = LOWER ( USERPRINCIPALNAME () ) Also lowercase the data during ingestion (see the notebook above). Don't rely on the engine's collation. Gotcha #6: RLS changes don't take effect until the model reframes We removed a departing manager from sec_user_region, the notebook ran, and the Delta table was updated. He could still see his old regions two hours later. Direct Lake doesn't read "live" Delta. It reads the version of the table captured at the last framing operation (a refresh). If you've turned off "Keep your Direct Lake data up to date" (common when you want facts and dimensions to update together), the model keeps using the old security table until the next refresh. Security changes follow your refresh schedule, not your data load. What we do now: The pipeline that rebuilds sec_user_region ends with a semantic model refresh activity, so the model reframes immediately. For urgent revocations such as terminations, we trigger an on-demand refresh and don't wait for the schedule. We added a small "Security last refreshed" card to the report, driven by a LastUpdated column in the security table, so support can check it quickly. Gotcha #7: "Test as role" isn't the same as a real user Testing with Security → Test as role in the service, using "Other user" and typing an email, is useful but has limits: Under SSO, data access still happens with your identity. You may see rows the real user can't access at the lakehouse level, or the reverse. It doesn't show you workspace-role bypass (Gotcha #1), because you're testing the role, not the user's actual permissions. We now add a hidden debug page to every RLS model: Debug_UPN = USERPRINCIPALNAME () Debug_RegionCount = COUNTROWS ( VALUES ( dim_region[RegionKey] ) ) Debug_SecRows = COUNTROWS ( sec_user_region ) For UAT, real test accounts (not admins) open the app and screenshot this page. It shows exactly what UPN the engine sees, which also helped with a B2B guest user whose UPN didn't match the email in our HR feed. Always compare against what USERPRINCIPALNAME() actually returns, not what you expect it to return. Gotcha #8: Large security tables affect memory and cold-cache performance Direct Lake loads columns into memory on demand. The security table and its relationship columns are touched by every query for every user under that role. Our first version of sec_user_region expanded the org hierarchy down to store level: about 2.1 million rows. After a reframe, the first visual for each user was noticeably slow because the security columns had to be transcoded, and memory pressure caused more column evictions under load. Collapsing it to region level (about 3,800 rows) and moving store-level granularity into the dimension fixed the problem. Tips: Filter at the highest grain that meets the business requirement. Avoid high-cardinality string keys in the security relationship; use integer keys. Keep bidirectional filtering limited to the one security relationship, not across the whole model. Consider a warm-up query (a scheduled DAX query after refresh) for large models so the first user of the morning doesn't pay the cold-cache cost. Our checklist now Before any Direct Lake model with RLS goes to production, we check: End users have no workspace role and access through an App or item sharing only End users have no direct lakehouse / SQL endpoint access Connection uses a fixed identity (if security lives in the model) No T-SQL RLS/OLS on tables used by the model (or fallback is intentional) Security table is a physical Delta table, not a view or calculated table Emails are lowercased on both sides of the comparison DirectLakeBehavior = DirectLakeOnly in dev/test to surface fallback Security table rebuild triggers a model refresh Debug page verified by real non-admin test accounts, including guests Security table kept small, integer-keyed, and at the coarsest grain possible Final thoughts The RLS logic in Direct Lake is the same as in Import mode, so it's easy to assume nothing else changes. What does change is the environment around it: permissions are shared with the lakehouse, security can exist in two layers, data only updates when the model reframes, and queries can fall back to DirectQuery without warning. Most of our problems came from those areas, not from DAX. If you're planning a migration, test RLS with real user accounts early, run a trace to check for fallback, and decide at the start which layer owns security. Have you run into other RLS issues with Direct Lake? Please share them in the comments. I'm especially interested in how people are handling OneLake security alongside model RLS as that feature matures.The Fabric Blueprint: Architecting Workspaces for Enterprise Success
This quick guide establishes the organizational standard for managing Microsoft Fabric within our enterprise environment. By adopting these patterns, we ensure security, maintainability and streamlined CI/CD deployments across all data projects.Beyond the Basics: Deep Dive into Microsoft Fabric Row-Level Security (RLS) & Purview Labelling
“Securing your data isn’t about locking it away – it’s about ensuring the right people see the right insights without compromising the rest.” In our previous beginner’s guide, we explored how to establish baseline governance, structure your workspaces and set up your core Admin Portal guardrails. If you fancy reading the previous beginner blog, you can refer it here. But once your data estate is up and running, you inevitably run into a critical next challenge: How do you handle data when different people are allowed to see different subsets of information within the very same table? Moving from basic visibility to active, granular protection is the hallmark of Stage 2 on the Governance Maturity Curve. In this deep dive, we’ll explore how to implement Row-Level Security (RLS) and enforce Microsoft Purview sensitivity labels through a visual journey.
