Fabric Warehouse Git commit fails with “Maximum commit size exceeded” potentially due to XMLA.json
Based on the Fabric Warehouse Git integration documentation, the XMLA.json file should be excluded from Git integration workflows. Fabric should exclude this file from commits and updates to prevent the default semantic model metadata from being unintentionally stored in Git. [learn.microsoft.com] However, it appears that XMLA.json, or the associated semantic model metadata, may still be getting processed on the Fabric side while calculating or preparing the commit. This potentially causes the commit to exceed the maximum allowed size. Any One Face Same Issue, how to avoid above issue & prevent XMLA.json file from getting processes while git commit ?
Direct Lake Semantic Model - RANKX Performance Issues on Large Datasets
Hi Community, We are facing performance challenges with RANKX measures in a Microsoft Fabric Direct Lake semantic model and would like to understand whether this is expected behavior or if there are optimization techniques available. Current DAX Measure: ```DAX YTD Rank Region = RANKX( ALLSELECTED('Sales Region Master'[Region_Name]), CALCULATE( [% Sales YTD], ALLEXCEPT( 'Sales Region Master', 'Sales Region Master'[Region_Name] ) ), , DESC, DENSE ) in above measure, % Sales YTD is created on 2 tables, target table and achievement table. Both tables are in relationship with Calendar Date and Sales Region master. Environment: Direct Lake semantic model Large fact tables (millions of records) Multiple slicers and visuals Several similar ranking measures across the model Issue: Visuals using rank measures are taking significantly longer to load compared to other calculations. Performance worsens when additional filters or slicers are applied. DAX Query View indicates that the rank calculations consume most of the execution time. Questions: Is RANKX inherently expensive in Direct Lake semantic models compared to Import mode? Does ALLSELECTED() contribute significantly to performance overhead in Direct Lake? Are there recommended alternatives such as: Pre-calculated ranks during ETL Aggregation tables Composite/Import models Has anyone successfully optimized similar ranking scenarios in Direct Lake semantic models? Are there Fabric-specific best practices for implementing ranking measures over large datasets? Any insights, recommendations, or real-world experiences would be greatly appreciated. Thank you.
Direct Lake Support for Private Workspace FQDN Connections in VNet-Restricted Microsoft Fabric
Hi Microsoft Team, We are seeking clarification on Direct Lake support for private workspace connections in Microsoft Fabric. From our understanding, Direct Lake semantic models should be able to connect to OneLake-backed data sources. We are specifically investigating whether Direct Lake supports connections using privately resolvable workspace fully qualified domain names (FQDN) in a VNet-restricted architecture. Current Infrastructure Setup: Private Link configured in VNet-restricted Gold Workspace Managed private endpoint enabled in Reporting Workspace Inbound networking traffic restricted in Gold Workspace Currently operational: Direct Lake connection using public OneLake URL (onelake.dfs.fabric.microsoft.com) routed through managed private endpoints What We Tested: Registered a warehouse in a VNet-restricted Gold Workspace Created a new connection using workspace FQDN with service principal authentication in the public Reporting Workspace Successfully validated the connection (connection test passed) Attempted to bind this FQDN connection to a Direct Lake semantic model Binding operation failed with error Error Observed: "Direct Lake On OneLake requires a valid OneLake data source. Please verify and fix the data source definitions." Questions: Is Direct Lake architecturally designed to work only with public OneLake URIs (onelake.dfs.fabric.microsoft.com), or is there a supported way to use Direct Lake with privately resolvable workspace FQDNs? Does Direct Lake support connections through private workspace endpoints in a VNet-restricted scenario without reverting to the public OneLake URL? If private workspace FQDNs are not currently supported for Direct Lake, are there any planned features or documented workarounds for this scenario? We would appreciate any guidance on supported connectivity patterns or known limitations for this architecture. Thank you.Microsoft Fabric Security Policies: How They Strengthen Data Security and Governance?
In today’s data-driven world, governance isn’t just about compliance — it’s about enabling trust, consistency, and scale. As organizations adopt Microsoft Fabric to unify their analytics, the challenge of managing who can do what, where, and under which conditions become critical. That’s where Microsoft Fabric policies step in. Think of them as the rulebooks that let administrators define guardrails across workspaces, security groups, and sensitivity labels. Instead of relying on manual oversight, policies provide a centralized, automated way to enforce governance — ensuring that innovation doesn’t come at the cost of control. In this blog will walk you through the key concepts, the entry points for creating and activating policies, and the end-to-end lifecycle that makes governance in Fabric both powerful and practical. Prerequisites: 1) A Valid Fabric Trail / Capacity License 2) A Fabric workspace assigned to a Capacity/Fabric Trail. 3) Proper admin role to activate the policy a) Fabric administrator: Can create and activate policies for tenant-scope policies. b) Capacity administrator: Can create and activate policies for capacity-scope policies. 4) Contributor, Member, or Admin access to the workspace so that you can create a policy item type. 5) The Policies tenant setting is enabled in Admin settings 6) Valid Security groups in AAD. Step 1: Enable the Policies feature in the admin settings Go to fabric admin settings, search for polices(preview) and enable the setting and click on apply. Step 2: Creating Policy item Create a workspace ‘Fabric-Polices-Ws’ which was assigned to ‘Fabric Trail’ and I’m (SI Babu) who’s the capacity admin. Make sure this should be a secured, admin-managed workspace to store policies. Click on ‘New Item’ and select ‘Policy set (preview)’ under ‘Govern data’ In New policy set dialog, enter a unique and descriptive Name ‘Capacity_PolicySet_1’ for the policy set and choose ‘ A capacity’ under policy scope then Click on ‘Create’ Create another policy set ‘Tenent_Policy_set2’ and choose ‘ Fabric Tenant’ under policy scope then Click on ‘Create’ Two policy sets are created. Step 3: Activating the Policy set Open Capacity policy set and select the ‘Active’ from Status Select the capacity and click on ‘Assign + activate’ Policy activated successfully Open Tenant policy set and select the ‘Active’ from Status Click on ‘Activate’ the policy Policy activated successfully Step 4: Create / Add Policy Rules(Capacity Policy) Click on ‘OneLake catalog’ Click on ‘Govern’ à‘Polices(preview)’ àCapacitiesàExpand the ‘Users can create items’ and select ‘Advanced’ Here, Select Allowed, Blocked, or Advanced (Set by user) to change the policy behavior. If you select Advanced, the rules configuration is enabled. Add a new rule with good name, description and Select ‘item type’ in Condition type, operator is ‘Is any of’ the select ‘Lakehouse’ and ‘Dataflow gen2’ then click on ‘+Add condition’ Choose ‘workspace’ in condition type, operator is ‘Is any of’ selected two workspaces gen2’ then click on ‘+Add condition’ Choose ‘Security group’ in condition type, operator is ‘Is any of’ selected my ‘Data Analyst’ AD group. Click on Apply Note: In this Data Analyst group we have three developers are there Click on “Save’ Now Policy successfully added Step 5: Validate policy enforcement (Capacity Policy) Added ‘Data Analyst’ AD group to both the workspaces ‘FabricRTI Training’ & ‘PowerBI-Demo-WS’ Now I was opened another window with account(SP Babu) who’s part of AD group & you can see two workspaces Create a ‘Notebook’ Now when I click on “Create” it’s throwing an error as you seen in below screenshot. Means it enforced a policy Again, When I was trying to create ‘Warehouse’ it’s throwing the same error. I (SP Babu) was able to create ‘DataFlowGen2’. Admin enforced to create only this item I (SP Babu) was able to create ‘Lakehouse’. Admin enforced to create only this item Step 6: Create / Add Policy Rules(Tenant Policy) Create another policy should be at tenant level to restrict workspace Outbound access protection and inbound network rules and click ‘add condition’ Add “Security group’ as I have IT Infra Team and click Apply Note: In this IT Infra Team group we have three resources are there Security policy created successfully Step 7: Validate policy enforcement (Tenant Policy) Now SP Babu unable to edit ‘Outbound access’ it’s showing ‘This setting is managed by a tenant policy.’ SP Babu unable to edit ‘Inbound access’ it’s showing ‘This setting is managed by a tenant policy.’ You can deactivate or replace an active policy set from the workspace where the policy set is stored. Only an administrator of the policy set's admin scope can deactivate or replace it. Best practices for creating a policy: Use a secured, admin-managed workspace to store policies. Store all policies for the admin scopes managed by the same administrator or admin team in the same workspace. This approach provides one location for tenant- and capacity-scoped policies and simplifies access management, discovery, and lifecycle management. Use clear policy set and rule names that identify the admin scope and purpose. Use test users who are members and non-members of the configured security groups to validate both allow and deny outcomes before broader rollout. Limitations as of now: Fabric policies aren't currently supported in the following capacity regions: West Europe, North Europe, and West US. We can currently create policy sets only from the item creation experience in a workspace. Only workspaces assigned to a Fabric capacity can store policy sets. We can't change a policy set's admin scope after creation. Only one policy set can be active for an admin scope at a time. Policy and tenant-setting changes can take time to propagate. Conditions support Any of and None of in the current preview. We can't activate a policy set through Git. So, start small with pilot policies, monitor audit logs for adoption and Communicate changes to workspace owners Conclusion Microsoft Fabric Policies are still in preview, but they signal a future where governance is not a blocker — it’s an enabler of responsible innovation. Happy learning! — Inturi Suparna Babu [LinkedIn]
PBI Report error for end user with OneLake security restrictions
Hello folks, we are having problems with end users not being able to open a report based on a semantic model in Direct Lake mode where the underlying Lakehouse defines OneLake security. Setup: 1. Lakehouse (LH) with OneLake security roles defined - there are multiple roles defining OLS and RLS, but never CLS 2. Semantic Model (SM) which connects to tables in LH over DirectLake on OneLake mode 3. Report based on SM What we experience is that an end user with restricted access to data accessing the report is greeted with the following pop-up error: (I will add it here in text so this question shows up when people search for the error): "The combination of active roles results in a dynamic security configuration that is not consistent. Please refresh the dataset, and if the issue still persists, contact the administrator of the database to resolve this issue." We have read the documentation on Direct Lake security and made sure to especially take care of the following points: - the end user is only member of one OneLake security role in the LH - there are no active bidirectional relationships in the model - the RLS predicates defined on the tables all take the form WHERE col = '<static_string_value>' Some additional info about our current setup: - we have switched the SQL analytics endpoint of LH over to "Use OneLake security for tables (User's identity access mode)" and the end user is able to query all permitted tables with SQL and have their RLS restrictions applied correctly - the tables in LH are materialized lake views - the end user has read access to report, semantic model, and lakehouse Any help is greatly appreciated. Thanks Patrick
When the data is centralized, but the KPI still does not match
A common assumption with a Fabric implementation is that once data from SAP, operational systems, SharePoint, or flat files lands in one Lakehouse, reporting consistency becomes much easier. The storage problem gets easier. The KPI problem does not necessarily disappear. In one enterprise reporting implementation, data was coming from SAP S/4HANA, SAP FSM, SharePoint, and flat files. The reporting environment had grown around these separate systems, so similar business metrics were being calculated differently depending on the source. The approach was to move the data through a Bronze–Silver–Gold structure in the Fabric Lakehouse and standardize the transformation logic before exposing the data to Power BI. The division of responsibility was important: Bronze: Preserve source data with minimal transformation. Silver: Clean, conform, and align entities coming from different systems. Gold: Expose business-ready datasets and standardized KPIs for reporting. Then Power BI handled the semantic layer, including DAX measures and role-based access. This helped us move business logic away from individual report files and toward a more governed data and semantic architecture. One lesson from this kind of setup: A single truth source does not mean a single storage location. It also needs a consistent definition of what "Revenue", "Utilization", "Downtime", "Productivity", or any other business KPI actually means. How are you handling this in any recent Fabric projects? Do you keep KPI definitions and transformation rules in a metadata layer, or do you primarily maintain them in the Gold layer and the Power BI semantic model?Public Parameter value passed from Fabric Pipeline is not reflected in Dataflow Gen2 at runtime
Hi Fabric Community, I am trying to pass a Pipeline Run ID from a Microsoft Fabric Pipeline to a Dataflow Gen2 using a Public Parameter, and then store that Run ID as a technical column in the Silver layer. My overall architecture is as follows: Fabric Pipeline ↓ Load Marketo raw data into Bronze Lakehouse / Files ↓ Dataflow Gen2 (Read and transform the Bronze data for the Silver layer) ↓ Save the transformed data into a Silver Lakehouse Table I would like to add the Pipeline Run ID to each record in the Silver table as a _run_id column, so that I can trace which Pipeline execution processed each record. Dataflow Gen2 configuration I created the following parameter in Dataflow Gen2: Parameter name: p_run_id Type: Text I also enabled: "Enable parameters to be discovered and overridden for execution" in the Dataflow options. The Manage Parameters dialog confirms that Public parameter mode is enabled. In the Dataflow query, I added a custom column as follows: Column name: _run_id Formula: p_run_id I also confirmed that this transformation step is not configured as a Preview-only step. Fabric Pipeline configuration I added a Dataflow activity to the Fabric Pipeline and configured the Dataflow parameter as follows: p_run_id = @pipeline().RunId When I run the Pipeline, the Dataflow activity input shows that the Pipeline Run ID is being passed correctly. For example: "parameters": { "p_run_id": { "value": "c8d21095-5329-42c4-8c97-d9f3a3f141d5", "type": "String" } } The Dataflow activity itself also completes successfully. Problem However, when I check the data written by Dataflow Gen2 to the Silver Lakehouse table, the Pipeline Run ID is not populated in the _run_id column. To isolate the issue from the actual Marketo data transformation, I also created a very simple test query containing only one test row and added: _run_id = p_run_id I then wrote this test query to a Lakehouse table. The same issue occurred. I also tested the behavior of the Current value of p_run_id: If the Current value is blank, _run_id is blank in the output table. If I set a fixed Current value, that fixed value appears in _run_id. However, the value passed from the Pipeline does not override the Current value. Therefore, it appears that the Dataflow can reference p_run_id, and the Pipeline activity input contains the correct runtime parameter value, but the Public Parameter is not being overridden at runtime. Instead, Dataflow Gen2 seems to use the parameter's Current value. Could anyone please advise on the following? Is there any known scenario where the Dataflow activity input contains the correct parameters.p_run_id.value, but the runtime value is not actually applied inside Dataflow Gen2? Is there any additional configuration required to pass a Public Parameter from a Fabric Pipeline to Dataflow Gen2 beyond the settings described above? Is there a way to verify which Public Parameter value Dataflow Gen2 actually received or evaluated during a specific execution? Is this a known limitation or issue with Public Parameters in Dataflow Gen2? If so, is there a recommended workaround? My goal is to pass the Fabric Pipeline Run ID to Dataflow Gen2 and store it in the Silver Lakehouse table as the _run_id technical column. Any guidance would be greatly appreciated. Thank you!
I Let Fabric's New Data Engineering Agent Run My ETL: Here's What Happened
Last week I handed Microsoft Fabric's new Data Engineering Agent a messy ETL job that took me 11 hours to build by hand last quarter. It had a working, validated pipeline in front of me in under an hour. I still changed 3 of its 9 steps before I'd let it near anything important. That is the short version. The long version is more interesting, because the places where it went wrong tell you exactly how to use it well. Some context first. At FabCon Europe 2026 in Barcelona (September 28 to October 1), Microsoft announced the Data Engineering Agent in preview. It can plan and run migrations, ETL and performance work under guardrails that engineers define. It builds on technology from Osmos, the agentic data engineering startup Microsoft acquired in January 2026, and was demoed on stage as "Project Osmos". Copilot has been helping us write code for a while now. This is a different idea. You describe the outcome, and the agent does long-running, multi-step work on its own while you supervise. I wanted to know whether that holds up on a real, slightly ugly dataset rather than a keynote demo. So I tried it. What the Data Engineering Agent actually is The agent is an autonomous worker inside your Fabric workspace, not a chat assistant that suggests code. You give it a goal, and it does the multi-step work itself. Its roots are in Osmos, a Seattle startup whose "AI Data Engineer" already generated production-grade PySpark notebooks inside Fabric. Microsoft bought the company in January 2026 and folded the team into Fabric engineering. Based on Microsoft's FabCon session on Project Osmos, a typical run looks like this: Read the request. You describe the outcome, even loosely ("land the orders CSVs in a clean silver table"). Inspect the workspace. It looks at your existing code, Lakehouses and data layout before writing anything. Plan. It breaks the job into steps and can hand parts to sub-agents for long-running work. Build. It creates and updates notebooks and other Fabric items. Validate. It checks its own output so you can trust what was built. Hand back to you. A human reviews and approves. Microsoft is explicit that supervision stays part of the process. The headline use cases are migrations, ETL and performance tuning. I chose ETL, because that is where most of us spend our week. The test: a real pipeline, not a demo I picked a job I had already built by hand, so I could judge the agent's work line by line against my own. The data is sales orders from a mid-sized retail client . Every day their order system drops a CSV export into a Lakehouse folder. The business wants a clean orders table for analysts and a daily sales summary for a Power BI report. Item My setup Source data 14 daily CSV exports, 18.4 million rows, about 2.1 GB, in Files/raw/orders/ Target silver_orders (cleaned Delta table) and gold_daily_sales (aggregate by day, store and category) Known traps Duplicate order IDs from system retries, two date formats, a currency column with nulls, cancelled orders flagged as negative quantities Capacity F64 trial capacity, sandbox workspace Hand-built baseline About 11 hours across two days, including testing The exact prompt I gave it: Ingest the daily order CSVs in Files/raw/orders into a cleaned silver Delta table called silver_orders. Remove duplicates, standardise dates and currency, and exclude cancelled orders. Then build gold_daily_sales with total revenue, order count and average basket size by day, store and category. The pipeline must run incrementally each day. I deliberately kept the prompt a little vague. I didn't say which date formats exist, how cancellations are encoded, or which column identifies a duplicate. In real life, requirements rarely arrive as a perfect spec. I wanted to see whether the agent would investigate, ask, or just guess. What happened, step by step 1. It read the workspace before writing anything The first thing I noticed: it didn't start coding. For the first four minutes it profiled the data. It sampled each CSV, inferred the schema and looked at my existing notebooks in the workspace. It came back with a short summary of what it found. Two findings impressed me: It spotted both date formats (2026-09-14 and 14/09/2026) and flagged that files from 9 September onwards switched format. It noticed my existing notebooks used a nb_<layer>_<entity> naming pattern and said it would follow it. It missed one thing, which matters later: it reported quantity as "integer, some negative values" without asking what negative quantities meant. 2. The plan it proposed Next it produced a 9-step plan and stopped for my approval: Create a staging notebook to read all CSVs with an explicit schema Normalise both date formats to a single order_date column Fill null currency values from the store's default currency Deduplicate on order_id, keeping the latest record Remove cancelled orders Write silver_orders as a Delta table, partitioned by order_date Build gold_daily_sales with the three requested measures Add incremental logic based on the source file date Run validation checks and produce a summary report This is a sensible plan. It is roughly what I wrote on a whiteboard last quarter. But reading it closely, I changed three things before approving: Step 4: it planned to overwrite silver_orders on each run. I changed it to a MERGE on order_id, because late-arriving corrections are common in this data. Step 5: it planned to remove cancellations using a status column. That column exists, but this client never fills it. Cancellations show up as negative quantities. I told it so. Step 8: it wanted to detect new data by file modified time. I asked it to use the date in the file name instead, which is more reliable when files are re-uploaded. It accepted all three changes and rewrote the plan in a few seconds. 3. The build Once approved, it worked for 38 minutes without needing me. It created three notebooks (nb_stg_orders, nb_silver_orders, nb_gold_daily_sales) and a small pipeline to run them in order. The code was clean and readable. Here is its date handling, which is close to what I would write myself: from pyspark.sql import functions as F df = df.withColumn( "order_date", F.coalesce( F.to_date("order_date_raw", "yyyy-MM-dd"), F.to_date("order_date_raw", "dd/MM/yyyy") ) ) And the merge into silver, after my change in the plan: from delta.tables import DeltaTable silver = DeltaTable.forName(spark, "silver_orders") (silver.alias("t") .merge(updates.alias("s"), "t.order_id = s.order_id") .whenMatchedUpdateAll() .whenNotMatchedInsertAll() .execute()) It asked me one question mid-way: whether average basket size should be revenue per order or items per order. That was exactly the right question, and one I had to ask the client myself last time. 4. Self-validation When the build finished, it ran its own checks and produced a short report: Row counts reconciled from raw to staging to silver, with each dropped row explained (duplicates, cancellations) No nulls left in order_date, currency or store_id No duplicate order_id values in silver_orders Daily totals in gold_daily_sales matched a direct aggregate of silver_orders One check failed on the first pass. 41,200 rows had a null order_date after parsing. It traced the cause itself: a third date format (14-Sep-2026) appeared in two of the older files, which its initial sampling had missed. It added a third pattern to the coalesce, re-ran, and the check passed. That moment sold me on the validation step. A human could easily miss 0.2% of rows quietly disappearing. 5. My review against my own version I compared its output with the pipeline I built by hand last quarter, running both on the same 14 files. Measure Agent Me, by hand Time to working pipeline 52 minutes (incl. my review) About 11 hours Plan steps I changed 3 of 9 Not applicable Data issues found 4 of 4, after one self-correction 4 of 4 Full run time on 18.4M rows 6 min 40 s 5 min 55 s Silver row count 17,962,118 17,962,118 The row counts matched exactly. My hand-built version ran about 45 seconds faster, mostly because I had tuned partitioning. More on that below. Would I merge its code as-is? After my three plan changes, yes, with one small performance fix. Guardrails: the part you shouldn't skip An agent that can create and change items in your workspace needs limits before it needs a prompt. Microsoft designed this agent around guardrails that engineers define, and setting them up was the best 15 minutes of the whole test. What I set up before the run: A sandbox workspace. It's a preview feature, so it ran on a copy of the data, nowhere near production. Scoped data. It could read Files/raw/orders/ and write only to new tables. My existing tables were off limits. Approval points. It had to stop for my approval after the plan, and again before the first write to any Delta table. Git in the loop. The workspace was connected to a Git repo, so every notebook it created or changed landed as a diff I could review line by line. In practice, it never tried to step outside these limits. The approval after the plan turned out to be the most valuable one: that is where I caught all three of my changes. If you set only one guardrail, make it that one. Where it struggled No agent review is credible without the failures, so here are the three that mattered. It assumed instead of asking about business rules. It saw negative quantities and planned to use an empty status column for cancellations. Had I approved the plan without reading it, about 310,000 cancelled orders would have flowed into the gold table and inflated revenue. It asked a good question about basket size, but not this one. Its sampling missed a rare date format. The third format appeared in only two files. Its validation caught the problem, but only after the first run. On a much larger dataset, that first run costs real capacity. Correct, but not tuned. It partitioned silver_orders by order_date, which produced many small files for this data volume. Switching to monthly partitions and running OPTIMIZE closed most of the 45-second gap with my version. When I asked it to "make the silver write faster", it suggested the same fix, but it didn't do it unprompted. Three caveats worth knowing before you try it: It's a preview. Features, behaviour and pricing can change before general availability. Don't build production processes on it yet. Watch your capacity. Long-running agent work consumes capacity units on top of the Spark jobs themselves. Check the Capacity Metrics app after your first run so you know what a typical job costs. It doesn't know your business. It can read your schema, not your meeting notes. Every rule I wrote into the prompt was handled correctly. Every rule I left out was a guess. The verdict The Data Engineering Agent is like a fast, tireless junior engineer: excellent at the mechanics, careful about checking its own work, and still in need of a senior reviewing the plan. What I'd tell my team after this test: Spend your time on the prompt and the plan, not the code. Every problem I found was visible in the plan before a single line ran. Write business rules down explicitly. Cancellations, duplicates, currency defaults. If it's tribal knowledge, the agent will guess. Trust the validation step, but read its report. It caught a data loss bug I might have missed myself. Tune afterwards. Treat its first version as correct, then ask it for performance improvements. Try it now if you have a sandbox workspace, a backlog of repetitive ingestion or migration jobs, and someone who can review PySpark. Wait for GA if you need it in production, your pipelines are full of undocumented business logic, or your capacity budget is tight. The bigger shift is in our role. Going from 11 hours to under an hour didn't remove me from the work. It moved my effort from typing code to writing clear requirements, setting guardrails and reviewing output. That is a skill worth building now, preview or not. Have you tried the Data Engineering Agent yet? Tell me in the comments what you threw at it, and where it surprised you. Sources FabCon Europe 2026 announcements (Kanerika) From Prompt to Finished Project: Introducing Project Osmos (FabCon Europe 2026 session) Microsoft Acquires Osmos (Redmond Magazine) Fabric January 2026 Feature Summary (Microsoft Fabric Blog)
