Fabric - VS Code Extension - Conflict Handling
Hi, I've been testing out the extension and whenever I there is a conflict, I'll be prompt to either pick the Local version, Virtual Workspace version or the Compare and Merge. It has been quite annoying when I pick Compare and Merge as I'm trying to compare and decide which one I would like to proceed. But apparently when you click on Compare and Merge, it allows you to compare but you're required to merge. I'm unable to cancel the merge if I try to. Closing the tab doesn't solve the issue. It will only say in Conflict mode for the notebook selected. Is there any explanation or fix for this issue? Thanks
Fabric pipelines audit
Hi community, I am looking for a way to log pipelines details into an audit delta table. To be more specific, I want to log the start time, end time, status, error message if any etc. Having this by activity name would be even better. I was thinking to use a notebook that's being called at the end of each pipeline, so somehow to have only a notebook that can be used in all of my pipelines (because i have a lot of them). Is this something that you were able to build? Any guidance would be very much appreciated. thanks!
How to design a metadata-driven framework to run 600+ Fabric notebooks with parallel execution?
Hi Fabric Community, We are currently planning to migrate 600+ notebooks from Azure Databricks to Microsoft Fabric. Our current requirement is to build a metadata-driven orchestration framework using: Fabric Data Pipelines for orchestration SQL Database for metadata/control tables Fabric Notebooks for data processing OneLake/Lakehouse as the target storage Instead of creating separate pipelines for each notebook, we would like to have a generic metadata-driven pipeline that dynamically reads notebook information from SQL Database and executes the required notebooks. For example, our metadata table could contain: Notebook ID Notebook name/path Priority Dependency Parameters Data volume Expected runtime Compute/pool requirement Retry count Active/inactive flag Our main challenge We may need to execute approximately 40 notebooks in parallel. We would like to understand the recommended Fabric architecture for this scenario. Specifically: How should we design the metadata-driven pipeline to dynamically execute 600+ Fabric notebooks? How should we control parallelism when around 40 notebooks need to run simultaneously? Should we use a single Spark environment/pool, multiple environments/pools, or some other approach? How should we decide which Spark compute configuration should be used for each notebook based on: Data volume Memory requirement Processing time Shuffle-intensive workloads Small vs. large workloads? If 40 notebooks run simultaneously, how does Fabric manage the underlying Spark compute and capacity? Should we be concerned about resource contention or throttling? Is it recommended to classify notebooks into workload groups such as:and then control concurrency separately for each group? Small / Medium / Large / XLarge What is the recommended way to implement dependency management? For example:where Notebook B should only start after A succeeds. Notebook A → Notebook B → Notebook C What is the recommended approach for retry, failure handling, logging, and restartability for 600+ notebooks? Is there a recommended metadata-driven orchestration pattern/reference architecture in Microsoft Fabric for this scale? Are there any Fabric-specific limitations or best practices we should consider when running hundreds of Spark notebooks with high concurrency? Our main objective is to build a scalable framework where we can manage 600+ notebooks from metadata instead of maintaining hundreds of individual pipelines, while still controlling Spark compute and parallel execution efficiently. Any architectural recommendations, reference implementations, or real-world experience with similar workloads would be highly appreciated. Thanks!
How are AI agents changing modern data engineering workflows?
I am exploring how AI agents can assist data engineering teams in managing complex data workflows and reducing repetitive operational tasks. Some areas I am interested in: AI agents for monitoring data pipelines and detecting failures Automated data quality checks and anomaly detection Intelligent assistance for ETL/ELT workflow optimization Generating documentation for datasets and transformations Using AI with Fabric pipelines, notebooks, and data workflows With the growth of AI-powered automation, I would like to understand how data engineers are approaching these patterns in real-world environments. What approaches, architectures, or best practices are teams using to combine Microsoft Fabric capabilities with AI agents?
SharePoint Excel ingestion, Dataflow Gen2 vs shortcut + notebook CU efficiency
I need to ingest and transform multiple Excel files stored in SharePoint folders. I am comparing: Pure Dataflow Gen2 using SharePoint.Files and Power Query transformations. Lakehouse Files shortcut to the SharePoint folder, followed by all transformations in a Fabric notebook using PySpark. For the same files, transformations, output, schedule, and capacity: Is the shortcut + notebook approach expected to consume fewer Fabric Capacity Units than pure Dataflow Gen2? What is the recommended approach for SharePoint Excel-folder ingestion?
Cross-tenant Fabric migration (no shared account): 6 things the docs don't tell you
Sharing this rather than asking — posting my notes in case they save someone else the time. I recently had to move semantic models, reports, notebooks, pipelines and a Lakehouse between two Fabric tenants where no single account had access to both — no guest access, no cross-tenant permissions, two entirely separate sign-ins. Deployment pipelines are same-tenant only. fabric-cicd assumes one tenant and a Git repo. So this became an export-to-file / import-from-file exercise against the REST API. Six things cost me real time. Posting them in case they save someone else the same days. A semantic model's connection lives in FOUR places in model.bim I started by rewriting the M expressions and the log said "0 expressions repointed" while the model kept pointing at the old source. The connection can sit in the M code (Sql.Database(...)), in a dataSources[] entry's connectionDetails.address, in Direct Lake entity partitions, and in M parameters with a meta annotation. Rewriting only one of them silently does nothing. I ended up operating on the raw model.bim JSON text so all four are covered. PBIR report binding changed shape between schema versions definition.pbir validates differently depending on its major version. Schema v2.0+ wants connectionString and nothing else — adding pbiServiceModelId gets you "the schema does not allow additional properties". Schema 1.x wants the full legacy set: pbiServiceModelId, pbiModelVirtualServerName ("sobe_wowvirtualserver"), pbiModelDatabaseName, name: "EntityDataSource", connectionType: "pbiServiceXmlaStyleLive". Omit them and you get "Cannot resolve neither report.json nor the PBIR report content in enhanced format". Preserve the original $schema and version from the source file and branch on the major version. Don't assume one shape. The Lakehouse SQL analytics endpoint is read-only over Delta CREATE VIEW, CREATE FUNCTION and CREATE PROCEDURE all work. CREATE TABLE does not — tables only come from Spark. If you're recreating a Lakehouse structure in a target tenant, that's two separate artefacts: a .sql script for the views and a PySpark notebook for the empty table structure. They also have to run in that order. Notebook writes need ?format=ipynb Creating or updating a notebook definition without it returns "PyToIpynbFailure: Convert py to ipynb failed" with no further explanation. Add the query parameter on both createItem and updateDefinition. Only notebooks need it. Delta rejects column names with spaces or punctuation AnalysisException [DELTA_INVALID_CHARACTERS_IN_COLUMN_NAMES] for anything containing space , ; { } ( ) newline tab = The obvious fix is to sanitise the names, but don't — your views and semantic models reference the original names. Enable column mapping on those tables instead: delta.columnMapping.mode = name, minReaderVersion = 2, minWriterVersion = 5. Original names preserved, Delta happy. Premium_ASWL_Error on refresh does not mean you need a gateway Most threads will tell you to set up an on-premises data gateway. In my case the model was set to "Default: Single Sign-On" and simply needed an explicit cloud connection created and bound to it. No gateway involved. Worth checking before you go down the gateway route. Bonus: pipelines fail with a bare "UnknownError" when they reference items that aren't in the target yet. The error never says which ones. I ended up scanning pipeline definitions for GUIDs and matching them against the source inventory to list the missing items by name before attempting the deploy, and deploying master pipelines after their children. Happy to go deeper on any of these if it's useful.Optimising Fabric Capacity Usage
Hi all, Due to the recent changes in Fabric capacity metering, I've been spending quite a bit of time trying to optimise our CU consumption. The challenge I've run into is that it's difficult to distinguish between SQL activity generated by user queries versus SQL activity generated during semantic model refreshes, which makes it harder to accurately evaluate different architectural approaches. Approach 1: Materialising Views as Delta Tables Historically, our semantic models have queried SQL views hosted in Fabric. To reduce SQL Endpoint consumption, I began converting the T-SQL view logic into notebooks that materialise the results into Delta tables. The semantic models then connect directly to these Delta tables via the ADLS Gen2 connector in Power BI, effectively bypassing the SQL endpoint during refresh. However, the results have been inconsistent. In some workspaces this appears to reduce overall CU consumption, while in others it actually increases costs because the notebook execution itself incurs significant compute usage. Current new architecture(old queried straight from omdb sql views): My questions are: From a CU consumption perspective, is it generally more efficient to: Have semantic models query Fabric SQL views directly, or Materialise those views into Delta tables and have semantic models consume the Delta tables through ADLS Gen2? Has anyone compared the total cost of repeatedly querying views during semantic model refreshes versus the cost of creating and maintaining materialised Delta tables? While I understand that a star schema is considered best practice for reporting and semantic modelling, if the same business logic can be represented through a set of SQL views, is there still a compelling reason to physically materialise the data into tables from a cost-efficiency perspective? I've attempted to benchmark both approaches, but isolating the relevant consumption metrics has proven more difficult than expected. Approach 2: Shifting Transformation Logic to the Semantic Model Another approach I'm exploring is moving the transformation and view logic out of our Fabric workspace and into semantic models hosted in the client's Power BI Pro tenant. My assumption is that this would reduce SQL Endpoint consumption within our Fabric capacity because less querying and transformation would occur on our side. However, I would expect semantic model refresh times to increase as more transformation work is pushed downstream. My questions here are: Would moving the transformation logic into the semantic model generally reduce or increase overall SQL-related CU consumption? Have others seen meaningful capacity savings using this approach? Is there a way to connect one semantic model to another semantic model without using DirectQuery? My concern is that DirectQuery would negatively impact performance and potentially introduce additional query-related costs. I'd be very interested to hear how others are approaching this, particularly now that SQL Endpoint consumption has become much more visible within Fabric capacity metrics. I have also reviewed the Query Insights for each workspace in an attempt to correlate SQL activity with overall CU consumption. However, this does not always translate into lower capacity usage. One of the challenges is that many users are querying the SQL Endpoint directly through Excel and other tools, without going through a semantic model at all. Under the new metering model, these ad hoc user queries appear to be a significant contributor to capacity consumption, making it difficult to isolate the impact of semantic model refreshes versus interactive user activity. As a result, determining whether a particular optimisation has genuinely reduced costs becomes far more challenging, as overall CU usage may be heavily influenced by user behaviour outside of the reporting layer. Thanks in advance!Cross-tenant Fabric migration (no shared account): 6 things the docs don't tell you
Sharing this rather than asking — posting my notes in case they save someone else the time. I recently had to move semantic models, reports, notebooks, pipelines and a Lakehouse between two Fabric tenants where no single account had access to both — no guest access, no cross-tenant permissions, two entirely separate sign-ins. Deployment pipelines are same-tenant only. fabric-cicd assumes one tenant and a Git repo. So this became an export-to-file / import-from-file exercise against the REST API. Six things cost me real time. Posting them in case they save someone else the same days. 1) A semantic model's connection lives in FOUR places in model.bim I started by rewriting the M expressions and the log said "0 expressions repointed" while the model kept pointing at the old source. The connection can sit in the M code (Sql.Database(...)), in a dataSources[] entry's connectionDetails.address, in Direct Lake entity partitions, and in M parameters with a meta annotation. Rewriting only one of them silently does nothing. I ended up operating on the raw model.bim JSON text so all four are covered. 2) PBIR report binding changed shape between schema versions definition.pbir validates differently depending on its major version. Schema v2.0+ wants connectionString and nothing else — adding pbiServiceModelId gets you "the schema does not allow additional properties". Schema 1.x wants the full legacy set: pbiServiceModelId, pbiModelVirtualServerName ("sobe_wowvirtualserver"), pbiModelDatabaseName, name: "EntityDataSource", connectionType: "pbiServiceXmlaStyleLive". Omit them and you get "Cannot resolve neither report.json nor the PBIR report content in enhanced format". Preserve the original $schema and version from the source file and branch on the major version. Don't assume one shape. 3) The Lakehouse SQL analytics endpoint is read-only over Delta CREATE VIEW, CREATE FUNCTION and CREATE PROCEDURE all work. CREATE TABLE does not — tables only come from Spark. If you're recreating a Lakehouse structure in a target tenant, that's two separate artefacts: a .sql script for the views and a PySpark notebook for the empty table structure. They also have to run in that order. 4) Notebook writes need ?format=ipynb Creating or updating a notebook definition without it returns "PyToIpynbFailure: Convert py to ipynb failed" with no further explanation. Add the query parameter on both createItem and updateDefinition. Only notebooks need it. 5) Delta rejects column names with spaces or punctuation AnalysisException [DELTA_INVALID_CHARACTERS_IN_COLUMN_NAMES] for anything containing space , ; { } ( ) newline tab = The obvious fix is to sanitise the names, but don't — your views and semantic models reference the original names. Enable column mapping on those tables instead: delta.columnMapping.mode = name, minReaderVersion = 2, minWriterVersion = 5. Original names preserved, Delta happy. 6) Premium_ASWL_Error on refresh does not mean you need a gateway Most threads will tell you to set up an on-premises data gateway. In my case the model was set to "Default: Single Sign-On" and simply needed an explicit cloud connection created and bound to it. No gateway involved. Worth checking before you go down the gateway route. Bonus: pipelines fail with a bare "UnknownError" when they reference items that aren't in the target yet. The error never says which ones. I ended up scanning pipeline definitions for GUIDs and matching them against the source inventory to list the missing items by name before attempting the deploy, and deploying master pipelines after their children. Happy to go deeper on any of these if it's useful.
