Data Engineering topics

Fabric Pipeline Error

odtJitendra — Fri, 18 Sep 2026 19:00:07 GMT

Today, we noticed that our production pipeline started failing with the following error:

Error:

BadRequest Error fetching pipeline default identity userToken{

"code": "LSROBOTokenFailure","message": "AADSTS50173: The provided grant has expired due to it being revoked, a fresh auth token is needed. The user might have changed or reset their password. The grant was issued on '2026-08-24T05:50:05.7907970Z' and the TokensValidFrom date (before which tokens are not valid) for this user is '2026-09-18T16:05:07.0000000Z'. Trace ID: 250ed633-08f0-4dcb-bad0-11596983e901 Correlation ID: 430f0833-fbc4-44f8-b19a-883ada2cad74 Timestamp: 2026-09-18 18:29:30Z",

"target": "PipelineDefaultIdentity-3e591cb6-2704-49f2-9efd-7196287c9feb","details": null,"error": null }FetchUserTokenForPipelineAsync

Has anyone encountered this issue before or have any idea what could be causing it?

Any suggestions on how to resolve this would be appreciated.

Designing AI Agent Workflows on Modern Data Platforms

codeautomation — Fri, 18 Sep 2026 15:21:22 GMT

Hi everyone,

I have been exploring how AI agents can work with modern data engineering platforms to automate business processes.

A common architecture I see is that data pipelines collect and prepare information, AI agents analyze the context and determine the next action, workflow services execute tasks through APIs or connected systems, and the results are stored for reporting or further processing.

I am interested in how teams are approaching this with Microsoft Fabric.

Are you using AI agents directly with Fabric workflows, or keeping the AI layer separate?

What patterns work well for connecting AI agents with Lakehouse, Data Pipelines, or notebooks?

How do you manage security and permissions when AI applications need access to enterprise data?

Are there recommended approaches for combining Fabric workloads with external AI services?

I would be interested to hear what architecture patterns others are using and what challenges you have encountered.

Optimising Fabric Capacity Usage

FabricEnjoyer — Fri, 18 Sep 2026 06:08:18 GMT

Hi all,

Due to the recent changes in Fabric capacity metering, I've been spending quite a bit of time trying to optimise our CU consumption. The challenge I've run into is that it's difficult to distinguish between SQL activity generated by user queries versus SQL activity generated during semantic model refreshes, which makes it harder to accurately evaluate different architectural approaches.

Approach 1: Materialising Views as Delta Tables

Historically, our semantic models have queried SQL views hosted in Fabric. To reduce SQL Endpoint consumption, I began converting the T-SQL view logic into notebooks that materialise the results into Delta tables. The semantic models then connect directly to these Delta tables via the ADLS Gen2 connector in Power BI, effectively bypassing the SQL endpoint during refresh.

However, the results have been inconsistent. In some workspaces this appears to reduce overall CU consumption, while in others it actually increases costs because the notebook execution itself incurs significant compute usage.

Current new architecture(old queried straight from omdb sql views):

My questions are:

From a CU consumption perspective, is it generally more efficient to:
- Have semantic models query Fabric SQL views directly, or
- Materialise those views into Delta tables and have semantic models consume the Delta tables through ADLS Gen2?
Has anyone compared the total cost of repeatedly querying views during semantic model refreshes versus the cost of creating and maintaining materialised Delta tables?
While I understand that a star schema is considered best practice for reporting and semantic modelling, if the same business logic can be represented through a set of SQL views, is there still a compelling reason to physically materialise the data into tables from a cost-efficiency perspective?

I've attempted to benchmark both approaches, but isolating the relevant consumption metrics has proven more difficult than expected.

Approach 2: Shifting Transformation Logic to the Semantic Model

Another approach I'm exploring is moving the transformation and view logic out of our Fabric workspace and into semantic models hosted in the client's Power BI Pro tenant.

My assumption is that this would reduce SQL Endpoint consumption within our Fabric capacity because less querying and transformation would occur on our side. However, I would expect semantic model refresh times to increase as more transformation work is pushed downstream.

My questions here are:

Would moving the transformation logic into the semantic model generally reduce or increase overall SQL-related CU consumption?
Have others seen meaningful capacity savings using this approach?
Is there a way to connect one semantic model to another semantic model without using DirectQuery? My concern is that DirectQuery would negatively impact performance and potentially introduce additional query-related costs.

I'd be very interested to hear how others are approaching this, particularly now that SQL Endpoint consumption has become much more visible within Fabric capacity metrics.

I have also reviewed the Query Insights for each workspace in an attempt to correlate SQL activity with overall CU consumption. However, this does not always translate into lower capacity usage. One of the challenges is that many users are querying the SQL Endpoint directly through Excel and other tools, without going through a semantic model at all. Under the new metering model, these ad hoc user queries appear to be a significant contributor to capacity consumption, making it difficult to isolate the impact of semantic model refreshes versus interactive user activity. As a result, determining whether a particular optimisation has genuinely reduced costs becomes far more challenging, as overall CU usage may be heavily influenced by user behaviour outside of the reporting layer.

Thanks in advance!

Storing and parsing HL7 data

ipkus — Thu, 17 Sep 2026 21:38:05 GMT

Any healthcare folks can share what best practices they have found around storing HL7 data for analytics ?

Diagnosing a slow Fabric Data Warehouse using the new skill

arthurfr23 — Thu, 17 Sep 2026 14:18:02 GMT

When a Fabric Data Warehouse slows down, the investigation usually means jumping between the Capacity Metrics app, Query Insights, and SQL pool diagnostics while manually correlating time ranges across three different tools.

Microsoft just shipped the SQL DW operations skill as Generally Available, you can find the official announcement here:

https://community.fabric.microsoft.com/t5/Fabric-Updates-Blog/Diagnose-Fabric-Data-Warehouse-workloads-with-the-SQL-DW/ba-p/5366102

I dug into what this actually changes in practice: how the skill consolidates capacity spikes, query regressions, and request failures into a single conversational interface, and where it still has blind spots worth knowing before you rely on it in a production incident. There is at least one behavior around time range correlation that surprised me during testing.

To do that, I wrote a full article, and would love to hear if others are already using this in their warehouse troubleshooting flow.

https://medium.com/@arthurfr23/diagnosing-fabric-data-warehouse-with-the-sql-dw-operations-skill-investigating-workloads-without-e4060fe8661e?sharedUserId=arthurfr23

Copy Job - Amazon S3 connector fails with region signing error on generic endpoint

Sharan_Srini99 — Thu, 17 Sep 2026 03:24:05 GMT

I'm trying to connect Fabric's Amazon S3 connector (Copy activity, pipelines) to a bucket in ap-southeast-2, using Access Key authentication.

When I create a connection with the generic regional endpoint:

https://s3.ap-southeast-2.amazonaws.com

connecting fails with:

Expression.Error: Invalid Url. The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'ap-southeast-2'

I've added s3:ListAllMyBuckets, s3:ListBucket, and s3:GetBucketLocation to the IAM user, and the error persists identically.

Building Self-Healing Data Pipelines

ipkus — Wed, 16 Sep 2026 21:34:37 GMT

This article shares practical examples on building self healing piplelines and data quality. Do take a read and share your inputs.

https://medium.com/@rishabh.gulati_94109/ms-fabric-building-self-healing-data-ecosystems-67a7e83c5380

Can a service principal with federated credentials be used for SQL cloud connection auth?

CharlieFab — Wed, 16 Sep 2026 00:00:14 GMT

Replacing OAuth 2.0 personal-account auth on our Fabric SQL cloud connections. Workspace identity doesn't fit, because semantic models in other workspaces share the same connection and each workspace has its own identity — those refreshes fail with invalid credentials unless every workspace identity is granted on the database.

A service principal solves that, but I'd like to avoid storing a secret. I added a federated identity credential to the app registration. In the connection settings, Service Principal auth only offers a key or a certificate, with nowhere to reference the federated credential.

Is that expected? I assume federation needs the caller to present a token it already holds, and the Fabric refresh engine has none — so it simply can't apply. Confirmation would be useful, as would any roadmap or Ideas link for secret less cloud connection auth.

If certificate is the answer, I'd also welcome hearing how others handle cert rotation across many connections given the encrypted payload on the update credentials API.\

Need to Migrate SP written in Synapse to Fabric

maxravi — Tue, 15 Sep 2026 14:50:23 GMT

Hello,

I need to migrate my stored prodecure SP from synapse to Fabric, can you please help me with best approach. I am seeing a lot of syntax changes. Suggest me some best, quickest and efficient ways to migrate.

Thanks,

Is it possible connect via a guest account to a Delta table in a Lakehouse from another tenant.

FabricEnjoyer — Tue, 15 Sep 2026 11:21:00 GMT

We have the following architecture in Microsoft Fabric:

Open Mirroring Database ↓ Notebook ↓ Materialized Delta Tables (Lakehouse) ↓ Shortcuts into Connection Lakehouse ↓ Power BI Desktop (Azure Data Lake Storage Gen2 connector) ↓ DeltaLake.Table()

Source = AzureStorage.DataLake( "https://onelake.dfs.fabric.microsoft.com/<workspace-id>/<lakehouse-id>/Tables", [HierarchicalNavigation = true] ), dbo = Source{[Name="dbo"]}[Content], TableName = dbo{[Name="<table_name>"]}[Content], Data = DeltaLake.Table(TableName)

What works

Internal users in the hosting tenant can successfully create the connection.
At least one external Entra B2B Guest user can successfully create the connection.
If we create and publish the semantic model ourselves, external customers can copy/use the model successfully.
Existing reports refresh successfully once deployed.

What fails

Several external customer users that have been invited into our tenant as Entra B2B Guests receive a 404 error when attempting to create a new Azure Data Lake Storage Gen2 connection directly from Power BI Desktop.

The users have:

Accepted the guest invitation.
Been granted access to the Fabric workspace(Contributor)
Been granted access to the Lakehouse.
Been added as Guest users in Entra ID.

Observations

The issue is affecting multiple customers, not just a single user.
The same Lakehouse, shortcut tables, and Power Query logic work for other users.
A Guest account from one customer tenant works, while Guest accounts from several other customer tenants receive a 404.
This suggests the issue may be related to cross-tenant authentication, OneLake access resolution, or the ADLS Gen2 connector rather than permissions on the Fabric artifacts themselves.

Questions

Are there known limitations or issues with Azure Data Lake Storage Gen2 connections to OneLake for some Entra B2B Guest users?
Are there specific cross-tenant access settings, trust settings, or Conditional Access configurations that can cause a 404 during OneLake path resolution?
Is there a recommended approach for external customers connecting directly to OneLake Delta tables via the ADLS Gen2 connector?

Any guidance would be appreciated.

Claude Code Integrate with Power Bi

Yash8160 — Tue, 15 Sep 2026 09:54:54 GMT

Hi everyone,

I’m working on a POC for Power BI + Claude Integration, specifically the Report Authoring skill.We need to use Claude Code to create the required report pages and visuals. I’m currently trying to estimate the Claude Code credit/token consumption and approximate cost for completing this POC.

Has anyone worked on a similar POC using Claude Code? If so, could you please share:

Approximate credits/tokens consumed

Estimated cost

Any recommendations for estimating the usage before starting

Any guidance or experience would be really helpful. Thanks!

Approach for automating Dataflow Gen2 connection binding across Dev/Prod?

lavginqo2 — Mon, 14 Sep 2026 19:53:22 GMT

We're deploying Dataflow Gen2 from Dev to Prod via fabric-cicd with all destination Lakehouse IDs and source parameters fully parameterized. Definitions deploy cleanly.

The blocker: every query's data source connection has to be manually reconfigured in the target workspace — "Configure connection" doesn't auto-bind to an existing connection object even when one already exists for that connector, and this has to be repeated per query/table, not once per Dataflow. Docs confirm this is expected ("connections are statically bound... can't be altered using workspace variables or parameterization"), but that leaves genuine automation impossible for Dataflows with many tables (~47 in our case).

And it is even not possible to update connection. It seems that each table has to be created fresh in Prod.

Has anyone found a real workaround — Service Principal-based automation, REST API call to bind connections post-deploy etc.

What's actually working for people running CI/CD from Dev to Prod at scale here?

Thanks

From On-Premises SQL Server to Microsoft Fabric with a Metadata-Driven Medallion Pipeline

ezalor — Mon, 14 Sep 2026 10:48:56 GMT

In this demo, I use DataCoolie to move WideWorldImporters data from an on-premises SQL Server into Microsoft Fabric. The pipeline is driven by one metadata.json file and follows the Bronze, Silver, and Gold pattern.

The focus here is the Fabric execution flow, not the earlier development process.

The complete sample is available in the datacoolie/dc-demo-fabric repository.

Architecture

The SQL Server remains on-premises, so ingestion runs from an external Polars runner. Downstream processing runs inside Fabric:

Stage	Runtime	Output
source2bronze	On-premises Polars	Parquet in Files/bronze
bronze2silver	Fabric Python 3.12 notebook with Polars and delta-rs	Delta tables in silver
silver2gold	Fabric Spark notebook	Delta tables in gold

The workspace uses two Lakehouses:

lh_wwi_medallion_demo
├── Files/bronze
├── Tables/silver
└── Tables/gold

lh_etl
└── Files/control/test-fabric
├── metadata/current/metadata.json
├── logs
└── watermarks

This keeps operational state separate from business data while allowing every runtime to use the same metadata and watermarks.

One metadata file, three stages

The final metadata contains four connections, 68 dataflows, and shared schema hints. Each dataflow declares its stage, source, destination, load type, watermark, merge keys, or SQL query. The runner remains generic: it reads this file and receives only the stage name it should execute.

The Fabric environment maps logical connections to:

bronze_parquet → Files/bronze
silver_delta → Tables/silver
gold_delta → Tables/gold

Note: An AI agent will assist in generating the metadata.json file.

Uploading the artifacts with Fabric CLI

After creating the two Lakehouses, install the Fabric CLI, authenticate, upload the metadata, and import the two notebooks:

pip install --upgrade ms-fabric-cli fab auth login $workspace = 'wwi_medallion_demo_test_fabric.Workspace' $etl = "$workspace/lh_etl.Lakehouse" fab mkdir "$etl/Files/control/test-fabric/metadata/current" fab cp '.\metadata.json' ` "$etl/Files/control/test-fabric/metadata/current/metadata.json" -f fab import "$workspace/run_fabric_polars.Notebook" ` -i '.\runners\run_fabric_polars.ipynb' --format ipynb -f fab import "$workspace/run_fabric_spark.Notebook" ` -i '.\runners\run_fabric_spark.ipynb' --format ipynb -f

Stage 1: SQL Server to Bronze

The on-premises runner reads 31 business tables and appends raw Parquet batches to OneLake. Every table has a watermark: ValidFrom, LastEditedWhen, or VehicleTemperatureID. Timestamp-based flows reread a one-day overlap to reduce the chance of missing late records.

Bronze files are organized by ingestion date:

Files/bronze/<schema>__<table>/YYYY/MM/DD/

This stage still runs on-premises because it requires SQL Server connectivity:

$control = 'abfss://wwi_medallion_demo_test_fabric@onelake.dfs.fabric.microsoft.com/lh_etl.Lakehouse/Files/control/test-fabric' python '.\runners\run_fabric_polars_azure_sdk.py' ` --metadata-path "$control/metadata/current/metadata.json" ` --watermark-base-path "$control/watermarks" ` --base-log-path "$control/logs" ` --stage source2bronze --max-workers 4

Stage 2: Bronze to Silver

A Fabric Python notebook runs this stage with Polars and delta-rs, without Spark. It reads new Bronze files through __file_modification_time, removes duplicate source keys, and performs a Delta merge_upsert for all 31 tables.

The result is a current-state Silver replica while Bronze keeps the raw append history.

After Bronze is ready, start the notebook with Fabric CLI. The Lakehouse configuration binds the notebook to lh_wwi_medallion_demo:

$lakehouseConfig = '{"defaultLakehouse":{"name":"lh_wwi_medallion_demo","id":"<lakehouse-id>","workspaceId":"<workspace-id>"}}' fab job run "$workspace/run_fabric_polars.Notebook" ` -P "STAGE:string=bronze2silver,_inlineInstallationEnabled:bool=true" ` -C $lakehouseConfig --timeout 3600

Stage 3: Silver to Gold

The final stage runs in a Fabric Spark notebook. SQL queries stored in metadata read two-level names such as silver.Sales__InvoiceLines and create six Gold products:

DimDate
DimGeography
DimProduct
DimPerson
DimDeliveryMethod
FactInvoiceLineProfitability

The source instance had no customer data, so customer-dependent Gold models were intentionally excluded.

Run Gold only after the Silver notebook succeeds:

fab job run "$workspace/run_fabric_spark.Notebook" ` -P "STAGE:string=silver2gold,_inlineInstallationEnabled:bool=true" ` -C $lakehouseConfig --timeout 3600

Monitoring the complete flow

Fabric Monitoring shows notebook status, duration, and execution details. DataCoolie writes dataflow-level logs and watermarks to lh_etl, allowing DataCoolie Studio to display metadata, assets, lineage, jobs, and individual dataflow runs across all three stages.

Result

The completed pipeline has 31 incremental Bronze flows, 31 key-merged Silver tables, and six Gold products. The runner stays generic; metadata defines the tables, watermarks, merge keys, paths, and SQL transformations.

This separation makes the pipeline easier to extend and keeps the hybrid on-premises/Fabric workflow observable as one system.

Pure Python Notebook: authenticate to Azure DevOps Artifacts using Fabric Workspace Identity

llueg — Sat, 12 Sep 2026 12:47:07 GMT

Can a Pure Python Fabric notebook running under a Workspace Identity obtain an Azure DevOps access token so that `pip install --index-url ...` can authenticate without a PAT?

It would be very nice if we could use the Workspace Identity to authenticate directly against a DevOps Feed in order to pull dependencies, as that would allow DevOps to directly target Feed-permissions for that Workspace (in particular, `Feed Reader`). For Spark Notebooks, we can use a Connection to do that, but for Pure Python Notebooks I can't find a documented way.

At the end of the day, one needs a token to pass to `pip` in order to bootstrap the notebook's runtime environment from the Feed, but as far as I can see there is no documented way to generate that token from the Workspace Identity. Currently, I'm using a keyvault'ed PAT. However, while this is fine-ish on the Fabric-side, it requires a dedicated Identity on the DevOps side, which I'd like to avoid.

Business Central Data to Fabric

Saum02 — Fri, 11 Sep 2026 11:18:28 GMT

Hii All,

Has anyone explored the BC connection to it's underlying SQL database? Is there limitation to this approach? I need to get data from BC to fabric, is SQL database of BC worth exploring?

Fabric Copy Activity fails in Query mode but succeeds in Table mode through an on-premises gateway

alisaleh — Thu, 10 Sep 2026 20:25:01 GMT

Hi Fabric Community,

I am using a Microsoft Fabric pipeline Copy Activity to copy data from a Fabric Lakehouse to an on-premises SQL Server database through a dedicated on-premises data gateway.

I tested the Copy Activity using both of the following source connection types:

Lakehouse connection
Lakehouse SQL analytics endpoint connection

With both connection types, the behavior is the same:

Lookup Activity: Successful
Script Activity: Successful
Copy Activity using Table mode: Successful
Copy Activity using Query mode: Fails

When the Copy Activity source is configured with Use query = Query, it fails with the following error:

ErrorCode=SqlFailedToConnect,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Cannot connect to SQL Database. Please contact SQL server team for further support. Server: 'xxxxxxxx.datawarehouse.fabric.microsoft.com', Database: 'LH_XYZ', User: ''. Check the connection configuration is correct, and make sure the SQL Database firewall allows the Data Factory runtime to access.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server),Source=.Net SqlClient Data Provider,SqlErrorNumber=53,Class=20,ErrorCode=-2146232060,State=0,Errors=[{Class=20,Number=53,State=0,Message=A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server),},],''Type=System.ComponentModel.Win32Exception,Message=The network path was not found,Source=,'

However, when I change the source configuration from Query to Table and select the source table directly, the Copy Activity completes successfully.

Has anyone encountered this behavior?

Does Query mode use a different runtime, connector path, or metadata-discovery process when the Copy Activity runs through an on-premises gateway?

Are there any known limitations or additional gateway requirements for using a custom query as the source?

Any guidance on how to diagnose or resolve this would be appreciated.

Bronze data contracts for REST/API sources — workaround for Get Metadata's connector gap

damo_mac85 — Thu, 10 Sep 2026 20:15:32 GMT

For SQL sources, our pattern is: Copy activity lands the table to Parquet, Get Metadata pulls the source structure, a Notebook compares that structure to a JSON contract, and if it matches, a Copy activity promotes Parquet to Delta — if not, we notify instead.

Wanted the same for REST/API sources, but Get Metadata doesn't support REST connectors — so there's no direct "get source structure" step for an API.

Workaround: since Parquet stores its own schema, we skip Get Metadata for API sources and just read it back off the landed file:

Copy: API → Parquet (raw landing)

Notebook: spark.read.parquet(path).schema vs. JSON contract

Match → Copy Parquet to Delta. Mismatch → notify,

Questions for the community:

Anyone found a more native way to get REST schema before landing data?

Is anyone doing a pre-flight sample call to validate before pulling the full payload?

Any word on Get Metadata adding REST support?

Curious how others doing bronze contracts in Fabric have handled this.

Materialized view vs delta tables in gold layer

ahmedshalabyy12 — Wed, 09 Sep 2026 08:06:10 GMT

Hello,

I asked ChatGPT this question, but I got a useless answer from the AI, and it changed its opinion twice during the session. So I decided to ask the experts: from your previous experience, which is better
Materialized views or delta tables in the gold layer? Both store the physical data, and both can do incremental refresh. But when and why, that's my question.

Thank you

Variable Library

Ira_27 — Tue, 08 Sep 2026 22:23:32 GMT

I am trying to use variable library in Lookup task in Fabric pipeline but choosing a GUID for a WH doesnt seem to work. As soon as i use dynamic content in connection it wipes off all the settings and changes to below screenshot.

Has anyone used variable library including connection reference and item reference in LookUp, Notebook tasks?

Background CU used in Fabric Capacity Metrics

dinhhoa — Tue, 08 Sep 2026 16:11:30 GMT

I'm trying to manage the CU usages to lower my billing.

My system has 2 main orchestrators to process data for my 2 reports. One will run every 15min and other is 8h. Both will use copy activity to retrieve data from 42 data sources (using ForEach) and then using notebook to process later on.

I was previously using an F62 capacity. Based on my observations in the Job tab under Spark Settings, my max workload was only taking around 88/192 CUs (when there was 2 15m orche and 1 8h orche running in the same time), so I decided to scale down to F32.

After the downgrade, my system became unresponsive and started hitting capacity limits. My Power BI reports couldn't load, and I was forced to scale the capacity back up to F62.

Then, I discovered the Fabric Capacity Metrics app and noticed that my Background workload was consuming nearly 80% of the available CUs. What's strange is that many pipelines, activity, notebooks, and other jobs that had already completed more than 12 hours earlier still appeared to be consuming a small amount of capacity in the background.

Does anyone know how to troubleshoot or resolve these background activities? is there a way to identify or clear them?