All Data Engineering posts

Re: Background CU used in Fabric Capacity Metrics

Barry-Bitmetric — Sat, 19 Sep 2026 07:47:48 GMT

What you’re seeing is probably not background activity that is still running.

Fabric smooths the CU consumption of background operations over 24 hours. So a pipeline or notebook that finished 12 hours ago can still show up as contributing to your current background usage. There is nothing to “clear” in that case. It is accounting for compute that was already consumed.

That also explains why scaling down from F64/F62-ish capacity to F32 can hurt much more than the Spark job view suggested. Looking only at the peak CUs used by Spark doesn’t give you the full capacity picture. You also have the pipelines, copy activities, notebooks and other operations contributing to the 24-hour smoothed usage.

I’d use the Capacity Metrics app to identify which operations are contributing most over that 24-hour window, and optimize those rather than trying to remove old background jobs.

The 15-minute orchestrator would be the first thing I’d look at. Pulling from 42 sources every 15 minutes is a lot of activity. I’d question whether all 42 sources actually need that frequency, whether you can process incrementally, and whether some parts of the pipeline can run less often.

In other words, I wouldn’t size the capacity based on the maximum Spark CUs you see during a job. I’d size it based on the total workload over time, including the smoothed background consumption.

The Capacity Metrics app is much more useful for that than the Spark job view alone.

Re: Fabric Pipeline Error

NandanHegde — Sat, 19 Sep 2026 05:17:46 GMT

It is a known bug/issue:

https://support.fabric.microsoft.com/known-issues/?product=Data%2520Factory&active=true&fixed=true&sort=published

Re: Fabric Pipeline Error

NandanHegde — Sat, 19 Sep 2026 05:16:15 GMT

Make a trivial/mnior change to the pipeline like a description edit works and save it. That would force the token to be re set up and the next run succeeds

Re: Current or upcoming DP-700 exam vouchers

tayloramy — Sat, 19 Sep 2026 03:23:06 GMT

Please send me a message :)

Re: Inexplicable DataflowsStagingWarehouse Load on Capacity

csm — Fri, 18 Sep 2026 21:14:53 GMT

We've recently started seeing entries for "High Scale Dataflow Compute - Warehouse Query" in the Capacity Metric App under Background records. These were not present a couple months ago and we had a good baseline on the workspace's total compute. Our dataflows have not changed but suddenly we're seeing a 6% increase in base capacity used under this workspace, attributed to these high-scale entries.

Within this workspace, no dataflows have staging enabled (all queries are italicized), but we are using SQL data destinations so that may be the underlying usage. I initially thought the billing was just broken out differently in the metric app, but the % base capacity used by the entire workspace has increased by the amount from this line. It doesn't show under the tooltip for each dataflow either, as NotebookEnjoyer experienced. The app is no help to narrow this down further.

Fabric Pipeline Error

odtJitendra — Fri, 18 Sep 2026 19:00:07 GMT

Today, we noticed that our production pipeline started failing with the following error:

Error:

BadRequest Error fetching pipeline default identity userToken{

"code": "LSROBOTokenFailure","message": "AADSTS50173: The provided grant has expired due to it being revoked, a fresh auth token is needed. The user might have changed or reset their password. The grant was issued on '2026-08-24T05:50:05.7907970Z' and the TokensValidFrom date (before which tokens are not valid) for this user is '2026-09-18T16:05:07.0000000Z'. Trace ID: 250ed633-08f0-4dcb-bad0-11596983e901 Correlation ID: 430f0833-fbc4-44f8-b19a-883ada2cad74 Timestamp: 2026-09-18 18:29:30Z",

"target": "PipelineDefaultIdentity-3e591cb6-2704-49f2-9efd-7196287c9feb","details": null,"error": null }FetchUserTokenForPipelineAsync

Has anyone encountered this issue before or have any idea what could be causing it?

Any suggestions on how to resolve this would be appreciated.

Designing AI Agent Workflows on Modern Data Platforms

codeautomation — Fri, 18 Sep 2026 15:21:22 GMT

Hi everyone,

I have been exploring how AI agents can work with modern data engineering platforms to automate business processes.

A common architecture I see is that data pipelines collect and prepare information, AI agents analyze the context and determine the next action, workflow services execute tasks through APIs or connected systems, and the results are stored for reporting or further processing.

I am interested in how teams are approaching this with Microsoft Fabric.

Are you using AI agents directly with Fabric workflows, or keeping the AI layer separate?

What patterns work well for connecting AI agents with Lakehouse, Data Pipelines, or notebooks?

How do you manage security and permissions when AI applications need access to enterprise data?

Are there recommended approaches for combining Fabric workloads with external AI services?

I would be interested to hear what architecture patterns others are using and what challenges you have encountered.

Re: Current or upcoming DP-700 exam vouchers

UshaSriG — Fri, 18 Sep 2026 13:55:37 GMT

Hi tayloramy ,

I'm still looking for the vouchers, would be great if you give one to me and I'll complete the exam in October.

Thank you.

Re: Need to Migrate SP written in Synapse to Fabric

maxravi — Fri, 18 Sep 2026 13:09:29 GMT

ShivekMaharaj and EduardoCastro for your response

Re: Need to Migrate SP written in Synapse to Fabric

v-saisrao-msft — Fri, 18 Sep 2026 12:52:28 GMT

Hi maxravi,

Have you had a chance to review the solution shared by ShivekMaharaj EduardoCastro? If the issue persists, feel free to reply so we can help further.

Thank you.

Re: Optimising Fabric Capacity Usage

v-achippa — Fri, 18 Sep 2026 12:13:02 GMT

Hi FabricEnjoyer,

Thank you for reaching out to Microsoft Fabric Community.

The CU consumption depends on the workload, so there is no general rule that materializing every sql view into delta tables will reduce capacity usage.

If the existing sql views are already optimized, I would recommend keeping the current approach rather than moving all the logic to notebooks just to reduce sql consumption. Materializing the views is useful when the same transformation is expensive and reused frequently, but the notebook execution and maintenance also consume fabric capacity.

Use the Capacity Metrics app to identify whether semantic model refreshes or direct queries from excel and other users are causing most of the consumption before changing the architecture.

Thanks and regards,
Anjan Kumar Chippa

Re: Pure Python Notebook: authenticate to Azure DevOps Artifacts using Fabric Workspace Identity

v-csrikanth — Fri, 18 Sep 2026 11:01:16 GMT

Hi llueg
We would like to inquire whether have you got the chance to check the solutions provided by other users in community to resolve the issue. We hope the information provided helps to clear the query. Should you have any further queries, kindly feel free to contact the Microsoft Fabric community.

Re: Storing and parsing HL7 data

nathancwatkins — Fri, 18 Sep 2026 10:51:10 GMT

Bring in all fields by parsing with |, grab and fill down each Message ID and segment code, make the parsed values a list type, then into JSON, so you have a column containing the values as JSON text for that message ID and segment. The values compress efficiently, and there’s your base ingestion table. Now your ETL query can easily extract values and zip them to a list of field names.

—Nate

Re: From On-Premises SQL Server to Microsoft Fabric with a Metadata-Driven Medallion Pipeline

ezalor — Fri, 18 Sep 2026 06:22:46 GMT

I think my current development direction aligns with this article. The areas I'm still lacking are data quality and AI self-healing. Other than that, I've already covered the AI Agent development part using skills to guide metadata setup. You can take a look at the video to evaluate: Build a Medallion Data Pipeline with DataCoolie + AI | AWS, Fabric & Databricks

Optimising Fabric Capacity Usage

FabricEnjoyer — Fri, 18 Sep 2026 06:08:18 GMT

Hi all,

Due to the recent changes in Fabric capacity metering, I've been spending quite a bit of time trying to optimise our CU consumption. The challenge I've run into is that it's difficult to distinguish between SQL activity generated by user queries versus SQL activity generated during semantic model refreshes, which makes it harder to accurately evaluate different architectural approaches.

Approach 1: Materialising Views as Delta Tables

Historically, our semantic models have queried SQL views hosted in Fabric. To reduce SQL Endpoint consumption, I began converting the T-SQL view logic into notebooks that materialise the results into Delta tables. The semantic models then connect directly to these Delta tables via the ADLS Gen2 connector in Power BI, effectively bypassing the SQL endpoint during refresh.

However, the results have been inconsistent. In some workspaces this appears to reduce overall CU consumption, while in others it actually increases costs because the notebook execution itself incurs significant compute usage.

Current new architecture(old queried straight from omdb sql views):

My questions are:

From a CU consumption perspective, is it generally more efficient to:
- Have semantic models query Fabric SQL views directly, or
- Materialise those views into Delta tables and have semantic models consume the Delta tables through ADLS Gen2?
Has anyone compared the total cost of repeatedly querying views during semantic model refreshes versus the cost of creating and maintaining materialised Delta tables?
While I understand that a star schema is considered best practice for reporting and semantic modelling, if the same business logic can be represented through a set of SQL views, is there still a compelling reason to physically materialise the data into tables from a cost-efficiency perspective?

I've attempted to benchmark both approaches, but isolating the relevant consumption metrics has proven more difficult than expected.

Approach 2: Shifting Transformation Logic to the Semantic Model

Another approach I'm exploring is moving the transformation and view logic out of our Fabric workspace and into semantic models hosted in the client's Power BI Pro tenant.

My assumption is that this would reduce SQL Endpoint consumption within our Fabric capacity because less querying and transformation would occur on our side. However, I would expect semantic model refresh times to increase as more transformation work is pushed downstream.

My questions here are:

Would moving the transformation logic into the semantic model generally reduce or increase overall SQL-related CU consumption?
Have others seen meaningful capacity savings using this approach?
Is there a way to connect one semantic model to another semantic model without using DirectQuery? My concern is that DirectQuery would negatively impact performance and potentially introduce additional query-related costs.

I'd be very interested to hear how others are approaching this, particularly now that SQL Endpoint consumption has become much more visible within Fabric capacity metrics.

I have also reviewed the Query Insights for each workspace in an attempt to correlate SQL activity with overall CU consumption. However, this does not always translate into lower capacity usage. One of the challenges is that many users are querying the SQL Endpoint directly through Excel and other tools, without going through a semantic model at all. Under the new metering model, these ad hoc user queries appear to be a significant contributor to capacity consumption, making it difficult to isolate the impact of semantic model refreshes versus interactive user activity. As a result, determining whether a particular optimisation has genuinely reduced costs becomes far more challenging, as overall CU usage may be heavily influenced by user behaviour outside of the reporting layer.

Thanks in advance!

Re: Storing and parsing HL7 data

Kagiyama_yutaka — Fri, 18 Sep 2026 00:57:19 GMT

Could you add a bit more detail about your setup? How you store HL7 for analytics can change a lot depending on whether you're working with HL7 v2 or FHIR, and whether your workflow is real‑time or batch.

Storing and parsing HL7 data

ipkus — Thu, 17 Sep 2026 21:38:05 GMT

Any healthcare folks can share what best practices they have found around storing HL7 data for analytics ?

Re: Building Self-Healing Data Pipelines

ipkus — Thu, 17 Sep 2026 21:35:27 GMT

Thanks

Re: Fabric Copy Activity fails in Query mode but succeeds in Table mode through an on-premises gateway

ipkus — Thu, 17 Sep 2026 21:34:48 GMT

Looks like the user / SP has permissions to the LH but does not have permissions to SQL endpoint. Make sure use has READALL ( which gives them permissions to SQL Endpoint ).

I suspect Query mode is forcing execution through the Fabric SQL endpoint and the gateway server either:

cannot resolve/reach that endpoint,
is not authorized to use the connection through the gateway,

Re: Copy Job - Amazon S3 connector fails with region signing error on generic endpoint

Sharan_Srini99 — Thu, 17 Sep 2026 21:26:56 GMT

Hi v-achippa ,

Thank you for your reply, I tried that approach and yes, the connection does work. But the problem there is it is mandatory to specify a bucket name. Is there any way to get around this?