rss.livelink.threads-in-node

How are AI agents changing modern data engineering workflows?

codeautomation — Mon, 28 Sep 2026 20:18:49 GMT

I am exploring how AI agents can assist data engineering teams in managing complex data workflows and reducing repetitive operational tasks.

Some areas I am interested in:

AI agents for monitoring data pipelines and detecting failures
Automated data quality checks and anomaly detection
Intelligent assistance for ETL/ELT workflow optimization
Generating documentation for datasets and transformations
Using AI with Fabric pipelines, notebooks, and data workflows

With the growth of AI-powered automation, I would like to understand how data engineers are approaching these patterns in real-world environments.

What approaches, architectures, or best practices are teams using to combine Microsoft Fabric capabilities with AI agents?

Title: Microsoft Fabric – Impact of Enabling Case-Insensitive Collation Settings

Murtaza_Ghafoor — Mon, 28 Sep 2026 17:19:52 GMT

Hi Fabric Community,

We are evaluating the impact of enabling Case Insensitive (CI) settings in a Microsoft Fabric workspace.

The workspace setting provides the following options:

Case Sensitive: Latin1_General_100_BIN2_UTF8
Case Insensitive: Latin1_General_100_CI_AS_KS_WS_SC_UTF8

According to the documentation, these settings affect how SQL processes capitalization in object names and string data.

Before enabling/using the Case Insensitive setting in our environment, we would appreciate the community's guidance on the following:

Existing objects:
If Case Insensitive is enabled at the workspace level, does it affect existing Warehouses or SQL Analytics Endpoints, or does it apply only to newly created objects?
Existing data:
Does changing the collation setting modify or transform the existing data stored in tables, or does it only change how SQL compares/processes string values?
JOIN behavior:
Could Case Insensitive collation change the results of existing SQL joins? For example, could the following values start matching when they previously did not?
- ABC vs abc
- Assets vs assets
- Project01 vs PROJECT01
WHERE conditions:
Would a Case Insensitive collation change the results of existing filters such as:WHERE Account = 'ABC'when the stored value is abc?
GROUP BY / DISTINCT:
Could Case Insensitive collation change the results of GROUP BY, DISTINCT, or aggregation queries where values differ only by capitalization?
Primary/unique keys:
Are there any implications for duplicate detection or uniqueness when values such as ABC and abc exist in columns used as keys?
Data pipelines / transformations:
Could changing from Case Sensitive to Case Insensitive cause any unexpected behavior in Fabric Data Pipelines, Dataflows, Notebooks, or SQL transformations?
Lakehouse vs Warehouse:
Does this Case Sensitivity setting affect only Warehouses and SQL Analytics Endpoints, or can it also affect queries against Lakehouse tables?
Power BI semantic models:
Could changing the SQL collation impact Power BI semantic models, relationships, Direct Lake, Import, or DirectQuery reports?
Performance:
Is there any measurable performance difference between:
Latin1_General_100_BIN2_UTF8
and
Latin1_General_100_CI_AS_KS_WS_SC_UTF8
for large tables and joins?
Recommended approach:
For an enterprise Finance reporting solution where consistency of joins and mappings is critical, is it recommended to use Case Sensitive or Case Insensitive collation?
Best practice for mixed requirements:
If some business fields require case-insensitive comparisons while others require case-sensitive comparisons, what is the recommended approach in Fabric?
Collation at column/query level:
If the workspace/database is configured as Case Insensitive, is there a supported way to perform a case-sensitive comparison for specific columns or queries?
Changing the setting later:
If we initially create our Warehouse/SQL Analytics Endpoint using Case Sensitive collation, can we later change it to Case Insensitive, or would we need to create a new object?
Production migration:
If changing the collation requires recreating the Warehouse/SQL Analytics Endpoint, what is the recommended migration approach for existing production workloads?

We are particularly interested in understanding whether enabling Case Insensitive collation can change query results, especially for existing SQL joins and mappings where source-system values may have differences in capitalization or leading/trailing spaces.

Any guidance or real-world experience with Fabric enterprise implementations would be greatly appreciated.

Thank you.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Fail Message Not Working in Fail Activity

arehman36 — Fri, 25 Sep 2026 19:02:19 GMT

Good day

I have a lookup activity that returns an output like this:

{ "count": 6, "value": [ { "SchemaTablename": "Schema.TableName", "QueryText": "Query goes here", "OnPremTableName": "[dbo].[TableName" },

After the Lookup, I have a ForEach Activity that contains 3 activities

Lookup: OnPremColumnCount: returns the column count for a table on-premise. example output:

{ "firstRow": { "OnPremColumnCount": 30 } }

Lookup: CloudColumnCount: returns the column count for a table in fabric warehouse.

{ "firstRow": { "OnPremColumnCount": 30 } }

IF Condition: Checks to see if both tables have the same number of columns. If False, there is a fail activity.

Now my issue is that I am trying to write an error message that tells me the number of columns on-prem, on cloud, and the on-premise and cloud table name.

Here is what I have inside the Fail Message:

@concat( 'Columns do not match. On-premise table column count: ', string(activity('OnPremColumnCount').output.firstRow),'.', ' ', 'Cloud table column count: ', string(activity('CloudColumnCount').output.firstRow),'.', string(activity('LookupGetTableList').output.OnPremTableName), string(activity('LookupGetTableList').output.SchemaTablename))

The Error Code is: ActivityFailed.

I want it to show an error message like

Columns do not match. On-premise table count: 20. Cloud table count: 15. [Schema].[MY_TABLE]. [Schema].[My_Table]

I keep getting this error when I run the pipeline.

I am not sure why this is happening. I suspect that I am not accessing the output of my previous activities correctly. More so I think it's an issue with accessing the output from the First LookUp Activity (LookupGetTableList).

Here is a screenshot of my entire pipeline

Your help is greatly appreciated.

Thanks,

Fabric Link Setup

Nirav_Synapx — Fri, 25 Sep 2026 15:32:30 GMT

Hello Community,

I'm trying to Configure Fabric Link with FnO to sync My FnO data to Fabric.

While creating Fabric Link for FnO I'm getting 2 Options as per below SS.

One is F&O Entities (Caption 1)

Second is F&O tables (Caption 2).

I want to Understand How "FnO entities" and "FnO tables" different from each other, In which case I need to choose from second Option(F&O tables Caption 2) & in which case I need to choose from (F&O Entities Caption 1).

Thank you

Cross-tenant Fabric migration (no shared account): 6 things the docs don't tell you

vksraiade — Thu, 24 Sep 2026 14:50:11 GMT

Sharing this rather than asking — posting my notes in case they save someone else the time.

I recently had to move semantic models, reports, notebooks, pipelines and a Lakehouse between two Fabric tenants where no single account had access to both — no guest access, no cross-tenant permissions, two entirely separate sign-ins.

Deployment pipelines are same-tenant only. fabric-cicd assumes one tenant and a Git repo. So this became an export-to-file / import-from-file exercise against the REST API.

Six things cost me real time. Posting them in case they save someone else the same days.

A semantic model's connection lives in FOUR places in model.bim I started by rewriting the M expressions and the log said "0 expressions repointed" while the model kept pointing at the old source. The connection can sit in the M code (Sql.Database(...)), in a dataSources[] entry's connectionDetails.address, in Direct Lake entity partitions, and in M parameters with a meta annotation. Rewriting only one of them silently does nothing. I ended up operating on the raw model.bim JSON text so all four are covered.
PBIR report binding changed shape between schema versions definition.pbir validates differently depending on its major version. Schema v2.0+ wants connectionString and nothing else — adding pbiServiceModelId gets you "the schema does not allow additional properties". Schema 1.x wants the full legacy set: pbiServiceModelId, pbiModelVirtualServerName ("sobe_wowvirtualserver"), pbiModelDatabaseName, name: "EntityDataSource", connectionType: "pbiServiceXmlaStyleLive". Omit them and you get "Cannot resolve neither report.json nor the PBIR report content in enhanced format". Preserve the original $schema and version from the source file and branch on the major version. Don't assume one shape.
The Lakehouse SQL analytics endpoint is read-only over Delta CREATE VIEW, CREATE FUNCTION and CREATE PROCEDURE all work. CREATE TABLE does not — tables only come from Spark. If you're recreating a Lakehouse structure in a target tenant, that's two separate artefacts: a .sql script for the views and a PySpark notebook for the empty table structure. They also have to run in that order.
Notebook writes need ?format=ipynb Creating or updating a notebook definition without it returns "PyToIpynbFailure: Convert py to ipynb failed" with no further explanation. Add the query parameter on both createItem and updateDefinition. Only notebooks need it.
Delta rejects column names with spaces or punctuation AnalysisException [DELTA_INVALID_CHARACTERS_IN_COLUMN_NAMES] for anything containing space , ; { } ( ) newline tab = The obvious fix is to sanitise the names, but don't — your views and semantic models reference the original names. Enable column mapping on those tables instead: delta.columnMapping.mode = name, minReaderVersion = 2, minWriterVersion = 5. Original names preserved, Delta happy.
Premium_ASWL_Error on refresh does not mean you need a gateway Most threads will tell you to set up an on-premises data gateway. In my case the model was set to "Default: Single Sign-On" and simply needed an explicit cloud connection created and bound to it. No gateway involved. Worth checking before you go down the gateway route. Bonus: pipelines fail with a bare "UnknownError" when they reference items that aren't in the target yet. The error never says which ones. I ended up scanning pipeline definitions for GUIDs and matching them against the source inventory to list the missing items by name before attempting the deploy, and deploying master pipelines after their children.

Happy to go deeper on any of these if it's useful.

Capacity Metrics schema validation failed for all supported versions in FUAM

Shreya_Barhate — Thu, 24 Sep 2026 11:28:16 GMT

Hello,

I am facing an issue while collecting data from the Microsoft Fabric Capacity Metrics App using the FUAM notebook code.

The schema compatibility validation fails for every version checked by the notebook:

INFO: Test for v53 failed

INFO: Test for v47 failed

INFO: Test for v40 failed

INFO: Test for v37 failed

The notebook then returns the following exception:

ERROR: Capacity Metrics data structure is not compatible or connection to capacity metrics is not possible.

Could someone please help clarify:

Whether the Capacity Metrics App schema has recently changed.
Whether the current FUAM release supports the latest Capacity Metrics App.
Whether a newer validation query or updated FUAM notebook is available.

Thank you.

Cross-tenant Fabric migration (no shared account): 6 things the docs don't tell you

vksraiade — Thu, 24 Sep 2026 10:50:32 GMT

Sharing this rather than asking — posting my notes in case they save someone else the time. I recently had to move semantic models, reports, notebooks, pipelines and a Lakehouse between two Fabric tenants where no single account had access to both — no guest access, no cross-tenant permissions, two entirely separate sign-ins. Deployment pipelines are same-tenant only. fabric-cicd assumes one tenant and a Git repo. So this became an export-to-file / import-from-file exercise against the REST API. Six things cost me real time. Posting them in case they save someone else the same days. 1) A semantic model's connection lives in FOUR places in model.bim I started by rewriting the M expressions and the log said "0 expressions repointed" while the model kept pointing at the old source. The connection can sit in the M code (Sql.Database(...)), in a dataSources[] entry's connectionDetails.address, in Direct Lake entity partitions, and in M parameters with a meta annotation. Rewriting only one of them silently does nothing. I ended up operating on the raw model.bim JSON text so all four are covered. 2) PBIR report binding changed shape between schema versions definition.pbir validates differently depending on its major version. Schema v2.0+ wants connectionString and nothing else — adding pbiServiceModelId gets you "the schema does not allow additional properties". Schema 1.x wants the full legacy set: pbiServiceModelId, pbiModelVirtualServerName ("sobe_wowvirtualserver"), pbiModelDatabaseName, name: "EntityDataSource", connectionType: "pbiServiceXmlaStyleLive". Omit them and you get "Cannot resolve neither report.json nor the PBIR report content in enhanced format". Preserve the original $schema and version from the source file and branch on the major version. Don't assume one shape. 3) The Lakehouse SQL analytics endpoint is read-only over Delta CREATE VIEW, CREATE FUNCTION and CREATE PROCEDURE all work. CREATE TABLE does not — tables only come from Spark. If you're recreating a Lakehouse structure in a target tenant, that's two separate artefacts: a .sql script for the views and a PySpark notebook for the empty table structure. They also have to run in that order. 4) Notebook writes need ?format=ipynb Creating or updating a notebook definition without it returns "PyToIpynbFailure: Convert py to ipynb failed" with no further explanation. Add the query parameter on both createItem and updateDefinition. Only notebooks need it. 5) Delta rejects column names with spaces or punctuation AnalysisException [DELTA_INVALID_CHARACTERS_IN_COLUMN_NAMES] for anything containing space , ; { } ( ) newline tab = The obvious fix is to sanitise the names, but don't — your views and semantic models reference the original names. Enable column mapping on those tables instead: delta.columnMapping.mode = name, minReaderVersion = 2, minWriterVersion = 5. Original names preserved, Delta happy. 6) Premium_ASWL_Error on refresh does not mean you need a gateway Most threads will tell you to set up an on-premises data gateway. In my case the model was set to "Default: Single Sign-On" and simply needed an explicit cloud connection created and bound to it. No gateway involved. Worth checking before you go down the gateway route. Bonus: pipelines fail with a bare "UnknownError" when they reference items that aren't in the target yet. The error never says which ones. I ended up scanning pipeline definitions for GUIDs and matching them against the source inventory to list the missing items by name before attempting the deploy, and deploying master pipelines after their children. Happy to go deeper on any of these if it's useful.

Duplicate Event-Based Triggers for Same OneLake File Despite Filtering on data.api = FlushWithClose

mahadev93 — Wed, 23 Sep 2026 05:35:58 GMT

Hi Team,

We are using a Microsoft Fabric Eventstream source configured with:

Event Type: Microsoft.Fabric.OneLake.FileCreated
Subject filter: /Files/people_counting/camera_history/ingest
Additional filter: data.api = FlushWithClose

However, we are still observing two pipeline triggers for the same file.

For a single file: /Files/people_counting/camera_history/ingest/visitation_people_YYYYMMDDHHMMSS.txt. the Eventstream shows multiple FileCreated events, and the downstream pipeline is triggered twice. Our expectation is that one completed file should result in only one pipeline execution.

Questions:

Is duplicate triggering a known behavior for Microsoft.Fabric.OneLake.FileCreated events?
Is filtering on data.api = FlushWithClose sufficient, or are additional filters required?
Is there a recommended deduplication strategy for Eventstream-triggered Fabric pipelines?
How can we guarantee exactly one pipeline trigger per file arrival?

Any guidance from the Fabric team or anyone who has implemented OneLake event-based ingestion would be appreciated.

SharePoint Excel ingestion, Dataflow Gen2 vs shortcut + notebook CU efficiency

bj132175 — Tue, 22 Sep 2026 04:26:35 GMT

I need to ingest and transform multiple Excel files stored in SharePoint folders.

I am comparing:

Pure Dataflow Gen2 using SharePoint.Files and Power Query transformations.
Lakehouse Files shortcut to the SharePoint folder, followed by all transformations in a Fabric notebook using PySpark.

For the same files, transformations, output, schedule, and capacity:

Is the shortcut + notebook approach expected to consume fewer Fabric Capacity Units than pure Dataflow Gen2?

What is the recommended approach for SharePoint Excel-folder ingestion?

How to Dynamically Select Lakehouse Tables with Different Schemas Using Pipeline Parameters?

Imran2706 — Mon, 21 Sep 2026 17:29:30 GMT

Hi Fabric Community,
I am trying to build a metadata-driven / parameterized Dataflow Gen2 where the user provides a table name only once in the Pipeline, and the Dataflow should dynamically read that Lakehouse table and write the result to the corresponding destination table.
My requirement
I have a Lakehouse containing multiple tables, for example:
Lakehouse

Sales
Customer
Employee

The tables have different schemas.
For example:
Sales:
OrderID
CustomerID
OrderDate
Region
Amount
Status

Customer:
CustomerID
CustomerName
City
Country
Phone

Employee:
EmployeeID
EmployeeName
Department
Salary
JoiningDate

The user should not have to open Dataflow Gen2 and manually change a parameter every time.

I want the user to provide only: Pipeline parameter: table_name = Customer
The Pipeline should pass this value to the Dataflow Gen2 parameter: p_table_name = @pipeline().parameters.table_name

Then the Dataflow should dynamically select the Customer table. If the user runs the Pipeline again with:
table_name = Sales
the same Dataflow should dynamically select Sales.

My Dataflow Gen2 setup
I created a Dataflow Gen2 with a parameter:
Parameter name: p_table_name
Type: Text/String
I enabled: Enable parameters to be discovered and overridden for execution

My source is a Lakehouse. The Dataflow query structure is approximately:

p_table_name
Source
Navigation 1
Navigation 2
Filtered rows
Custom

Navigation 2 contains metadata such as:
Name
Id
Data
Schema
ItemKind
ItemName
IsLeaf

I filter the table using the following M expression:
Table.SelectRows(#"Navigation 2", each [Name] = p_table_name)

Then I select the actual table data using: #"Filtered rows"{0}[Data]
This works correctly in the Dataflow editor.

For example, when:
p_table_name = Sales
the Custom step shows the Sales data.

When:
p_table_name = Customer
the Custom step shows the Customer data.
So the dynamic source selection is working

Pipeline setup
I created a Pipeline parameter:
Name: table_name
Type: String

In the Dataflow activity, I map:
p_table_name = @pipeline().parameters.table_name
So the intended flow is:

Pipeline parameter
table_name
Dataflow activity
p_table_name
Dynamic Lakehouse table selection
Destination setup

The Dataflow has a Lakehouse destination

I parameterized the destination table name using p_table_name

The destination is configured approximately as:
Lakehouse
Schema: dbo
Table name: p_table_name
Update method: Replace
Schema option: Dynamic schema

I also tested the destination configuration with Dynamic schema.

The problem
When the Pipeline parameter and Dataflow parameter have the same value, the Pipeline succeeds.
For example:
Dataflow p_table_name = Customer
Pipeline table_name = Customer

Result: Pipeline succeeds
Customer data is loaded to the destination However, when I leave the Dataflow design-time parameter as:
p_table_name = Customer and run the Pipeline with: table_name = Sales

the Pipeline/Dataflow fails with:

Dataflow refresh job failed with status: Failed.

Failure reason:

requestId: ...

errorCode: EntityUserFailure

message: Something went wrong, please try again later.

The same happens in the reverse situation:

Dataflow p_table_name = Sales
Pipeline table_name = Customer

The Pipeline fails.

Important observation

If both values are the same:

Dataflow = Sales
Pipeline = Sales

it works.

If:

Dataflow = Customer
Pipeline = Customer

it works.

But if:

Dataflow = Customer
Pipeline = Sales
it fails.

This makes me wonder whether the runtime Pipeline parameter is actually overriding the Dataflow parameter correctly, or whether the problem is related to the destination or schema.

My main question

Since Dataflow Gen2 supports public parameters that can be overridden at Pipeline execution time, shouldn't I be able to do the following?

Pipeline:
table_name = Sales
Dataflow:
p_table_name = Sales
Source:
Sales
Destination:
Sales

without manually changing the Dataflow's p_table_name?

In other words, I want the Pipeline parameter to be the only value the user has to change.

The user should never have to:

1. Open the Dataflow.

2. Change p_table_name.

3. Save or publish the Dataflow.

4. Go back to the Pipeline.

5. Run it.

Instead, they should only do:

Pipeline -> table_name = Sales -> Run

or:

Pipeline -> table_name = Customer -> Run

Additional question about different schemas

Because Sales, Customer, and Employee have completely different schemas, is this scenario supported with one parameterized Dataflow Gen2?

For example:

table_name = Sales

Sales schema

Sales destination

and:

table_name = Customer

Customer schema

Customer destination

using the same Dataflow?

I have selected:

Update method = Replace

Schema option = Dynamic schema

I also noticed the destination has:

Use automatic settings

Would enabling automatic settings allow the destination schema to dynamically adapt to the selected table?

Expected behavior

I am trying to achieve the following:

Pipeline

table_name

Sales / Customer / Employee

Dataflow Gen2

p_table_name

Dynamic Lakehouse source

Dynamic destination

The key requirement is:

The user should provide the table name only once in the Pipeline, and the same Dataflow should dynamically process whichever Lakehouse table was selected, even when the tables have different schemas.

Is this supported in Dataflow Gen2?

If yes, what is the correct configuration for the source and destination?

If not, what is the recommended Fabric architecture for achieving this requirement?

Thank you.

How to design a metadata-driven framework to run 600+ Fabric notebooks with parallel execution?

sunilyadavv4u — Sat, 19 Sep 2026 12:52:35 GMT

Hi Fabric Community,

We are currently planning to migrate 600+ notebooks from Azure Databricks to Microsoft Fabric.

Our current requirement is to build a metadata-driven orchestration framework using:

Fabric Data Pipelines for orchestration
SQL Database for metadata/control tables
Fabric Notebooks for data processing
OneLake/Lakehouse as the target storage

Instead of creating separate pipelines for each notebook, we would like to have a generic metadata-driven pipeline that dynamically reads notebook information from SQL Database and executes the required notebooks.

For example, our metadata table could contain:

Notebook ID
Notebook name/path
Priority
Dependency
Parameters
Data volume
Expected runtime
Compute/pool requirement
Retry count
Active/inactive flag

Our main challenge

We may need to execute approximately 40 notebooks in parallel.

We would like to understand the recommended Fabric architecture for this scenario.

Specifically:

How should we design the metadata-driven pipeline to dynamically execute 600+ Fabric notebooks?
How should we control parallelism when around 40 notebooks need to run simultaneously?
Should we use a single Spark environment/pool, multiple environments/pools, or some other approach?
How should we decide which Spark compute configuration should be used for each notebook based on:
- Data volume
- Memory requirement
- Processing time
- Shuffle-intensive workloads
- Small vs. large workloads?
If 40 notebooks run simultaneously, how does Fabric manage the underlying Spark compute and capacity? Should we be concerned about resource contention or throttling?
Is it recommended to classify notebooks into workload groups such as:and then control concurrency separately for each group?
Small / Medium / Large / XLarge
What is the recommended way to implement dependency management? For example:where Notebook B should only start after A succeeds.
Notebook A → Notebook B → Notebook C
What is the recommended approach for retry, failure handling, logging, and restartability for 600+ notebooks?
Is there a recommended metadata-driven orchestration pattern/reference architecture in Microsoft Fabric for this scale?
Are there any Fabric-specific limitations or best practices we should consider when running hundreds of Spark notebooks with high concurrency?

Our main objective is to build a scalable framework where we can manage 600+ notebooks from metadata instead of maintaining hundreds of individual pipelines, while still controlling Spark compute and parallel execution efficiently.

Any architectural recommendations, reference implementations, or real-world experience with similar workloads would be highly appreciated.

Thanks!

Fabric Pipeline Error

odtJitendra — Fri, 18 Sep 2026 19:00:07 GMT

Today, we noticed that our production pipeline started failing with the following error:

Error:

BadRequest Error fetching pipeline default identity userToken{

"code": "LSROBOTokenFailure","message": "AADSTS50173: The provided grant has expired due to it being revoked, a fresh auth token is needed. The user might have changed or reset their password. The grant was issued on '2026-08-24T05:50:05.7907970Z' and the TokensValidFrom date (before which tokens are not valid) for this user is '2026-09-18T16:05:07.0000000Z'. Trace ID: 250ed633-08f0-4dcb-bad0-11596983e901 Correlation ID: 430f0833-fbc4-44f8-b19a-883ada2cad74 Timestamp: 2026-09-18 18:29:30Z",

"target": "PipelineDefaultIdentity-3e591cb6-2704-49f2-9efd-7196287c9feb","details": null,"error": null }FetchUserTokenForPipelineAsync

Has anyone encountered this issue before or have any idea what could be causing it?

Any suggestions on how to resolve this would be appreciated.

Designing AI Agent Workflows on Modern Data Platforms

codeautomation — Fri, 18 Sep 2026 15:21:22 GMT

Hi everyone,

I have been exploring how AI agents can work with modern data engineering platforms to automate business processes.

A common architecture I see is that data pipelines collect and prepare information, AI agents analyze the context and determine the next action, workflow services execute tasks through APIs or connected systems, and the results are stored for reporting or further processing.

I am interested in how teams are approaching this with Microsoft Fabric.

Are you using AI agents directly with Fabric workflows, or keeping the AI layer separate?

What patterns work well for connecting AI agents with Lakehouse, Data Pipelines, or notebooks?

How do you manage security and permissions when AI applications need access to enterprise data?

Are there recommended approaches for combining Fabric workloads with external AI services?

I would be interested to hear what architecture patterns others are using and what challenges you have encountered.

Optimising Fabric Capacity Usage

FabricEnjoyer — Fri, 18 Sep 2026 06:08:18 GMT

Hi all,

Due to the recent changes in Fabric capacity metering, I've been spending quite a bit of time trying to optimise our CU consumption. The challenge I've run into is that it's difficult to distinguish between SQL activity generated by user queries versus SQL activity generated during semantic model refreshes, which makes it harder to accurately evaluate different architectural approaches.

Approach 1: Materialising Views as Delta Tables

Historically, our semantic models have queried SQL views hosted in Fabric. To reduce SQL Endpoint consumption, I began converting the T-SQL view logic into notebooks that materialise the results into Delta tables. The semantic models then connect directly to these Delta tables via the ADLS Gen2 connector in Power BI, effectively bypassing the SQL endpoint during refresh.

However, the results have been inconsistent. In some workspaces this appears to reduce overall CU consumption, while in others it actually increases costs because the notebook execution itself incurs significant compute usage.

Current new architecture(old queried straight from omdb sql views):

My questions are:

From a CU consumption perspective, is it generally more efficient to:
- Have semantic models query Fabric SQL views directly, or
- Materialise those views into Delta tables and have semantic models consume the Delta tables through ADLS Gen2?
Has anyone compared the total cost of repeatedly querying views during semantic model refreshes versus the cost of creating and maintaining materialised Delta tables?
While I understand that a star schema is considered best practice for reporting and semantic modelling, if the same business logic can be represented through a set of SQL views, is there still a compelling reason to physically materialise the data into tables from a cost-efficiency perspective?

I've attempted to benchmark both approaches, but isolating the relevant consumption metrics has proven more difficult than expected.

Approach 2: Shifting Transformation Logic to the Semantic Model

Another approach I'm exploring is moving the transformation and view logic out of our Fabric workspace and into semantic models hosted in the client's Power BI Pro tenant.

My assumption is that this would reduce SQL Endpoint consumption within our Fabric capacity because less querying and transformation would occur on our side. However, I would expect semantic model refresh times to increase as more transformation work is pushed downstream.

My questions here are:

Would moving the transformation logic into the semantic model generally reduce or increase overall SQL-related CU consumption?
Have others seen meaningful capacity savings using this approach?
Is there a way to connect one semantic model to another semantic model without using DirectQuery? My concern is that DirectQuery would negatively impact performance and potentially introduce additional query-related costs.

I'd be very interested to hear how others are approaching this, particularly now that SQL Endpoint consumption has become much more visible within Fabric capacity metrics.

I have also reviewed the Query Insights for each workspace in an attempt to correlate SQL activity with overall CU consumption. However, this does not always translate into lower capacity usage. One of the challenges is that many users are querying the SQL Endpoint directly through Excel and other tools, without going through a semantic model at all. Under the new metering model, these ad hoc user queries appear to be a significant contributor to capacity consumption, making it difficult to isolate the impact of semantic model refreshes versus interactive user activity. As a result, determining whether a particular optimisation has genuinely reduced costs becomes far more challenging, as overall CU usage may be heavily influenced by user behaviour outside of the reporting layer.

Thanks in advance!

Storing and parsing HL7 data

ipkus — Thu, 17 Sep 2026 21:38:05 GMT

Any healthcare folks can share what best practices they have found around storing HL7 data for analytics ?

Diagnosing a slow Fabric Data Warehouse using the new skill

arthurfr23 — Thu, 17 Sep 2026 14:18:02 GMT

When a Fabric Data Warehouse slows down, the investigation usually means jumping between the Capacity Metrics app, Query Insights, and SQL pool diagnostics while manually correlating time ranges across three different tools.

Microsoft just shipped the SQL DW operations skill as Generally Available, you can find the official announcement here:

https://community.fabric.microsoft.com/t5/Fabric-Updates-Blog/Diagnose-Fabric-Data-Warehouse-workloads-with-the-SQL-DW/ba-p/5366102

I dug into what this actually changes in practice: how the skill consolidates capacity spikes, query regressions, and request failures into a single conversational interface, and where it still has blind spots worth knowing before you rely on it in a production incident. There is at least one behavior around time range correlation that surprised me during testing.

To do that, I wrote a full article, and would love to hear if others are already using this in their warehouse troubleshooting flow.

https://medium.com/@arthurfr23/diagnosing-fabric-data-warehouse-with-the-sql-dw-operations-skill-investigating-workloads-without-e4060fe8661e?sharedUserId=arthurfr23

Copy Job - Amazon S3 connector fails with region signing error on generic endpoint

Sharan_Srini99 — Thu, 17 Sep 2026 03:24:05 GMT

I'm trying to connect Fabric's Amazon S3 connector (Copy activity, pipelines) to a bucket in ap-southeast-2, using Access Key authentication.

When I create a connection with the generic regional endpoint:

https://s3.ap-southeast-2.amazonaws.com

connecting fails with:

Expression.Error: Invalid Url. The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'ap-southeast-2'

I've added s3:ListAllMyBuckets, s3:ListBucket, and s3:GetBucketLocation to the IAM user, and the error persists identically.

Building Self-Healing Data Pipelines

ipkus — Wed, 16 Sep 2026 21:34:37 GMT

This article shares practical examples on building self healing piplelines and data quality. Do take a read and share your inputs.

https://medium.com/@rishabh.gulati_94109/ms-fabric-building-self-healing-data-ecosystems-67a7e83c5380

Can a service principal with federated credentials be used for SQL cloud connection auth?

CharlieFab — Wed, 16 Sep 2026 00:00:14 GMT

Replacing OAuth 2.0 personal-account auth on our Fabric SQL cloud connections. Workspace identity doesn't fit, because semantic models in other workspaces share the same connection and each workspace has its own identity — those refreshes fail with invalid credentials unless every workspace identity is granted on the database.

A service principal solves that, but I'd like to avoid storing a secret. I added a federated identity credential to the app registration. In the connection settings, Service Principal auth only offers a key or a certificate, with nowhere to reference the federated credential.

Is that expected? I assume federation needs the caller to present a token it already holds, and the Fabric refresh engine has none — so it simply can't apply. Confirmation would be useful, as would any roadmap or Ideas link for secret less cloud connection auth.

If certificate is the answer, I'd also welcome hearing how others handle cert rotation across many connections given the encrypted payload on the update credentials API.\

Need to Migrate SP written in Synapse to Fabric

maxravi — Tue, 15 Sep 2026 14:50:23 GMT

Hello,

I need to migrate my stored prodecure SP from synapse to Fabric, can you please help me with best approach. I am seeing a lot of syntax changes. Suggest me some best, quickest and efficient ways to migrate.

Thanks,