All Data Engineering posts

Diagnosing a slow Fabric Data Warehouse using the new skill

arthurfr23 — Thu, 17 Sep 2026 14:18:02 GMT

When a Fabric Data Warehouse slows down, the investigation usually means jumping between the Capacity Metrics app, Query Insights, and SQL pool diagnostics while manually correlating time ranges across three different tools.

Microsoft just shipped the SQL DW operations skill as Generally Available, you can find the official announcement here:

https://community.fabric.microsoft.com/t5/Fabric-Updates-Blog/Diagnose-Fabric-Data-Warehouse-workloads-with-the-SQL-DW/ba-p/5366102

I dug into what this actually changes in practice: how the skill consolidates capacity spikes, query regressions, and request failures into a single conversational interface, and where it still has blind spots worth knowing before you rely on it in a production incident. There is at least one behavior around time range correlation that surprised me during testing.

To do that, I wrote a full article, and would love to hear if others are already using this in their warehouse troubleshooting flow.

https://medium.com/@arthurfr23/diagnosing-fabric-data-warehouse-with-the-sql-dw-operations-skill-investigating-workloads-without-e4060fe8661e?sharedUserId=arthurfr23

Re: Fabric Copy Activity fails in Query mode but succeeds in Table mode through an on-premises gateway

alisaleh — Thu, 17 Sep 2026 12:29:48 GMT

Hi,

Im not sure how this is going to help. I know the query works as I have tested and simplified multiple times by reducing or removing certain filters or reducing columns, however I still got the same error . Not only that but I used Copy Job instead of copy activity and set it to be incremental with using watermark column option on the source table. this kind of setup and despite the source is set to a direct table not TSQL still fails with the same error because internally the copy job will produce query against the source table to fetch data with where clause filter on the watermark column based on last saved watermark value. So Im 100% sure that it doesnt have anything to do with the query. By the way when I created this setup initially my sql query had some syntax issue but I was getting a different error indicating as such and not connectivity error like the one posted in the initial message.

Re: Building Self-Healing Data Pipelines

v-csrikanth — Thu, 17 Sep 2026 10:45:57 GMT

Hi ipkus
Thanks for reaching out to Fabric community.

Re: Copy Job - Amazon S3 connector fails with region signing error on generic endpoint

v-achippa — Thu, 17 Sep 2026 10:37:13 GMT

Hi Sharan_Srini99,

Thank you for reaching out to Microsoft Fabric Community.

Here the error is that the S3 request is being signed for the wrong AWS region. Fabric is signing the request for us-east-1 but the S3 bucket is in ap-southeast-2.

Please configure the Amazon S3 connection using the bucket-specific endpoint instead of the generic regional endpoint:
https://<bucket-name>.s3.ap-southeast-2.amazonaws.com

Then test the connection and rerun the Copy Job. This error is caused by the region mismatch, not by the IAM permissions.

Thanks and regards,
Anjan Kumar Chippa

Re: Fabric Copy Activity fails in Query mode but succeeds in Table mode through an on-premises gateway

v-aatheeque — Thu, 17 Sep 2026 10:20:49 GMT

Hi alisaleh

Thanks for the clarification. That helps narrow down the issue.

Since the same connections work successfully in Table mode, while Query/T-SQL mode fails for both the Lakehouse and SQL endpoint connections, it looks like the issue is specific to the Query mode execution rather than the connection or the query itself.

As a next step, could you please try a simple query such as:

SELECT TOP 10 * FROM <table_name>

If the same error occurs, please share the full error details from the Copy Activity run along with a screenshot of the Source configuration

Re: Can a service principal with federated credentials be used for SQL cloud connection auth?

v-kathullac — Thu, 17 Sep 2026 10:01:03 GMT

Hi CharlieFab ,

Sorry for the above reply that was my mistake. I was about to share the correct debugging steps to help resolve your issue, but I was in a hurry and accidentally posted the wrong steps.

Please find the correct debugging steps below to resolve the issue.

In the current Fabric connection experience, Service Principal authentication supports a service principal key or certificate, as shown in your screenshot, but there is no option to reference an Entra Federated Identity Credential (FIC).

FIC requires the calling service to obtain and present a federated token, and the Fabric-managed refresh process currently doesn't expose a way to configure that federated identity for the connection. So, FIC cannot currently be selected as the credential for this connection.

Since your requirement is to avoid storing a client secret, Service Principal certificate authentication is the available option shown in the connection UI. You would need to manage the certificate lifecycle/rotation and update the connection when the certificate changes.

For your scenario, where the same connection is shared by semantic models across multiple workspaces, using a dedicated Service Principal with certificate authentication can avoid granting every workspace identity access to the SQL database.

Thanks,
Chaithanya.

Re: Business Central Data to Fabric

wardy912 — Thu, 17 Sep 2026 08:48:08 GMT

Hi Saum02, have you considered open mirroring using BC2ADLS?

https://youtu.be/A2HKJ7QDOKU?si=TPnitKnybDcxsuoO

Near real-time BC data in Fabric, 0 capacity usage and free storage (F16 - 16GB, F32 - 32GB etc)

--------------------------------

I hope this helps, please give kudos and mark as solved if it does!

Connect with me on LinkedIn.

Subscribe to my YouTube channel for Fabric/Power Platform related content!

Copy Job - Amazon S3 connector fails with region signing error on generic endpoint

Sharan_Srini99 — Thu, 17 Sep 2026 03:24:05 GMT

I'm trying to connect Fabric's Amazon S3 connector (Copy activity, pipelines) to a bucket in ap-southeast-2, using Access Key authentication.

When I create a connection with the generic regional endpoint:

https://s3.ap-southeast-2.amazonaws.com

connecting fails with:

Expression.Error: Invalid Url. The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'ap-southeast-2'

I've added s3:ListAllMyBuckets, s3:ListBucket, and s3:GetBucketLocation to the IAM user, and the error persists identically.

Re: Business Central Data to Fabric

CharlieFab — Thu, 17 Sep 2026 01:38:23 GMT

Hi Saum02

It depends on whether you're on BC Online or on-prem.

BC Online: Direct SQL isn't an option, because users can no longer access SQL Server directly once you move to cloud-based BC.

BC on-prem: You can connect to SQL, but I wouldn't build on it long term. Each company has its own tables, extensions add extra tables you have to join, FlowFields aren't stored, and upgrades can change the schema. It also isn't available for BC Online, so it could block a future move to the cloud.

API / Dataflow Gen2 BC connector: This is the supported route and fine for smaller data. For incremental loads, filter on lastModifiedDateTime and run queries on read scale-out.

Re: Can a service principal with federated credentials be used for SQL cloud connection auth?

CharlieFab — Thu, 17 Sep 2026 00:27:34 GMT

Kagiyama_yutaka Thanks for the clarification, the backup option I had was the workspace identities with no key rotation at all

Re: Can a service principal with federated credentials be used for SQL cloud connection auth?

CharlieFab — Thu, 17 Sep 2026 00:09:06 GMT

v-kathullac

Seems the incorrect thread

Building Self-Healing Data Pipelines

ipkus — Wed, 16 Sep 2026 21:34:37 GMT

This article shares practical examples on building self healing piplelines and data quality. Do take a read and share your inputs.

https://medium.com/@rishabh.gulati_94109/ms-fabric-building-self-healing-data-ecosystems-67a7e83c5380

Re: Current or upcoming DP-700 exam vouchers

tayloramy — Wed, 16 Sep 2026 20:53:36 GMT

Hi UshaSriG,

If you're still looking for a dp-700 voucher, and will be able to complete the exam in October, send me a message. My user group has a few vouchers left over and I would be happy to give you one :)

Re: Integrate Microsoft Fabric with Oracle Fusion Cloud EPM

sradhakr — Wed, 16 Sep 2026 19:24:47 GMT

Hi seblopezarteta
Were you able to bring the EPM data into your Lakehouse pipeline? What approach did you take?

Are you also bringing Fusion ERP data into your Lakehouse pipeline?

Thanks

Shankar

Re: GraphQL update... mutation for Fabric SQL DB failing

msturzen — Wed, 16 Sep 2026 19:07:24 GMT

Ping v-achippa

Re: GraphQL update... mutation for Fabric SQL DB failing

msturzen — Wed, 16 Sep 2026 19:07:00 GMT

Hi everyone,

After corresponding with MSFT support team I got some feedback and insights. Summing it up here hoping it might be useful to others some time. (This is the result as of Sept. 16th 2026)

On the traces support found the following error message: "The target table 'dbo.table' of the DML statement cannot have any enabled triggers if the statement contains an OUTPUT clause without INTO clause."

Addtional bit of info, that I stripped initially since I didn't think it could be the culprit 😯:
The table used has an update trigger (updates the updated_at column)
Fabric's GraphQL engine generates an UPDATE ... OUTPUT ... SQL statement (without an INTO clause)
This is done (my guess) to be able to return the requested columns without an additional select after the update (update does not return the updated rows directly)
SQL Server does not allow the above statement if there are enabled update triggers on the table. See: https://learn.microsoft.com/en-us/sql/t-sql/queries/output-clause-transact-sql?view=sql-server-ver17#triggers. This is due to timing/racing and consistency issues.

This leaves me with the following options:

Do not use triggers:
- Delete or disable (or do not create) them for the mutations you want to use (UPDATE trigger in my case), but then you lose the action the the trigger performs
- If the action is necessary replace the trigger logic (possible in my case: Caller could just add a value for UPDATED_AT column, but would everyone do so?)
Use a stored procedure to do the update with an INTO clause using a @table_variable

I'm not a big fan of either. I'd be very fond of the GraphQL implementation being changed to use the OUTPUT ... INTO ... solution suggested. It seems strange that a (imo) legitimate use of trigger is prohibited by an implementation detail of the GraphQL layer. I'm not very confident though this will land any time soon.

I want to thank Tharun from the support team for helping.

Best Martin

Re: GraphQL update... mutation for Fabric SQL DB failing

msturzen — Wed, 16 Sep 2026 18:59:32 GMT

After corresponding with MSFT support team I got some feedback and insights. Summing it up here hoping it might be useful to others some time. (This is the result as of Sept. 16th 2026)

On the traces support found the following error message: "The target table 'dbo.table' of the DML statement cannot have any enabled triggers if the statement contains an OUTPUT clause without INTO clause."

Additional bit of info, that I stripped initially since I didn't think it could be the culprit 😯:
The table used has an update trigger (updates the updated_at column)
Fabric's GraphQL engine generates SQL similar like this for the update mutation:
UPDATE _table_
SET _fields_
OUTPUT _requested-fields_ -- no 'INTO table_variable or table' option here!!!
WHERE _primary-key_
This is done to be able to return the requested columns without an additional select after the update (update does not return the updated rows directly)
SQL Server does not allow the above statement if there are enabled update triggers on the table. See: https://learn.microsoft.com/en-us/sql/t-sql/queries/output-clause-transact-sql?view=sql-server-ver17#triggers This is due to timing/racing and consistency issues.

This leaves me with the following options:

Do not use triggers:
- Delete or disable (or do not create) them for the mutations you want to use (UPDATE trigger in my case), but then you lose the action the the trigger performs
- If the action is necessary replace the trigger logic (possible in my case: Caller could just add a value for UPDATED_AT column, but would everyone do so?)
Use a stored procedure to do the update with an INTO clause using a table_variable

I want to thank Tharun from the support team for helping.

Best Martin

Re: Business Central Data to Fabric

Murtaza_Ghafoor — Wed, 16 Sep 2026 17:24:10 GMT

Saum02
There are multiple option you can exercise while connecting with BC. I would prefer these options

If Business Central Online, then it is not recommended to connect directly to the underlying SQL database. It’s not a supported approach and the schema can change with BC updates and extensions.

For Fabric, BC APIs/OData or other supported integration options is recommended.

For BC On-Premises, direct SQL extraction can be a good option, preferably through a read-only replica to avoid impacting the production BC database.

Please accept this answer yes and give kudos if this is helpful

Re: SharePoint Shortcut showing no Content

mkjit256 — Wed, 16 Sep 2026 13:59:59 GMT

Thanks DreamITLearn v-aatheeque GilbertQ Kagiyama_yutaka and ShivekMaharaj for your help.
When creating a new connection from the mange gateway and connections page, using only the site name https://companyname.sharepoint.com/sites/BIA the problem was solved and i can browser the sub folders with no issue.

Re: Claude Code Integrate with Power Bi

Kagiyama_yutaka — Wed, 16 Sep 2026 13:39:45 GMT

Power BI adds no token cost and the simple way is to run one small page end to end and check the Claude usage on that including the semantic model read, the page design, the PBIR generation, and the validation. Claude cost comes only from the model pricing and that single measurement becomes your baseline for the POC.