All Data Engineering posts

Building Self-Healing Data Pipelines

ipkus — Wed, 16 Sep 2026 21:34:37 GMT

This article shares practical examples on building self healing piplelines and data quality. Do take a read and share your inputs.

https://medium.com/@rishabh.gulati_94109/ms-fabric-building-self-healing-data-ecosystems-67a7e83c5380

Re: Current or upcoming DP-700 exam vouchers

tayloramy — Wed, 16 Sep 2026 20:53:36 GMT

Hi UshaSriG,

If you're still looking for a dp-700 voucher, and will be able to complete the exam in October, send me a message. My user group has a few vouchers left over and I would be happy to give you one :)

Re: Integrate Microsoft Fabric with Oracle Fusion Cloud EPM

sradhakr — Wed, 16 Sep 2026 19:24:47 GMT

Hi seblopezarteta
Were you able to bring the EPM data into your Lakehouse pipeline? What approach did you take?

Are you also bringing Fusion ERP data into your Lakehouse pipeline?

Thanks

Shankar

Re: GraphQL update... mutation for Fabric SQL DB failing

msturzen — Wed, 16 Sep 2026 19:07:24 GMT

Ping v-achippa

Re: GraphQL update... mutation for Fabric SQL DB failing

msturzen — Wed, 16 Sep 2026 19:07:00 GMT

Hi everyone,

After corresponding with MSFT support team I got some feedback and insights. Summing it up here hoping it might be useful to others some time. (This is the result as of Sept. 16th 2026)

On the traces support found the following error message: "The target table 'dbo.table' of the DML statement cannot have any enabled triggers if the statement contains an OUTPUT clause without INTO clause."

Addtional bit of info, that I stripped initially since I didn't think it could be the culprit 😯:
The table used has an update trigger (updates the updated_at column)
Fabric's GraphQL engine generates an UPDATE ... OUTPUT ... SQL statement (without an INTO clause)
This is done (my guess) to be able to return the requested columns without an additional select after the update (update does not return the updated rows directly)
SQL Server does not allow the above statement if there are enabled update triggers on the table. See: https://learn.microsoft.com/en-us/sql/t-sql/queries/output-clause-transact-sql?view=sql-server-ver17#triggers. This is due to timing/racing and consistency issues.

This leaves me with the following options:

Do not use triggers:
- Delete or disable (or do not create) them for the mutations you want to use (UPDATE trigger in my case), but then you lose the action the the trigger performs
- If the action is necessary replace the trigger logic (possible in my case: Caller could just add a value for UPDATED_AT column, but would everyone do so?)
Use a stored procedure to do the update with an INTO clause using a @table_variable

I'm not a big fan of either. I'd be very fond of the GraphQL implementation being changed to use the OUTPUT ... INTO ... solution suggested. It seems strange that a (imo) legitimate use of trigger is prohibited by an implementation detail of the GraphQL layer. I'm not very confident though this will land any time soon.

I want to thank Tharun from the support team for helping.

Best Martin

Re: GraphQL update... mutation for Fabric SQL DB failing

msturzen — Wed, 16 Sep 2026 18:59:32 GMT

After corresponding with MSFT support team I got some feedback and insights. Summing it up here hoping it might be useful to others some time. (This is the result as of Sept. 16th 2026)

On the traces support found the following error message: "The target table 'dbo.table' of the DML statement cannot have any enabled triggers if the statement contains an OUTPUT clause without INTO clause."

Additional bit of info, that I stripped initially since I didn't think it could be the culprit 😯:
The table used has an update trigger (updates the updated_at column)
Fabric's GraphQL engine generates SQL similar like this for the update mutation:
UPDATE _table_
SET _fields_
OUTPUT _requested-fields_ -- no 'INTO table_variable or table' option here!!!
WHERE _primary-key_
This is done to be able to return the requested columns without an additional select after the update (update does not return the updated rows directly)
SQL Server does not allow the above statement if there are enabled update triggers on the table. See: https://learn.microsoft.com/en-us/sql/t-sql/queries/output-clause-transact-sql?view=sql-server-ver17#triggers This is due to timing/racing and consistency issues.

This leaves me with the following options:

Do not use triggers:
- Delete or disable (or do not create) them for the mutations you want to use (UPDATE trigger in my case), but then you lose the action the the trigger performs
- If the action is necessary replace the trigger logic (possible in my case: Caller could just add a value for UPDATED_AT column, but would everyone do so?)
Use a stored procedure to do the update with an INTO clause using a table_variable

I want to thank Tharun from the support team for helping.

Best Martin

Re: Business Central Data to Fabric

Murtaza_Ghafoor — Wed, 16 Sep 2026 17:24:10 GMT

Saum02
There are multiple option you can exercise while connecting with BC. I would prefer these options

If Business Central Online, then it is not recommended to connect directly to the underlying SQL database. It’s not a supported approach and the schema can change with BC updates and extensions.

For Fabric, BC APIs/OData or other supported integration options is recommended.

For BC On-Premises, direct SQL extraction can be a good option, preferably through a read-only replica to avoid impacting the production BC database.

Please accept this answer yes and give kudos if this is helpful

Re: SharePoint Shortcut showing no Content

mkjit256 — Wed, 16 Sep 2026 13:59:59 GMT

Thanks DreamITLearn v-aatheeque GilbertQ Kagiyama_yutaka and ShivekMaharaj for your help.
When creating a new connection from the mange gateway and connections page, using only the site name https://companyname.sharepoint.com/sites/BIA the problem was solved and i can browser the sub folders with no issue.

Re: Claude Code Integrate with Power Bi

Kagiyama_yutaka — Wed, 16 Sep 2026 13:39:45 GMT

Power BI adds no token cost and the simple way is to run one small page end to end and check the Claude usage on that including the semantic model read, the page design, the PBIR generation, and the validation. Claude cost comes only from the model pricing and that single measurement becomes your baseline for the POC.

Re: Can a service principal with federated credentials be used for SQL cloud connection auth?

Kagiyama_yutaka — Wed, 16 Sep 2026 12:17:06 GMT

Fabric SQL cloud connections only list service‑principal sign‑in by secret and by certificate. The refresh engine has no flow for getting a token through OIDC federation, so adding a federated credential to the app does not give Fabric a way to use it.

A small cert added and re‑saved in the connection settings is the simple and safe workaround beginners can use, and cert rotation just means re‑saving the updated cert in the same place.

Re: Bronze data contracts for REST/API sources — workaround for Get Metadata's connector gap

v-achippa — Wed, 16 Sep 2026 11:49:14 GMT

Hi damo_mac85,

Your current workaround is a valid approach. Since Get Metadata does not support REST/API sources, you can first land the API response as a file in the Lakehouse and then validate its schema from the landed file.

If the schema validation succeeds, you can proceed with writing the data to the Delta table. If it fails, stop the pipeline and handle the schema change before loading the data.

Thanks and regards,
Anjan Kumar Chippa

Re: SharePoint Shortcut showing no Content

v-aatheeque — Wed, 16 Sep 2026 10:21:22 GMT

Hi mkjit256

Following up to confirm if the earlier responses addressed your query. If not, please share your questions and we’ll assist further.

Re: Business Central Data to Fabric

v-abhinavmu — Wed, 16 Sep 2026 09:48:38 GMT

Hi Saum02,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.

Thank you.

Re: Can a service principal with federated credentials be used for SQL cloud connection auth?

v-kathullac — Wed, 16 Sep 2026 09:32:54 GMT

Hi CharlieFab ,

Thank you for reaching out to Microsoft Fabric Community Forum, Below are the few points which can resolve your issue. Let us know if you need any further assistance.

Yes, this layout can be achieved in Power BI. The exact configuration depends on whether the fields are actual hierarchy levels or cost categories. If they are hierarchy levels, configure the Matrix as:

Rows: Customer → Job ID → Production → Department → Work Center
Values: Estimated Cost, Actual Cost, Variance

Then enable the +/- icons and stepped layout/drill-down so users can expand from Customer → Job ID → Production → Department → Work Center.

If Sales Price, Material, Outside Services, and Production are cost categories that should appear as separate rows with their corresponding Estimated Cost, Actual Cost, and Variance, I recommend using a disconnected Category table with SWITCH() measures instead of adding all of them as hierarchy levels.

This approach provides more control over the display and can more closely match the required layout. If you share the underlying fields/columns and the expected final Matrix layout, we can determine the appropriate approach and provide the exact DAX measures.

Thanks,
Chaithanya

Can a service principal with federated credentials be used for SQL cloud connection auth?

CharlieFab — Wed, 16 Sep 2026 00:00:14 GMT

Replacing OAuth 2.0 personal-account auth on our Fabric SQL cloud connections. Workspace identity doesn't fit, because semantic models in other workspaces share the same connection and each workspace has its own identity — those refreshes fail with invalid credentials unless every workspace identity is granted on the database.

A service principal solves that, but I'd like to avoid storing a secret. I added a federated identity credential to the app registration. In the connection settings, Service Principal auth only offers a key or a certificate, with nowhere to reference the federated credential.

Is that expected? I assume federation needs the caller to present a token it already holds, and the Fabric refresh engine has none — so it simply can't apply. Confirmation would be useful, as would any roadmap or Ideas link for secret less cloud connection auth.

If certificate is the answer, I'd also welcome hearing how others handle cert rotation across many connections given the encrypted payload on the update credentials API.\

Re: Approach for automating Dataflow Gen2 connection binding across Dev/Prod?

lavginqo2 — Tue, 15 Sep 2026 23:53:04 GMT

Thanks — this is useful, particularly the definition APIs. One clarification: source connection rebinding is already solved on our side.

It is working now as there was something else that I missed.

Thanks !

Re: Approach for automating Dataflow Gen2 connection binding across Dev/Prod?

ShivekMaharaj — Tue, 15 Sep 2026 19:15:28 GMT

Hi lavginqo2,

I think there is one newer API capability worth adding here, because the current situation is slightly more nuanced than "there is no REST way to rebind it."

You are correct that Dataflow Gen2 still does not have a native Dev → Prod connection deployment rule. Microsoft's current Dataflow Gen2 CI/CD architecture guidance explicitly says that data-source connections are statically bound and cannot be switched using workspace variables or Dataflow parameters. Deployment rules don't currently alter Dataflow connections either.

However, Microsoft now exposes the Dataflow's public definition through the Fabric REST API.

The documented Dataflow definition format includes a connections collection inside queryMetadata.json, and each entry contains the connection path, kind, and connectionId.

That opens up a post-deployment automation pattern:

Deploy Dev → Prod → get the Prod Dataflow definition → replace the Dev connection ID with the corresponding Prod connection ID → update the definition → publish the Dataflow

Microsoft provides both Get Dataflow Definition and Update Dataflow Definition endpoints for this.

I would therefore separate environment configuration from the Dataflow deployment itself.

For example:

Create the required shareable connections in Prod once, ideally with stable naming.
Maintain a small environment mapping such as DevConnectionId → ProdConnectionId.
Deploy the Dataflow through your normal Fabric CI/CD process.
Call getDefinition on the deployed Dataflow.
Decode queryMetadata.json and replace the relevant connections[].connectionId values.
Send the modified definition through updateDefinition.
Publish the Dataflow and run a validation refresh.

Fabric also has public connection APIs for creating, listing, testing and managing the actual connection objects, so provisioning the Prod connections can be automated separately.

One important distinction is that the Update Connection API isn't itself a Dataflow-rebinding operation. It updates an existing connection object. The Dataflow-to-connection reference is represented in the Dataflow definition, which is why I would change the connectionId there instead.

For ~47 tables, this is the route I would prototype before manually reconnecting every query or splitting the Dataflow purely to make the manual work manageable. If those queries share a small number of underlying source connections, you are effectively translating a small connection map rather than maintaining 47 independent bindings by hand.

One caveat: Microsoft's API documentation is currently a little inconsistent around unattended identities. The specific Update Dataflow Definition API lists service principals and managed identities as supported, while the broader Dataflow Gen2 public-API documentation still lists service-principal authentication as a limitation. I would therefore validate the full getDefinition → updateDefinition → publish sequence with your intended CI/CD identity before making it the production deployment path.

So I agree that the lack of a first-class connection deployment rule is still a gap, but with the newer public Dataflow definition APIs I don't think manual per-query rebinding necessarily has to be the end state anymore.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Re: Claude Code Integrate with Power Bi

ShivekMaharaj — Tue, 15 Sep 2026 19:04:20 GMT

Hi Yash8160,

v-achippa is right that there isn't a fixed token/credit figure per Power BI page. I would add one billing distinction that is important when estimating the POC.

The Power BI Report Authoring skill is the Power BI-specific instruction/tooling layer used by the coding agent. Microsoft currently lists Claude Code as a compatible agent, but the token consumption comes from the Claude model and the amount of agent work required rather than from Fabric assigning a fixed number of credits to each report or visual.

The cost therefore depends on things such as:

which Claude model you use;
how much of the semantic model/report definition has to be inspected;
number and complexity of pages/visuals;
how many authoring and validation iterations are required;
how much existing PBIR/PBIP context is repeatedly read.

I would also first establish how Claude Code will be billed. Anthropic documents that Claude Code can be used through a Pro/Max subscription allocation or through pay-as-you-go API/Console usage. Those are different models: subscription usage is governed by the plan's usage limits, while API-credit usage is charged at the model's token rates.

For a POC, I would therefore benchmark one representative report workflow rather than trying to estimate from page count alone.

For example:

Pick one page that is representative of the final POC - not the simplest page.
Run the complete workflow you expect to use: inspect the semantic model → plan/design the page → author the PBIR → open/validate it → make any corrections.
Record the actual Claude usage/cost for that complete iteration.
Repeat with one simple page and one complex page if the report varies significantly.
Use those measurements to estimate the remaining pages and add some allowance for rework and prompt iteration.

Microsoft's current Power BI agentic authoring guidance also separates report planning, design, authoring and management into different skills, so I would include whichever of those steps you intend to use when measuring the baseline.

If you are using Anthropic API/pay-as-you-go rather than staying within a Claude subscription, you can then convert the measured model usage using Anthropic's current token pricing. If you are using Pro/Max, I would monitor the remaining Claude Code allocation instead of treating every page as a directly billed token cost.

So I would avoid quoting a generic "X credits per report" number. One or two representative end-to-end builds using your actual semantic model will give you a much more defensible POC estimate.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Re: Is it possible connect via a guest account to a Delta table in a Lakehouse from another tenant.

ShivekMaharaj — Tue, 15 Sep 2026 18:58:55 GMT

Hi FabricEnjoyer,

I don't think the Delta table itself would be my first suspect here.

Microsoft explicitly supports accessing OneLake through existing ADLS/Blob APIs using the onelake.dfs.fabric.microsoft.com endpoint, as described in the OneLake connection guidance. Also, because at least one B2B guest can create the same connection successfully, I wouldn't describe guest access through this path as universally unsupported.

The fact that several guests from one customer fail with a 404 while another customer's guest works makes me look more closely at the identity and authorization path.

I would check these areas in this order:

1. Fabric/OneLake tenant settings

Verify that Guest users can access Microsoft Fabric is enabled for the affected guests.

I would also check the separate OneLake setting Users can access data stored in OneLake with apps external to Fabric. Microsoft's OneLake tenant-settings documentation describes this setting as controlling access from applications using ADLS APIs outside the Fabric environment.

This is especially worth checking if either setting is scoped to selected security groups rather than enabled tenant-wide.

2. The permissions on the shortcut target

This could be easy to miss in your architecture.

If the shortcuts from the Connection Lakehouse to the materialized Lakehouse are using passthrough authentication, OneLake evaluates the calling user's identity against the target as well. Microsoft's shortcut security documentation explains that passthrough OneLake-to-OneLake shortcuts require the user to be authorized on both the shortcut path and the target path.

So I would compare a working guest and a failing guest not only on the Connection Lakehouse, but also on the Lakehouse containing the materialized Delta tables.

If the shortcut uses delegated authentication instead, then this particular requirement changes because the configured connection identity is used for the target.

3. OneLake security, if you are using it

If any of these permissions are being supplied through OneLake security roles, there is a specific B2B limitation worth checking. Microsoft's current OneLake security documentation states that when a OneLake security role is assigned to a B2B guest, the Entra External ID guest-access setting must be Guest users have the same access as members (most inclusive).

If you aren't using OneLake security roles, I wouldn't change that setting just for this test.

4. Compare the customers' Entra cross-tenant policies

Because one external customer's guest works while several users from another external organization fail, I would compare the organization-specific inbound B2B collaboration settings and Conditional Access policies for those two customer tenants.

Microsoft's cross-tenant access documentation confirms that B2B inbound access can be configured separately for individual external organizations, with those organization-specific settings overriding the defaults.

One diagnostic I would also use is to test the failing identity directly against the OneLake DFS API with a Storage-audience Entra token for the host tenant. If that request also fails, I would concentrate on the guest identity, tenant policy and shortcut permissions. If it succeeds while AzureStorage.DataLake() still returns 404, I would then clear the cached OneLake credentials in Power BI Desktop and investigate the connector's authentication context.

For a longer-term multi-customer design, you may also want to evaluate Fabric External Data Sharing. It was designed specifically for cross-tenant OneLake sharing and uses Fabric-to-Fabric authentication rather than requiring the consumer to access the source as a B2B guest. The source data remains in place, so it is still a no-copy pattern.

Given your current evidence though, my first comparison would be:

working guest vs failing guest → external-app tenant setting → shortcut target permissions → Entra cross-tenant policy

The fact that the behaviour follows particular customer identities/tenants is a strong reason to troubleshoot that path before changing the Delta or Power Query implementation.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Re: Need to Migrate SP written in Synapse to Fabric

EduardoCastro — Tue, 15 Sep 2026 18:05:21 GMT

Hello,

There is a challenge in migrating stored procedures from Synapse to Fabric Warehouse, the syntax differences you are seeing are expected because Fabric Warehouse supports a subset of T-SQL compared to Synapse. The best and quickest approach is to use the Fabric Migration Assistant, which is built natively into Fabric.

The reference for T-SQL limitations in Fabric Data Warehouse is the official Microsoft Learn, you can find it here: T-SQL surface area in Fabric Data Warehouse

In my experience, we converted the stored procedures to Notebooks (PySpark/SQL) instead of migrating them 1:1 to Fabric Warehouse stored procedures.