All Data Engineering posts

Re: Business Central Data to Fabric

Murtaza_Ghafoor — Wed, 16 Sep 2026 17:24:10 GMT

Saum02
There are multiple option you can exercise while connecting with BC. I would prefer these options

If Business Central Online, then it is not recommended to connect directly to the underlying SQL database. It’s not a supported approach and the schema can change with BC updates and extensions.

For Fabric, BC APIs/OData or other supported integration options is recommended.

For BC On-Premises, direct SQL extraction can be a good option, preferably through a read-only replica to avoid impacting the production BC database.

Please accept this answer yes and give kudos if this is helpful

Re: SharePoint Shortcut showing no Content

mkjit256 — Wed, 16 Sep 2026 13:59:59 GMT

Thanks DreamITLearn v-aatheeque GilbertQ Kagiyama_yutaka and ShivekMaharaj for your help.
When creating a new connection from the mange gateway and connections page, using only the site name https://companyname.sharepoint.com/sites/BIA the problem was solved and i can browser the sub folders with no issue.

Re: Claude Code Integrate with Power Bi

Kagiyama_yutaka — Wed, 16 Sep 2026 13:39:45 GMT

Power BI adds no token cost and the simple way is to run one small page end to end and check the Claude usage on that including the semantic model read, the page design, the PBIR generation, and the validation. Claude cost comes only from the model pricing and that single measurement becomes your baseline for the POC.

Re: Can a service principal with federated credentials be used for SQL cloud connection auth?

Kagiyama_yutaka — Wed, 16 Sep 2026 12:17:06 GMT

Fabric SQL cloud connections only list service‑principal sign‑in by secret and by certificate. The refresh engine has no flow for getting a token through OIDC federation, so adding a federated credential to the app does not give Fabric a way to use it.

A small cert added and re‑saved in the connection settings is the simple and safe workaround beginners can use, and cert rotation just means re‑saving the updated cert in the same place.

Re: Bronze data contracts for REST/API sources — workaround for Get Metadata's connector gap

v-achippa — Wed, 16 Sep 2026 11:49:14 GMT

Hi damo_mac85,

Your current workaround is a valid approach. Since Get Metadata does not support REST/API sources, you can first land the API response as a file in the Lakehouse and then validate its schema from the landed file.

If the schema validation succeeds, you can proceed with writing the data to the Delta table. If it fails, stop the pipeline and handle the schema change before loading the data.

Thanks and regards,
Anjan Kumar Chippa

Re: SharePoint Shortcut showing no Content

v-aatheeque — Wed, 16 Sep 2026 10:21:22 GMT

Hi mkjit256

Following up to confirm if the earlier responses addressed your query. If not, please share your questions and we’ll assist further.

Re: Business Central Data to Fabric

v-abhinavmu — Wed, 16 Sep 2026 09:48:38 GMT

Hi Saum02,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.

Thank you.

Re: Can a service principal with federated credentials be used for SQL cloud connection auth?

v-kathullac — Wed, 16 Sep 2026 09:32:54 GMT

Hi CharlieFab ,

Thank you for reaching out to Microsoft Fabric Community Forum, Below are the few points which can resolve your issue. Let us know if you need any further assistance.

Yes, this layout can be achieved in Power BI. The exact configuration depends on whether the fields are actual hierarchy levels or cost categories. If they are hierarchy levels, configure the Matrix as:

Rows: Customer → Job ID → Production → Department → Work Center
Values: Estimated Cost, Actual Cost, Variance

Then enable the +/- icons and stepped layout/drill-down so users can expand from Customer → Job ID → Production → Department → Work Center.

If Sales Price, Material, Outside Services, and Production are cost categories that should appear as separate rows with their corresponding Estimated Cost, Actual Cost, and Variance, I recommend using a disconnected Category table with SWITCH() measures instead of adding all of them as hierarchy levels.

This approach provides more control over the display and can more closely match the required layout. If you share the underlying fields/columns and the expected final Matrix layout, we can determine the appropriate approach and provide the exact DAX measures.

Thanks,
Chaithanya

Can a service principal with federated credentials be used for SQL cloud connection auth?

CharlieFab — Wed, 16 Sep 2026 00:00:14 GMT

Replacing OAuth 2.0 personal-account auth on our Fabric SQL cloud connections. Workspace identity doesn't fit, because semantic models in other workspaces share the same connection and each workspace has its own identity — those refreshes fail with invalid credentials unless every workspace identity is granted on the database.

A service principal solves that, but I'd like to avoid storing a secret. I added a federated identity credential to the app registration. In the connection settings, Service Principal auth only offers a key or a certificate, with nowhere to reference the federated credential.

Is that expected? I assume federation needs the caller to present a token it already holds, and the Fabric refresh engine has none — so it simply can't apply. Confirmation would be useful, as would any roadmap or Ideas link for secret less cloud connection auth.

If certificate is the answer, I'd also welcome hearing how others handle cert rotation across many connections given the encrypted payload on the update credentials API.\

Re: Approach for automating Dataflow Gen2 connection binding across Dev/Prod?

lavginqo2 — Tue, 15 Sep 2026 23:53:04 GMT

Thanks — this is useful, particularly the definition APIs. One clarification: source connection rebinding is already solved on our side.

It is working now as there was something else that I missed.

Thanks !

Re: Approach for automating Dataflow Gen2 connection binding across Dev/Prod?

ShivekMaharaj — Tue, 15 Sep 2026 19:15:28 GMT

Hi lavginqo2,

I think there is one newer API capability worth adding here, because the current situation is slightly more nuanced than "there is no REST way to rebind it."

You are correct that Dataflow Gen2 still does not have a native Dev → Prod connection deployment rule. Microsoft's current Dataflow Gen2 CI/CD architecture guidance explicitly says that data-source connections are statically bound and cannot be switched using workspace variables or Dataflow parameters. Deployment rules don't currently alter Dataflow connections either.

However, Microsoft now exposes the Dataflow's public definition through the Fabric REST API.

The documented Dataflow definition format includes a connections collection inside queryMetadata.json, and each entry contains the connection path, kind, and connectionId.

That opens up a post-deployment automation pattern:

Deploy Dev → Prod → get the Prod Dataflow definition → replace the Dev connection ID with the corresponding Prod connection ID → update the definition → publish the Dataflow

Microsoft provides both Get Dataflow Definition and Update Dataflow Definition endpoints for this.

I would therefore separate environment configuration from the Dataflow deployment itself.

For example:

Create the required shareable connections in Prod once, ideally with stable naming.
Maintain a small environment mapping such as DevConnectionId → ProdConnectionId.
Deploy the Dataflow through your normal Fabric CI/CD process.
Call getDefinition on the deployed Dataflow.
Decode queryMetadata.json and replace the relevant connections[].connectionId values.
Send the modified definition through updateDefinition.
Publish the Dataflow and run a validation refresh.

Fabric also has public connection APIs for creating, listing, testing and managing the actual connection objects, so provisioning the Prod connections can be automated separately.

One important distinction is that the Update Connection API isn't itself a Dataflow-rebinding operation. It updates an existing connection object. The Dataflow-to-connection reference is represented in the Dataflow definition, which is why I would change the connectionId there instead.

For ~47 tables, this is the route I would prototype before manually reconnecting every query or splitting the Dataflow purely to make the manual work manageable. If those queries share a small number of underlying source connections, you are effectively translating a small connection map rather than maintaining 47 independent bindings by hand.

One caveat: Microsoft's API documentation is currently a little inconsistent around unattended identities. The specific Update Dataflow Definition API lists service principals and managed identities as supported, while the broader Dataflow Gen2 public-API documentation still lists service-principal authentication as a limitation. I would therefore validate the full getDefinition → updateDefinition → publish sequence with your intended CI/CD identity before making it the production deployment path.

So I agree that the lack of a first-class connection deployment rule is still a gap, but with the newer public Dataflow definition APIs I don't think manual per-query rebinding necessarily has to be the end state anymore.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Re: Claude Code Integrate with Power Bi

ShivekMaharaj — Tue, 15 Sep 2026 19:04:20 GMT

Hi Yash8160,

v-achippa is right that there isn't a fixed token/credit figure per Power BI page. I would add one billing distinction that is important when estimating the POC.

The Power BI Report Authoring skill is the Power BI-specific instruction/tooling layer used by the coding agent. Microsoft currently lists Claude Code as a compatible agent, but the token consumption comes from the Claude model and the amount of agent work required rather than from Fabric assigning a fixed number of credits to each report or visual.

The cost therefore depends on things such as:

which Claude model you use;
how much of the semantic model/report definition has to be inspected;
number and complexity of pages/visuals;
how many authoring and validation iterations are required;
how much existing PBIR/PBIP context is repeatedly read.

I would also first establish how Claude Code will be billed. Anthropic documents that Claude Code can be used through a Pro/Max subscription allocation or through pay-as-you-go API/Console usage. Those are different models: subscription usage is governed by the plan's usage limits, while API-credit usage is charged at the model's token rates.

For a POC, I would therefore benchmark one representative report workflow rather than trying to estimate from page count alone.

For example:

Pick one page that is representative of the final POC - not the simplest page.
Run the complete workflow you expect to use: inspect the semantic model → plan/design the page → author the PBIR → open/validate it → make any corrections.
Record the actual Claude usage/cost for that complete iteration.
Repeat with one simple page and one complex page if the report varies significantly.
Use those measurements to estimate the remaining pages and add some allowance for rework and prompt iteration.

Microsoft's current Power BI agentic authoring guidance also separates report planning, design, authoring and management into different skills, so I would include whichever of those steps you intend to use when measuring the baseline.

If you are using Anthropic API/pay-as-you-go rather than staying within a Claude subscription, you can then convert the measured model usage using Anthropic's current token pricing. If you are using Pro/Max, I would monitor the remaining Claude Code allocation instead of treating every page as a directly billed token cost.

So I would avoid quoting a generic "X credits per report" number. One or two representative end-to-end builds using your actual semantic model will give you a much more defensible POC estimate.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Re: Is it possible connect via a guest account to a Delta table in a Lakehouse from another tenant.

ShivekMaharaj — Tue, 15 Sep 2026 18:58:55 GMT

Hi FabricEnjoyer,

I don't think the Delta table itself would be my first suspect here.

Microsoft explicitly supports accessing OneLake through existing ADLS/Blob APIs using the onelake.dfs.fabric.microsoft.com endpoint, as described in the OneLake connection guidance. Also, because at least one B2B guest can create the same connection successfully, I wouldn't describe guest access through this path as universally unsupported.

The fact that several guests from one customer fail with a 404 while another customer's guest works makes me look more closely at the identity and authorization path.

I would check these areas in this order:

1. Fabric/OneLake tenant settings

Verify that Guest users can access Microsoft Fabric is enabled for the affected guests.

I would also check the separate OneLake setting Users can access data stored in OneLake with apps external to Fabric. Microsoft's OneLake tenant-settings documentation describes this setting as controlling access from applications using ADLS APIs outside the Fabric environment.

This is especially worth checking if either setting is scoped to selected security groups rather than enabled tenant-wide.

2. The permissions on the shortcut target

This could be easy to miss in your architecture.

If the shortcuts from the Connection Lakehouse to the materialized Lakehouse are using passthrough authentication, OneLake evaluates the calling user's identity against the target as well. Microsoft's shortcut security documentation explains that passthrough OneLake-to-OneLake shortcuts require the user to be authorized on both the shortcut path and the target path.

So I would compare a working guest and a failing guest not only on the Connection Lakehouse, but also on the Lakehouse containing the materialized Delta tables.

If the shortcut uses delegated authentication instead, then this particular requirement changes because the configured connection identity is used for the target.

3. OneLake security, if you are using it

If any of these permissions are being supplied through OneLake security roles, there is a specific B2B limitation worth checking. Microsoft's current OneLake security documentation states that when a OneLake security role is assigned to a B2B guest, the Entra External ID guest-access setting must be Guest users have the same access as members (most inclusive).

If you aren't using OneLake security roles, I wouldn't change that setting just for this test.

4. Compare the customers' Entra cross-tenant policies

Because one external customer's guest works while several users from another external organization fail, I would compare the organization-specific inbound B2B collaboration settings and Conditional Access policies for those two customer tenants.

Microsoft's cross-tenant access documentation confirms that B2B inbound access can be configured separately for individual external organizations, with those organization-specific settings overriding the defaults.

One diagnostic I would also use is to test the failing identity directly against the OneLake DFS API with a Storage-audience Entra token for the host tenant. If that request also fails, I would concentrate on the guest identity, tenant policy and shortcut permissions. If it succeeds while AzureStorage.DataLake() still returns 404, I would then clear the cached OneLake credentials in Power BI Desktop and investigate the connector's authentication context.

For a longer-term multi-customer design, you may also want to evaluate Fabric External Data Sharing. It was designed specifically for cross-tenant OneLake sharing and uses Fabric-to-Fabric authentication rather than requiring the consumer to access the source as a B2B guest. The source data remains in place, so it is still a no-copy pattern.

Given your current evidence though, my first comparison would be:

working guest vs failing guest → external-app tenant setting → shortcut target permissions → Entra cross-tenant policy

The fact that the behaviour follows particular customer identities/tenants is a strong reason to troubleshoot that path before changing the Delta or Power Query implementation.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Re: Need to Migrate SP written in Synapse to Fabric

EduardoCastro — Tue, 15 Sep 2026 18:05:21 GMT

Hello,

There is a challenge in migrating stored procedures from Synapse to Fabric Warehouse, the syntax differences you are seeing are expected because Fabric Warehouse supports a subset of T-SQL compared to Synapse. The best and quickest approach is to use the Fabric Migration Assistant, which is built natively into Fabric.

The reference for T-SQL limitations in Fabric Data Warehouse is the official Microsoft Learn, you can find it here: T-SQL surface area in Fabric Data Warehouse

In my experience, we converted the stored procedures to Notebooks (PySpark/SQL) instead of migrating them 1:1 to Fabric Warehouse stored procedures.

Re: Need to Migrate SP written in Synapse to Fabric

ShivekMaharaj — Tue, 15 Sep 2026 15:20:34 GMT

Hi ,

If these stored procedures are coming from a Synapse dedicated SQL pool, I would not migrate them one-by-one manually as the first approach.

Microsoft now has a Fabric Migration Assistant for Data Warehouse specifically designed for migrations such as Synapse dedicated SQL pool → Fabric Warehouse. It can assess and migrate database metadata including tables, views, functions and stored procedures, and then flag the T-SQL that is not compatible with Fabric.

You can feed it a DACPAC, SQL files or a SQL database project, so a practical approach would be:

Extract the Synapse schema/code into a DACPAC or SQL database project.
Run it through the Migration Assistant.
Let the assistant migrate the compatible objects automatically.
Use the Fix problems step for the stored procedures that contain unsupported or different T-SQL.
Test the converted procedures side-by-side against Synapse before cutting over.

Microsoft's current Synapse dedicated SQL pool migration guidance also recommends assessing and refactoring database code rather than assuming a completely syntax-compatible lift-and-shift.

Fabric Warehouse does support stored procedures, but the current T-SQL surface area is not identical to Synapse/SQL Server, which is why you are seeing syntax differences. Some constructs still need refactoring, while other capabilities have improved recently - for example, MERGE is now generally available and temporary tables are supported.

So I would avoid manually rewriting everything until you have run the procedures through the Migration Assistant and identified exactly which patterns are incompatible.

One important question: are your procedures running in a Synapse dedicated SQL pool or serverless SQL pool? The automated Migration Assistant documentation currently targets dedicated SQL pools, so the recommended path is slightly different for serverless.

If you can share one of the stored procedures that is producing several syntax errors, we could also look at the specific Synapse constructs that need to change in Fabric.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Need to Migrate SP written in Synapse to Fabric

maxravi — Tue, 15 Sep 2026 14:50:23 GMT

Hello,

I need to migrate my stored prodecure SP from synapse to Fabric, can you please help me with best approach. I am seeing a lot of syntax changes. Suggest me some best, quickest and efficient ways to migrate.

Thanks,

Re: Bronze data contracts for REST/API sources — workaround for Get Metadata's connector gap

Damomac — Tue, 15 Sep 2026 13:10:42 GMT

Thanks all. The feedback has largely confirmed my thinking that there isn't currently a native REST equivalent of Get Metadata in Fabric pipelines. My current leaning is to keep a common bronze contract pattern across source types:

SQL → land → obtain source schema → compare to contract

API → land → read Parquet schema → compare to contract

The goal is to keep contract validation source-agnostic once data has landed.

The raw JSON landing suggestion is interesting, particularly from a replay/audit perspective. I'm currently balancing that against the simplicity of landing directly to Parquet and validating the materialised schema.

Re: Claude Code Integrate with Power Bi

v-achippa — Tue, 15 Sep 2026 12:23:21 GMT

Hi Yash8160,

Thank you for reaching out to Microsoft Fabric Community.

For this POC, you can integrate Claude code with power bi using Microsoft’s Power BI Authoring skill for Fabric. This allows Claude code to work with power bi report and semantic model artifacts.

There is no fixed number of Claude code credits or tokens per power bi report or page. The consumption depends on the Claude model used and the complexity of the authoring task. So, build one representative report page, record its token usage, and use it as a baseline to estimate the remaining POC, including some allowance for testing and rework

For reference, follow the below documentation:
Skills for Fabric

Install Skills for Fabric

Thanks and regards,
Anjan Kumar Chippa

Is it possible connect via a guest account to a Delta table in a Lakehouse from another tenant.

FabricEnjoyer — Tue, 15 Sep 2026 11:21:00 GMT

We have the following architecture in Microsoft Fabric:

Open Mirroring Database ↓ Notebook ↓ Materialized Delta Tables (Lakehouse) ↓ Shortcuts into Connection Lakehouse ↓ Power BI Desktop (Azure Data Lake Storage Gen2 connector) ↓ DeltaLake.Table()

Source = AzureStorage.DataLake( "https://onelake.dfs.fabric.microsoft.com/<workspace-id>/<lakehouse-id>/Tables", [HierarchicalNavigation = true] ), dbo = Source{[Name="dbo"]}[Content], TableName = dbo{[Name="<table_name>"]}[Content], Data = DeltaLake.Table(TableName)

What works

Internal users in the hosting tenant can successfully create the connection.
At least one external Entra B2B Guest user can successfully create the connection.
If we create and publish the semantic model ourselves, external customers can copy/use the model successfully.
Existing reports refresh successfully once deployed.

What fails

Several external customer users that have been invited into our tenant as Entra B2B Guests receive a 404 error when attempting to create a new Azure Data Lake Storage Gen2 connection directly from Power BI Desktop.

The users have:

Accepted the guest invitation.
Been granted access to the Fabric workspace(Contributor)
Been granted access to the Lakehouse.
Been added as Guest users in Entra ID.

Observations

The issue is affecting multiple customers, not just a single user.
The same Lakehouse, shortcut tables, and Power Query logic work for other users.
A Guest account from one customer tenant works, while Guest accounts from several other customer tenants receive a 404.
This suggests the issue may be related to cross-tenant authentication, OneLake access resolution, or the ADLS Gen2 connector rather than permissions on the Fabric artifacts themselves.

Questions

Are there known limitations or issues with Azure Data Lake Storage Gen2 connections to OneLake for some Entra B2B Guest users?
Are there specific cross-tenant access settings, trust settings, or Conditional Access configurations that can cause a 404 during OneLake path resolution?
Is there a recommended approach for external customers connecting directly to OneLake Delta tables via the ADLS Gen2 connector?

Any guidance would be appreciated.

Re: Bronze data contracts for REST/API sources — workaround for Get Metadata's connector gap

v-achippa — Tue, 15 Sep 2026 09:57:27 GMT

Hi damo_mac85,

Thank you for reaching out to Microsoft Fabric Community.

Thank you GilbertQ and ShivekMaharaj for the prompt response.

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user's for the issue worked? or let us know if you need any further assistance.

Thanks and regards,
Anjan Kumar Chippa