All Data Engineering posts

Re: Approach for automating Dataflow Gen2 connection binding across Dev/Prod?

lavginqo2 — Tue, 15 Sep 2026 21:52:37 GMT

Thanks — this is useful, particularly the definition APIs. One clarification: source connection rebinding is already solved on our side.

We map the Dev DatasourceId to the Prod one through parameter.yml and it lands correctly. Our open issue is the Lakehouse *destination* binding, which is a different thing. The Lakehouse connection ID is identical across workspaces. We've diffed two tables in the same Dataflow, one that works and one that fails, in the deployed Prod artifact — same workspace and Lakehouse IDs, same navigation, same destination settings (IsNewTarget, AllowCreation, DynamicSchema, Replace). Both target tables exist, and in each case the mapped columns match that table's actual schema exactly, nothing missing or extra. Structurally there's no difference between them. The only thing that makes the failing one work is opening it in the UI and clicking Save. We did take your mechanism and try it slightly differently: a no-change round-trip, getDefinition then updateDefinition with the identical payload, on the theory that re-processing the definition might trigger whatever the UI save triggers. The call returns 200. Results inconclusive so far because the item had already been opened in the editor, which auto-saves a draft, so we can't cleanly attribute the outcome. Retesting on an untouched Dataflow. The part we still can't explain: in one Dataflow roughly 6 of 29 tables bind after deployment and the rest don't; in a second Dataflow none of them do. Definitions are identical throughout.

Thanks !

Re: Approach for automating Dataflow Gen2 connection binding across Dev/Prod?

ShivekMaharaj — Tue, 15 Sep 2026 19:15:28 GMT

Hi lavginqo2,

I think there is one newer API capability worth adding here, because the current situation is slightly more nuanced than "there is no REST way to rebind it."

You are correct that Dataflow Gen2 still does not have a native Dev → Prod connection deployment rule. Microsoft's current Dataflow Gen2 CI/CD architecture guidance explicitly says that data-source connections are statically bound and cannot be switched using workspace variables or Dataflow parameters. Deployment rules don't currently alter Dataflow connections either.

However, Microsoft now exposes the Dataflow's public definition through the Fabric REST API.

The documented Dataflow definition format includes a connections collection inside queryMetadata.json, and each entry contains the connection path, kind, and connectionId.

That opens up a post-deployment automation pattern:

Deploy Dev → Prod → get the Prod Dataflow definition → replace the Dev connection ID with the corresponding Prod connection ID → update the definition → publish the Dataflow

Microsoft provides both Get Dataflow Definition and Update Dataflow Definition endpoints for this.

I would therefore separate environment configuration from the Dataflow deployment itself.

For example:

Create the required shareable connections in Prod once, ideally with stable naming.
Maintain a small environment mapping such as DevConnectionId → ProdConnectionId.
Deploy the Dataflow through your normal Fabric CI/CD process.
Call getDefinition on the deployed Dataflow.
Decode queryMetadata.json and replace the relevant connections[].connectionId values.
Send the modified definition through updateDefinition.
Publish the Dataflow and run a validation refresh.

Fabric also has public connection APIs for creating, listing, testing and managing the actual connection objects, so provisioning the Prod connections can be automated separately.

One important distinction is that the Update Connection API isn't itself a Dataflow-rebinding operation. It updates an existing connection object. The Dataflow-to-connection reference is represented in the Dataflow definition, which is why I would change the connectionId there instead.

For ~47 tables, this is the route I would prototype before manually reconnecting every query or splitting the Dataflow purely to make the manual work manageable. If those queries share a small number of underlying source connections, you are effectively translating a small connection map rather than maintaining 47 independent bindings by hand.

One caveat: Microsoft's API documentation is currently a little inconsistent around unattended identities. The specific Update Dataflow Definition API lists service principals and managed identities as supported, while the broader Dataflow Gen2 public-API documentation still lists service-principal authentication as a limitation. I would therefore validate the full getDefinition → updateDefinition → publish sequence with your intended CI/CD identity before making it the production deployment path.

So I agree that the lack of a first-class connection deployment rule is still a gap, but with the newer public Dataflow definition APIs I don't think manual per-query rebinding necessarily has to be the end state anymore.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Re: Claude Code Integrate with Power Bi

ShivekMaharaj — Tue, 15 Sep 2026 19:04:20 GMT

Hi Yash8160,

v-achippa is right that there isn't a fixed token/credit figure per Power BI page. I would add one billing distinction that is important when estimating the POC.

The Power BI Report Authoring skill is the Power BI-specific instruction/tooling layer used by the coding agent. Microsoft currently lists Claude Code as a compatible agent, but the token consumption comes from the Claude model and the amount of agent work required rather than from Fabric assigning a fixed number of credits to each report or visual.

The cost therefore depends on things such as:

which Claude model you use;
how much of the semantic model/report definition has to be inspected;
number and complexity of pages/visuals;
how many authoring and validation iterations are required;
how much existing PBIR/PBIP context is repeatedly read.

I would also first establish how Claude Code will be billed. Anthropic documents that Claude Code can be used through a Pro/Max subscription allocation or through pay-as-you-go API/Console usage. Those are different models: subscription usage is governed by the plan's usage limits, while API-credit usage is charged at the model's token rates.

For a POC, I would therefore benchmark one representative report workflow rather than trying to estimate from page count alone.

For example:

Pick one page that is representative of the final POC - not the simplest page.
Run the complete workflow you expect to use: inspect the semantic model → plan/design the page → author the PBIR → open/validate it → make any corrections.
Record the actual Claude usage/cost for that complete iteration.
Repeat with one simple page and one complex page if the report varies significantly.
Use those measurements to estimate the remaining pages and add some allowance for rework and prompt iteration.

Microsoft's current Power BI agentic authoring guidance also separates report planning, design, authoring and management into different skills, so I would include whichever of those steps you intend to use when measuring the baseline.

If you are using Anthropic API/pay-as-you-go rather than staying within a Claude subscription, you can then convert the measured model usage using Anthropic's current token pricing. If you are using Pro/Max, I would monitor the remaining Claude Code allocation instead of treating every page as a directly billed token cost.

So I would avoid quoting a generic "X credits per report" number. One or two representative end-to-end builds using your actual semantic model will give you a much more defensible POC estimate.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Re: Is it possible connect via a guest account to a Delta table in a Lakehouse from another tenant.

ShivekMaharaj — Tue, 15 Sep 2026 18:58:55 GMT

Hi FabricEnjoyer,

I don't think the Delta table itself would be my first suspect here.

Microsoft explicitly supports accessing OneLake through existing ADLS/Blob APIs using the onelake.dfs.fabric.microsoft.com endpoint, as described in the OneLake connection guidance. Also, because at least one B2B guest can create the same connection successfully, I wouldn't describe guest access through this path as universally unsupported.

The fact that several guests from one customer fail with a 404 while another customer's guest works makes me look more closely at the identity and authorization path.

I would check these areas in this order:

1. Fabric/OneLake tenant settings

Verify that Guest users can access Microsoft Fabric is enabled for the affected guests.

I would also check the separate OneLake setting Users can access data stored in OneLake with apps external to Fabric. Microsoft's OneLake tenant-settings documentation describes this setting as controlling access from applications using ADLS APIs outside the Fabric environment.

This is especially worth checking if either setting is scoped to selected security groups rather than enabled tenant-wide.

2. The permissions on the shortcut target

This could be easy to miss in your architecture.

If the shortcuts from the Connection Lakehouse to the materialized Lakehouse are using passthrough authentication, OneLake evaluates the calling user's identity against the target as well. Microsoft's shortcut security documentation explains that passthrough OneLake-to-OneLake shortcuts require the user to be authorized on both the shortcut path and the target path.

So I would compare a working guest and a failing guest not only on the Connection Lakehouse, but also on the Lakehouse containing the materialized Delta tables.

If the shortcut uses delegated authentication instead, then this particular requirement changes because the configured connection identity is used for the target.

3. OneLake security, if you are using it

If any of these permissions are being supplied through OneLake security roles, there is a specific B2B limitation worth checking. Microsoft's current OneLake security documentation states that when a OneLake security role is assigned to a B2B guest, the Entra External ID guest-access setting must be Guest users have the same access as members (most inclusive).

If you aren't using OneLake security roles, I wouldn't change that setting just for this test.

4. Compare the customers' Entra cross-tenant policies

Because one external customer's guest works while several users from another external organization fail, I would compare the organization-specific inbound B2B collaboration settings and Conditional Access policies for those two customer tenants.

Microsoft's cross-tenant access documentation confirms that B2B inbound access can be configured separately for individual external organizations, with those organization-specific settings overriding the defaults.

One diagnostic I would also use is to test the failing identity directly against the OneLake DFS API with a Storage-audience Entra token for the host tenant. If that request also fails, I would concentrate on the guest identity, tenant policy and shortcut permissions. If it succeeds while AzureStorage.DataLake() still returns 404, I would then clear the cached OneLake credentials in Power BI Desktop and investigate the connector's authentication context.

For a longer-term multi-customer design, you may also want to evaluate Fabric External Data Sharing. It was designed specifically for cross-tenant OneLake sharing and uses Fabric-to-Fabric authentication rather than requiring the consumer to access the source as a B2B guest. The source data remains in place, so it is still a no-copy pattern.

Given your current evidence though, my first comparison would be:

working guest vs failing guest → external-app tenant setting → shortcut target permissions → Entra cross-tenant policy

The fact that the behaviour follows particular customer identities/tenants is a strong reason to troubleshoot that path before changing the Delta or Power Query implementation.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Re: Need to Migrate SP written in Synapse to Fabric

EduardoCastro — Tue, 15 Sep 2026 18:05:21 GMT

Hello,

There is a challenge in migrating stored procedures from Synapse to Fabric Warehouse, the syntax differences you are seeing are expected because Fabric Warehouse supports a subset of T-SQL compared to Synapse. The best and quickest approach is to use the Fabric Migration Assistant, which is built natively into Fabric.

The reference for T-SQL limitations in Fabric Data Warehouse is the official Microsoft Learn, you can find it here: T-SQL surface area in Fabric Data Warehouse

In my experience, we converted the stored procedures to Notebooks (PySpark/SQL) instead of migrating them 1:1 to Fabric Warehouse stored procedures.

Re: Need to Migrate SP written in Synapse to Fabric

ShivekMaharaj — Tue, 15 Sep 2026 15:20:34 GMT

Hi ,

If these stored procedures are coming from a Synapse dedicated SQL pool, I would not migrate them one-by-one manually as the first approach.

Microsoft now has a Fabric Migration Assistant for Data Warehouse specifically designed for migrations such as Synapse dedicated SQL pool → Fabric Warehouse. It can assess and migrate database metadata including tables, views, functions and stored procedures, and then flag the T-SQL that is not compatible with Fabric.

You can feed it a DACPAC, SQL files or a SQL database project, so a practical approach would be:

Extract the Synapse schema/code into a DACPAC or SQL database project.
Run it through the Migration Assistant.
Let the assistant migrate the compatible objects automatically.
Use the Fix problems step for the stored procedures that contain unsupported or different T-SQL.
Test the converted procedures side-by-side against Synapse before cutting over.

Microsoft's current Synapse dedicated SQL pool migration guidance also recommends assessing and refactoring database code rather than assuming a completely syntax-compatible lift-and-shift.

Fabric Warehouse does support stored procedures, but the current T-SQL surface area is not identical to Synapse/SQL Server, which is why you are seeing syntax differences. Some constructs still need refactoring, while other capabilities have improved recently - for example, MERGE is now generally available and temporary tables are supported.

So I would avoid manually rewriting everything until you have run the procedures through the Migration Assistant and identified exactly which patterns are incompatible.

One important question: are your procedures running in a Synapse dedicated SQL pool or serverless SQL pool? The automated Migration Assistant documentation currently targets dedicated SQL pools, so the recommended path is slightly different for serverless.

If you can share one of the stored procedures that is producing several syntax errors, we could also look at the specific Synapse constructs that need to change in Fabric.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Need to Migrate SP written in Synapse to Fabric

maxravi — Tue, 15 Sep 2026 14:50:23 GMT

Hello,

I need to migrate my stored prodecure SP from synapse to Fabric, can you please help me with best approach. I am seeing a lot of syntax changes. Suggest me some best, quickest and efficient ways to migrate.

Thanks,

Re: Bronze data contracts for REST/API sources — workaround for Get Metadata's connector gap

Damomac — Tue, 15 Sep 2026 13:10:42 GMT

Thanks all. The feedback has largely confirmed my thinking that there isn't currently a native REST equivalent of Get Metadata in Fabric pipelines. My current leaning is to keep a common bronze contract pattern across source types:

SQL → land → obtain source schema → compare to contract

API → land → read Parquet schema → compare to contract

The goal is to keep contract validation source-agnostic once data has landed.

The raw JSON landing suggestion is interesting, particularly from a replay/audit perspective. I'm currently balancing that against the simplicity of landing directly to Parquet and validating the materialised schema.

Re: Claude Code Integrate with Power Bi

v-achippa — Tue, 15 Sep 2026 12:23:21 GMT

Hi Yash8160,

Thank you for reaching out to Microsoft Fabric Community.

For this POC, you can integrate Claude code with power bi using Microsoft’s Power BI Authoring skill for Fabric. This allows Claude code to work with power bi report and semantic model artifacts.

There is no fixed number of Claude code credits or tokens per power bi report or page. The consumption depends on the Claude model used and the complexity of the authoring task. So, build one representative report page, record its token usage, and use it as a baseline to estimate the remaining POC, including some allowance for testing and rework

For reference, follow the below documentation:
Skills for Fabric

Install Skills for Fabric

Thanks and regards,
Anjan Kumar Chippa

Is it possible connect via a guest account to a Delta table in a Lakehouse from another tenant.

FabricEnjoyer — Tue, 15 Sep 2026 11:21:00 GMT

We have the following architecture in Microsoft Fabric:

Open Mirroring Database ↓ Notebook ↓ Materialized Delta Tables (Lakehouse) ↓ Shortcuts into Connection Lakehouse ↓ Power BI Desktop (Azure Data Lake Storage Gen2 connector) ↓ DeltaLake.Table()

Source = AzureStorage.DataLake( "https://onelake.dfs.fabric.microsoft.com/<workspace-id>/<lakehouse-id>/Tables", [HierarchicalNavigation = true] ), dbo = Source{[Name="dbo"]}[Content], TableName = dbo{[Name="<table_name>"]}[Content], Data = DeltaLake.Table(TableName)

What works

Internal users in the hosting tenant can successfully create the connection.
At least one external Entra B2B Guest user can successfully create the connection.
If we create and publish the semantic model ourselves, external customers can copy/use the model successfully.
Existing reports refresh successfully once deployed.

What fails

Several external customer users that have been invited into our tenant as Entra B2B Guests receive a 404 error when attempting to create a new Azure Data Lake Storage Gen2 connection directly from Power BI Desktop.

The users have:

Accepted the guest invitation.
Been granted access to the Fabric workspace(Contributor)
Been granted access to the Lakehouse.
Been added as Guest users in Entra ID.

Observations

The issue is affecting multiple customers, not just a single user.
The same Lakehouse, shortcut tables, and Power Query logic work for other users.
A Guest account from one customer tenant works, while Guest accounts from several other customer tenants receive a 404.
This suggests the issue may be related to cross-tenant authentication, OneLake access resolution, or the ADLS Gen2 connector rather than permissions on the Fabric artifacts themselves.

Questions

Are there known limitations or issues with Azure Data Lake Storage Gen2 connections to OneLake for some Entra B2B Guest users?
Are there specific cross-tenant access settings, trust settings, or Conditional Access configurations that can cause a 404 during OneLake path resolution?
Is there a recommended approach for external customers connecting directly to OneLake Delta tables via the ADLS Gen2 connector?

Any guidance would be appreciated.

Re: Bronze data contracts for REST/API sources — workaround for Get Metadata's connector gap

v-achippa — Tue, 15 Sep 2026 09:57:27 GMT

Hi damo_mac85,

Thank you for reaching out to Microsoft Fabric Community.

Thank you GilbertQ and ShivekMaharaj for the prompt response.

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user's for the issue worked? or let us know if you need any further assistance.

Thanks and regards,
Anjan Kumar Chippa

Re: Pure Python Notebook: authenticate to Azure DevOps Artifacts using Fabric Workspace Identity

Kagiyama_yutaka — Tue, 15 Sep 2026 09:55:46 GMT

Python notebook in Fabric has no step that turns its Workspace Identity into an Azure DevOps / Azure Artifacts token for pip install --index-url, and the simple path is reading a PAT from Azure Key Vault and giving that value to pip.

The token API only gives audiences like storage, pbi, keyvault, and kusto, and Python notebooks do not take part in the Environment flow that Spark uses for Azure Artifacts. The Python kernel has no credential provider capable of producing an Azure DevOps audience token. This is a structural limitation of the current Python kernel architecture, not a missing feature in the documentation.　

Claude Code Integrate with Power Bi

Yash8160 — Tue, 15 Sep 2026 09:54:54 GMT

Hi everyone,

I’m working on a POC for Power BI + Claude Integration, specifically the Report Authoring skill.We need to use Claude Code to create the required report pages and visuals. I’m currently trying to estimate the Claude Code credit/token consumption and approximate cost for completing this POC.

Has anyone worked on a similar POC using Claude Code? If so, could you please share:

Approximate credits/tokens consumed

Estimated cost

Any recommendations for estimating the usage before starting

Any guidance or experience would be really helpful. Thanks!

Re: FTL4 Trial Spark failing - InvalidRequestClusterFromFabricDenyList - Cluster Cancelled before Ready

v-achippa — Tue, 15 Sep 2026 09:44:26 GMT

Hi pritwade,

As we haven’t heard back from you, we wanted to kindly follow up to check if your issue is resolved? or have you raised a support ticket?

Thanks and regards,
Anjan Kumar Chippa

Re: Background CU used in Fabric Capacity Metrics

Kagiyama_yutaka — Tue, 15 Sep 2026 06:14:05 GMT

If you don’t mind, could you add a little more detail about how your pipelines and notebooks are running? For example, it helps to know roughly how many activities are running at the same time, and whether your notebooks are using autoscale or fixed Spark settings. This makes it easier to understand whether the background CU is coming from parallel pipeline runs, Spark sessions that stayed alive, or carry‑forward from previous workloads.

Re: Approach for automating Dataflow Gen2 connection binding across Dev/Prod?

Kagiyama_yutaka — Tue, 15 Sep 2026 05:00:02 GMT

Dataflow Gen2 keeps the connection tied to the workspace, and CI/CD doesn’t move it, and there’s no REST or SPN way to rebind it after a deploy.

Just make the same connection in Prod ahead of time, push the flow, open each query and pick that Prod connection yourself. For bigger flows people usually split the dataflow so the reconnect work stays manageable.　

Re: Background CU used in Fabric Capacity Metrics

v-saisrao-msft — Tue, 15 Sep 2026 04:27:51 GMT

HI dinhhoa,

Checking in to see if your issue has been resolved. let us know if you still need any assistance.

Thank you.

Approach for automating Dataflow Gen2 connection binding across Dev/Prod?

lavginqo2 — Mon, 14 Sep 2026 19:53:22 GMT

We're deploying Dataflow Gen2 from Dev to Prod via fabric-cicd with all destination Lakehouse IDs and source parameters fully parameterized. Definitions deploy cleanly.

The blocker: every query's data source connection has to be manually reconfigured in the target workspace — "Configure connection" doesn't auto-bind to an existing connection object even when one already exists for that connector, and this has to be repeated per query/table, not once per Dataflow. Docs confirm this is expected ("connections are statically bound... can't be altered using workspace variables or parameterization"), but that leaves genuine automation impossible for Dataflows with many tables (~47 in our case).

And it is even not possible to update connection. It seems that each table has to be created fresh in Prod.

Has anyone found a real workaround — Service Principal-based automation, REST API call to bind connections post-deploy etc.

What's actually working for people running CI/CD from Dev to Prod at scale here?

Thanks

Re: Pure Python Notebook: authenticate to Azure DevOps Artifacts using Fabric Workspace Identity

ShivekMaharaj — Mon, 14 Sep 2026 19:42:36 GMT

Hi llueg,

I think you have found a real gap between the current pure Python notebook experience and the newer Workspace Identity support for Azure Artifacts.

As far as the current documentation goes, I don't see a supported way for a pure Python notebook to directly request an Azure DevOps/Azure Artifacts access token from its Workspace Identity and pass that token to pip.

The reason is that NotebookUtils credential token acquisition currently documents only storage, pbi, keyvault and kusto as supported token audiences. Azure DevOps isn't exposed as a documented audience, so I wouldn't rely on passing an Azure DevOps resource ID to getToken() as an undocumented workaround.

There is, however, a newer PAT-less path that gets very close to what you want. Fabric now supports authenticating an Azure Artifact Feed connection with the Workspace Identity. The Workspace Identity can be added to the Azure DevOps feed with at least Feed Reader, and Fabric then uses that authenticated connection when resolving packages in a Fabric Environment.

Microsoft documents that flow under library management for Fabric Environments.

The limitation for your scenario is that the current pure Python notebook documentation explicitly lists Environment integration as unavailable for Python notebooks. That Workspace Identity → Azure Artifacts flow therefore currently helps Spark notebooks using Fabric Environments, but it doesn't give a pure Python notebook a token that can be handed directly to pip install --index-url.

There is also the newer Fabric Connection integration for notebooks, which supports Workspace Identity authentication and is currently in Preview, but I haven't found Microsoft documentation showing an Azure Artifact Feed connection credential being exposed for direct pip authentication in a pure Python notebook.

So at the moment I would describe it as:

Spark notebook + Fabric Environment: Workspace Identity → Azure Artifacts is supported.
Pure Python notebook + direct pip install: I don't see a documented Workspace Identity-based Azure DevOps token flow yet.

For pure Python, your Key Vault-backed PAT remains the documented practical option unless Microsoft extends the Environment/Azure Artifact Feed integration to the Python kernel or exposes Azure DevOps as a supported notebook credential audience.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Re: New Browser Tab Opening to Previous or Existing Session

KentKMishra — Mon, 14 Sep 2026 18:16:31 GMT

This seems to be a "feature" rolled out over the summer and it drives me crazy. We actually need different tabs within the same workspace (dealing with items from different source systems) and the number of tabs takes up too much real estate and can cause notebook connections to go weird. Then I'm jumping through 10 tabs to see where the problem is. Has slowed down me and my teams' productivity.