All Data Engineering posts

Re: Need to Migrate SP written in Synapse to Fabric

ShivekMaharaj — Tue, 15 Sep 2026 15:20:34 GMT

Hi ,

If these stored procedures are coming from a Synapse dedicated SQL pool, I would not migrate them one-by-one manually as the first approach.

Microsoft now has a Fabric Migration Assistant for Data Warehouse specifically designed for migrations such as Synapse dedicated SQL pool → Fabric Warehouse. It can assess and migrate database metadata including tables, views, functions and stored procedures, and then flag the T-SQL that is not compatible with Fabric.

You can feed it a DACPAC, SQL files or a SQL database project, so a practical approach would be:

Extract the Synapse schema/code into a DACPAC or SQL database project.
Run it through the Migration Assistant.
Let the assistant migrate the compatible objects automatically.
Use the Fix problems step for the stored procedures that contain unsupported or different T-SQL.
Test the converted procedures side-by-side against Synapse before cutting over.

Microsoft's current Synapse dedicated SQL pool migration guidance also recommends assessing and refactoring database code rather than assuming a completely syntax-compatible lift-and-shift.

Fabric Warehouse does support stored procedures, but the current T-SQL surface area is not identical to Synapse/SQL Server, which is why you are seeing syntax differences. Some constructs still need refactoring, while other capabilities have improved recently - for example, MERGE is now generally available and temporary tables are supported.

So I would avoid manually rewriting everything until you have run the procedures through the Migration Assistant and identified exactly which patterns are incompatible.

One important question: are your procedures running in a Synapse dedicated SQL pool or serverless SQL pool? The automated Migration Assistant documentation currently targets dedicated SQL pools, so the recommended path is slightly different for serverless.

If you can share one of the stored procedures that is producing several syntax errors, we could also look at the specific Synapse constructs that need to change in Fabric.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Need to Migrate SP written in Synapse to Fabric

maxravi — Tue, 15 Sep 2026 14:50:23 GMT

Hello,

I need to migrate my stored prodecure SP from synapse to Fabric, can you please help me with best approach. I am seeing a lot of syntax changes. Suggest me some best, quickest and efficient ways to migrate.

Thanks,

Re: Bronze data contracts for REST/API sources — workaround for Get Metadata's connector gap

Damomac — Tue, 15 Sep 2026 13:10:42 GMT

Thanks all. The feedback has largely confirmed my thinking that there isn't currently a native REST equivalent of Get Metadata in Fabric pipelines. My current leaning is to keep a common bronze contract pattern across source types:

SQL → land → obtain source schema → compare to contract

API → land → read Parquet schema → compare to contract

The goal is to keep contract validation source-agnostic once data has landed.

The raw JSON landing suggestion is interesting, particularly from a replay/audit perspective. I'm currently balancing that against the simplicity of landing directly to Parquet and validating the materialised schema.

Re: Claude Code Integrate with Power Bi

v-achippa — Tue, 15 Sep 2026 12:23:21 GMT

Hi Yash8160,

Thank you for reaching out to Microsoft Fabric Community.

For this POC, you can integrate Claude code with power bi using Microsoft’s Power BI Authoring skill for Fabric. This allows Claude code to work with power bi report and semantic model artifacts.

There is no fixed number of Claude code credits or tokens per power bi report or page. The consumption depends on the Claude model used and the complexity of the authoring task. So, build one representative report page, record its token usage, and use it as a baseline to estimate the remaining POC, including some allowance for testing and rework

For reference, follow the below documentation:
Skills for Fabric

Install Skills for Fabric

Thanks and regards,
Anjan Kumar Chippa

Is it possible connect via a guest account to a Delta table in a Lakehouse from another tenant.

FabricEnjoyer — Tue, 15 Sep 2026 11:21:00 GMT

We have the following architecture in Microsoft Fabric:

Open Mirroring Database ↓ Notebook ↓ Materialized Delta Tables (Lakehouse) ↓ Shortcuts into Connection Lakehouse ↓ Power BI Desktop (Azure Data Lake Storage Gen2 connector) ↓ DeltaLake.Table()

Source = AzureStorage.DataLake( "https://onelake.dfs.fabric.microsoft.com/<workspace-id>/<lakehouse-id>/Tables", [HierarchicalNavigation = true] ), dbo = Source{[Name="dbo"]}[Content], TableName = dbo{[Name="<table_name>"]}[Content], Data = DeltaLake.Table(TableName)

What works

Internal users in the hosting tenant can successfully create the connection.
At least one external Entra B2B Guest user can successfully create the connection.
If we create and publish the semantic model ourselves, external customers can copy/use the model successfully.
Existing reports refresh successfully once deployed.

What fails

Several external customer users that have been invited into our tenant as Entra B2B Guests receive a 404 error when attempting to create a new Azure Data Lake Storage Gen2 connection directly from Power BI Desktop.

The users have:

Accepted the guest invitation.
Been granted access to the Fabric workspace(Contributor)
Been granted access to the Lakehouse.
Been added as Guest users in Entra ID.

Observations

The issue is affecting multiple customers, not just a single user.
The same Lakehouse, shortcut tables, and Power Query logic work for other users.
A Guest account from one customer tenant works, while Guest accounts from several other customer tenants receive a 404.
This suggests the issue may be related to cross-tenant authentication, OneLake access resolution, or the ADLS Gen2 connector rather than permissions on the Fabric artifacts themselves.

Questions

Are there known limitations or issues with Azure Data Lake Storage Gen2 connections to OneLake for some Entra B2B Guest users?
Are there specific cross-tenant access settings, trust settings, or Conditional Access configurations that can cause a 404 during OneLake path resolution?
Is there a recommended approach for external customers connecting directly to OneLake Delta tables via the ADLS Gen2 connector?

Any guidance would be appreciated.

Re: Bronze data contracts for REST/API sources — workaround for Get Metadata's connector gap

v-achippa — Tue, 15 Sep 2026 09:57:27 GMT

Hi damo_mac85,

Thank you for reaching out to Microsoft Fabric Community.

Thank you GilbertQ and ShivekMaharaj for the prompt response.

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user's for the issue worked? or let us know if you need any further assistance.

Thanks and regards,
Anjan Kumar Chippa

Re: Pure Python Notebook: authenticate to Azure DevOps Artifacts using Fabric Workspace Identity

Kagiyama_yutaka — Tue, 15 Sep 2026 09:55:46 GMT

Python notebook in Fabric has no step that turns its Workspace Identity into an Azure DevOps / Azure Artifacts token for pip install --index-url, and the simple path is reading a PAT from Azure Key Vault and giving that value to pip.

The token API only gives audiences like storage, pbi, keyvault, and kusto, and Python notebooks do not take part in the Environment flow that Spark uses for Azure Artifacts. The Python kernel has no credential provider capable of producing an Azure DevOps audience token. This is a structural limitation of the current Python kernel architecture, not a missing feature in the documentation.　

Claude Code Integrate with Power Bi

Yash8160 — Tue, 15 Sep 2026 09:54:54 GMT

Hi everyone,

I’m working on a POC for Power BI + Claude Integration, specifically the Report Authoring skill.We need to use Claude Code to create the required report pages and visuals. I’m currently trying to estimate the Claude Code credit/token consumption and approximate cost for completing this POC.

Has anyone worked on a similar POC using Claude Code? If so, could you please share:

Approximate credits/tokens consumed

Estimated cost

Any recommendations for estimating the usage before starting

Any guidance or experience would be really helpful. Thanks!

Re: FTL4 Trial Spark failing - InvalidRequestClusterFromFabricDenyList - Cluster Cancelled before Ready

v-achippa — Tue, 15 Sep 2026 09:44:26 GMT

Hi pritwade,

As we haven’t heard back from you, we wanted to kindly follow up to check if your issue is resolved? or have you raised a support ticket?

Thanks and regards,
Anjan Kumar Chippa

Re: Background CU used in Fabric Capacity Metrics

Kagiyama_yutaka — Tue, 15 Sep 2026 06:14:05 GMT

If you don’t mind, could you add a little more detail about how your pipelines and notebooks are running? For example, it helps to know roughly how many activities are running at the same time, and whether your notebooks are using autoscale or fixed Spark settings. This makes it easier to understand whether the background CU is coming from parallel pipeline runs, Spark sessions that stayed alive, or carry‑forward from previous workloads.

Re: Approach for automating Dataflow Gen2 connection binding across Dev/Prod?

Kagiyama_yutaka — Tue, 15 Sep 2026 05:00:02 GMT

Dataflow Gen2 keeps the connection tied to the workspace, and CI/CD doesn’t move it, and there’s no REST or SPN way to rebind it after a deploy.

Just make the same connection in Prod ahead of time, push the flow, open each query and pick that Prod connection yourself. For bigger flows people usually split the dataflow so the reconnect work stays manageable.　

Re: Background CU used in Fabric Capacity Metrics

v-saisrao-msft — Tue, 15 Sep 2026 04:27:51 GMT

HI dinhhoa,

Checking in to see if your issue has been resolved. let us know if you still need any assistance.

Thank you.

Approach for automating Dataflow Gen2 connection binding across Dev/Prod?

lavginqo2 — Mon, 14 Sep 2026 19:53:22 GMT

We're deploying Dataflow Gen2 from Dev to Prod via fabric-cicd with all destination Lakehouse IDs and source parameters fully parameterized. Definitions deploy cleanly.

The blocker: every query's data source connection has to be manually reconfigured in the target workspace — "Configure connection" doesn't auto-bind to an existing connection object even when one already exists for that connector, and this has to be repeated per query/table, not once per Dataflow. Docs confirm this is expected ("connections are statically bound... can't be altered using workspace variables or parameterization"), but that leaves genuine automation impossible for Dataflows with many tables (~47 in our case).

And it is even not possible to update connection. It seems that each table has to be created fresh in Prod.

Has anyone found a real workaround — Service Principal-based automation, REST API call to bind connections post-deploy etc.

What's actually working for people running CI/CD from Dev to Prod at scale here?

Thanks

Re: Pure Python Notebook: authenticate to Azure DevOps Artifacts using Fabric Workspace Identity

ShivekMaharaj — Mon, 14 Sep 2026 19:42:36 GMT

Hi llueg,

I think you have found a real gap between the current pure Python notebook experience and the newer Workspace Identity support for Azure Artifacts.

As far as the current documentation goes, I don't see a supported way for a pure Python notebook to directly request an Azure DevOps/Azure Artifacts access token from its Workspace Identity and pass that token to pip.

The reason is that NotebookUtils credential token acquisition currently documents only storage, pbi, keyvault and kusto as supported token audiences. Azure DevOps isn't exposed as a documented audience, so I wouldn't rely on passing an Azure DevOps resource ID to getToken() as an undocumented workaround.

There is, however, a newer PAT-less path that gets very close to what you want. Fabric now supports authenticating an Azure Artifact Feed connection with the Workspace Identity. The Workspace Identity can be added to the Azure DevOps feed with at least Feed Reader, and Fabric then uses that authenticated connection when resolving packages in a Fabric Environment.

Microsoft documents that flow under library management for Fabric Environments.

The limitation for your scenario is that the current pure Python notebook documentation explicitly lists Environment integration as unavailable for Python notebooks. That Workspace Identity → Azure Artifacts flow therefore currently helps Spark notebooks using Fabric Environments, but it doesn't give a pure Python notebook a token that can be handed directly to pip install --index-url.

There is also the newer Fabric Connection integration for notebooks, which supports Workspace Identity authentication and is currently in Preview, but I haven't found Microsoft documentation showing an Azure Artifact Feed connection credential being exposed for direct pip authentication in a pure Python notebook.

So at the moment I would describe it as:

Spark notebook + Fabric Environment: Workspace Identity → Azure Artifacts is supported.
Pure Python notebook + direct pip install: I don't see a documented Workspace Identity-based Azure DevOps token flow yet.

For pure Python, your Key Vault-backed PAT remains the documented practical option unless Microsoft extends the Environment/Azure Artifact Feed integration to the Python kernel or exposes Azure DevOps as a supported notebook credential audience.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Re: New Browser Tab Opening to Previous or Existing Session

KentKMishra — Mon, 14 Sep 2026 18:16:31 GMT

This seems to be a "feature" rolled out over the summer and it drives me crazy. We actually need different tabs within the same workspace (dealing with items from different source systems) and the number of tabs takes up too much real estate and can cause notebook connections to go weird. Then I'm jumping through 10 tabs to see where the problem is. Has slowed down me and my teams' productivity.

Re: GraphQL update... mutation for Fabric SQL DB failing

msturzen — Mon, 14 Sep 2026 13:47:20 GMT

Hi Kagiyama_yutaka,
the update mutation requires the pk column as the query's first argument so it can identify the row to update. The PK column itself is not updated in the update snippet I posted initially.
I only send the writable column or am I misreading something?

Best
Martin

Re: Is a table ordinary, or special (= from a shortcut, a system table or materialized lake view table)?

Ben-Dev — Mon, 14 Sep 2026 13:43:14 GMT

Thank you, v-aatheeque. Your response is helpful. Hopefully, an easier--and unified--way to get these details will be added in the future. Thanks, again, for your help!

Re: GraphQL update... mutation for Fabric SQL DB failing

msturzen — Mon, 14 Sep 2026 13:38:30 GMT

Sorry for being unresponsive for the last 2 weeks. Had to take a few days off.
I raised a support ticket that is currently being worked on. I'll post updates asap.
Best
Martin

From On-Premises SQL Server to Microsoft Fabric with a Metadata-Driven Medallion Pipeline

ezalor — Mon, 14 Sep 2026 10:48:56 GMT

In this demo, I use DataCoolie to move WideWorldImporters data from an on-premises SQL Server into Microsoft Fabric. The pipeline is driven by one metadata.json file and follows the Bronze, Silver, and Gold pattern.

The focus here is the Fabric execution flow, not the earlier development process.

The complete sample is available in the datacoolie/dc-demo-fabric repository.

Architecture

The SQL Server remains on-premises, so ingestion runs from an external Polars runner. Downstream processing runs inside Fabric:

Stage	Runtime	Output
source2bronze	On-premises Polars	Parquet in Files/bronze
bronze2silver	Fabric Python 3.12 notebook with Polars and delta-rs	Delta tables in silver
silver2gold	Fabric Spark notebook	Delta tables in gold

The workspace uses two Lakehouses:

lh_wwi_medallion_demo
├── Files/bronze
├── Tables/silver
└── Tables/gold

lh_etl
└── Files/control/test-fabric
├── metadata/current/metadata.json
├── logs
└── watermarks

This keeps operational state separate from business data while allowing every runtime to use the same metadata and watermarks.

One metadata file, three stages

The final metadata contains four connections, 68 dataflows, and shared schema hints. Each dataflow declares its stage, source, destination, load type, watermark, merge keys, or SQL query. The runner remains generic: it reads this file and receives only the stage name it should execute.

The Fabric environment maps logical connections to:

bronze_parquet → Files/bronze
silver_delta → Tables/silver
gold_delta → Tables/gold

Note: An AI agent will assist in generating the metadata.json file.

Uploading the artifacts with Fabric CLI

After creating the two Lakehouses, install the Fabric CLI, authenticate, upload the metadata, and import the two notebooks:

pip install --upgrade ms-fabric-cli fab auth login $workspace = 'wwi_medallion_demo_test_fabric.Workspace' $etl = "$workspace/lh_etl.Lakehouse" fab mkdir "$etl/Files/control/test-fabric/metadata/current" fab cp '.\metadata.json' ` "$etl/Files/control/test-fabric/metadata/current/metadata.json" -f fab import "$workspace/run_fabric_polars.Notebook" ` -i '.\runners\run_fabric_polars.ipynb' --format ipynb -f fab import "$workspace/run_fabric_spark.Notebook" ` -i '.\runners\run_fabric_spark.ipynb' --format ipynb -f

Stage 1: SQL Server to Bronze

The on-premises runner reads 31 business tables and appends raw Parquet batches to OneLake. Every table has a watermark: ValidFrom, LastEditedWhen, or VehicleTemperatureID. Timestamp-based flows reread a one-day overlap to reduce the chance of missing late records.

Bronze files are organized by ingestion date:

Files/bronze/<schema>__<table>/YYYY/MM/DD/

This stage still runs on-premises because it requires SQL Server connectivity:

$control = 'abfss://wwi_medallion_demo_test_fabric@onelake.dfs.fabric.microsoft.com/lh_etl.Lakehouse/Files/control/test-fabric' python '.\runners\run_fabric_polars_azure_sdk.py' ` --metadata-path "$control/metadata/current/metadata.json" ` --watermark-base-path "$control/watermarks" ` --base-log-path "$control/logs" ` --stage source2bronze --max-workers 4

Stage 2: Bronze to Silver

A Fabric Python notebook runs this stage with Polars and delta-rs, without Spark. It reads new Bronze files through __file_modification_time, removes duplicate source keys, and performs a Delta merge_upsert for all 31 tables.

The result is a current-state Silver replica while Bronze keeps the raw append history.

After Bronze is ready, start the notebook with Fabric CLI. The Lakehouse configuration binds the notebook to lh_wwi_medallion_demo:

$lakehouseConfig = '{"defaultLakehouse":{"name":"lh_wwi_medallion_demo","id":"<lakehouse-id>","workspaceId":"<workspace-id>"}}' fab job run "$workspace/run_fabric_polars.Notebook" ` -P "STAGE:string=bronze2silver,_inlineInstallationEnabled:bool=true" ` -C $lakehouseConfig --timeout 3600

Stage 3: Silver to Gold

The final stage runs in a Fabric Spark notebook. SQL queries stored in metadata read two-level names such as silver.Sales__InvoiceLines and create six Gold products:

DimDate
DimGeography
DimProduct
DimPerson
DimDeliveryMethod
FactInvoiceLineProfitability

The source instance had no customer data, so customer-dependent Gold models were intentionally excluded.

Run Gold only after the Silver notebook succeeds:

fab job run "$workspace/run_fabric_spark.Notebook" ` -P "STAGE:string=silver2gold,_inlineInstallationEnabled:bool=true" ` -C $lakehouseConfig --timeout 3600

Monitoring the complete flow

Fabric Monitoring shows notebook status, duration, and execution details. DataCoolie writes dataflow-level logs and watermarks to lh_etl, allowing DataCoolie Studio to display metadata, assets, lineage, jobs, and individual dataflow runs across all three stages.

Result

The completed pipeline has 31 incremental Bronze flows, 31 key-merged Silver tables, and six Gold products. The runner stays generic; metadata defines the tables, watermarks, merge keys, paths, and SQL transformations.

This separation makes the pipeline easier to extend and keeps the hybrid on-premises/Fabric workflow observable as one system.

Re: DataFlow Gen2 Issue

EduardoCastro — Sat, 12 Sep 2026 22:43:37 GMT

You are likely correct that this is a firewall issue. The error The underlying connection was closed: An unexpected error occurred on a send typically indicates that the request from the Dataflow service is being blocked before it can complete the handshake with the Fabric Lakehouse API.

Because you can access the Lakehouse from Notebooks (which run directly within the Fabric environment) but cannot from Dataflow Gen2 (which may route metadata or data traffic differently via the on-premises data gateway or specific outbound IP ranges), it strongly suggests a one-way firewall rule is currently in place.

To resolve this, the firewall configuration must be changed to allow two-way traffic. It is not enough to simply allow outbound traffic from your network; the connection requires a request-and-response cycle. The firewall must explicitly permit the return traffic from the Power BI/Fabric API endpoints back to the Dataflow service or the data gateway.

All Data Engineering posts

Re: Need to Migrate SP written in Synapse to Fabric

Need to Migrate SP written in Synapse to Fabric

Re: Bronze data contracts for REST/API sources — workaround for Get Metadata's connector gap

Re: Claude Code Integrate with Power Bi

Is it possible connect via a guest account to a Delta table in a Lakehouse from another tenant.

What works

What fails

Observations

Questions

Re: Bronze data contracts for REST/API sources — workaround for Get Metadata's connector gap

Re: Pure Python Notebook: authenticate to Azure DevOps Artifacts using Fabric Workspace Identity

Claude Code Integrate with Power Bi

Re: FTL4 Trial Spark failing - InvalidRequestClusterFromFabricDenyList - Cluster Cancelled before Ready

Re: Background CU used in Fabric Capacity Metrics

Re: Approach for automating Dataflow Gen2 connection binding across Dev/Prod?

Re: Background CU used in Fabric Capacity Metrics

Approach for automating Dataflow Gen2 connection binding across Dev/Prod?

Re: Pure Python Notebook: authenticate to Azure DevOps Artifacts using Fabric Workspace Identity

Re: New Browser Tab Opening to Previous or Existing Session

Re: GraphQL update... mutation for Fabric SQL DB failing

Re: Is a table ordinary, or special (= from a shortcut, a system table or materialized lake view table)?

Re: GraphQL update... mutation for Fabric SQL DB failing

From On-Premises SQL Server to Microsoft Fabric with a Metadata-Driven Medallion Pipeline

Architecture

One metadata file, three stages

Uploading the artifacts with Fabric CLI

Stage 1: SQL Server to Bronze

Stage 2: Bronze to Silver

Stage 3: Silver to Gold

Monitoring the complete flow

Result

Further reading

Re: DataFlow Gen2 Issue