All Data Engineering posts

Re: DataFlow Gen2 Issue

ShivekMaharaj — Sat, 12 Sep 2026 18:48:45 GMT

One thing I would separate here is the Pipeline network path from the Dataflow Gen2 network path.

The fact that your Pipeline Copy activity started working after the firewall maintenance is useful, but it does not necessarily prove that every endpoint required by Dataflow Gen2 is now reachable.

For an on-premises gateway, Dataflow Gen2 can also need access to OneLake and the staging Lakehouse endpoints, depending on whether staging, referenced queries or Fast Copy are involved.

I would also not focus too much on the missing Azure connections setting. That setting is mainly for workspace-level Azure storage / Log Analytics configuration and is not normally required just to write to a standard Fabric Lakehouse.

I would try one very small isolation test:

one SQL Server query
one Lakehouse destination
no referenced queries
staging disabled
Fast Copy disabled
only a few rows

Microsoft documents that in that specific pattern, Dataflow Gen2 should not require the staging-Lakehouse connection over port 1433.

If that minimal dataflow succeeds, I would then re-enable staging / Fast Copy or add the query references one at a time. That should tell you exactly when the failure appears.

If the minimal version still fails at the Lakehouse destination, I would treat this as a destination-validation / network-path issue rather than a SQL Server source problem, especially since your source preview succeeds and notebooks can already write to the same Lakehouse.

The current Dataflow Gen2 gateway guidance is particularly useful here because it lists the different endpoint requirements for staging and Lakehouse destination scenarios.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Re: Bronze data contracts for REST/API sources — workaround for Get Metadata's connector gap

ShivekMaharaj — Sat, 12 Sep 2026 18:45:35 GMT

Hi damo_mac85,

I think your workaround is reasonable, and I don't think you're missing a native Get Metadata option here.

The current Fabric REST connector lists Pipeline Copy activity as its supported pipeline capability, but it does not currently list Get Metadata.

One thing I would consider changing is where the contract validation happens.

If Bronze is intended to preserve the source payload, I would lean toward:

REST -> raw JSON -> schema/contract validation -> Parquet/Delta

rather than:

REST -> Parquet -> inspect Parquet schema

The reason is that if the API introduces a structural change that prevents the response from being serialized into your expected Parquet shape, the Copy could fail before your notebook ever gets the opportunity to compare schemas. Keeping the raw JSON first gives you an immutable payload to inspect and replay.

For a pre-flight check, Fabric's Web activity can call a REST endpoint before the Copy activity.

If the API exposes an OpenAPI/Swagger or dedicated schema endpoint, I would use that as the contract source where possible. Otherwise, a small sample call could be used to inspect the returned JSON before starting the full extraction.

I would just be careful about treating a sample response as the complete schema, since optional fields may not appear in every payload.

So personally I would use:

Web/schema or sample check -> contract validation -> full raw landing -> typed promotion

and still validate the landed payload as a second guard before promoting it to Delta.

I also couldn't find a currently published Fabric roadmap item for adding REST support to Get Metadata, so I wouldn't design around that arriving at a particular time.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Re: Business Central Data to Fabric

ShivekMaharaj — Sat, 12 Sep 2026 17:35:45 GMT

Hi Saum02,

If this is Business Central Online, I would not build the Fabric solution around direct access to the underlying SQL database.

Microsoft's current Business Central integration guidance recommends using the REST API stack for integrations rather than treating the SaaS database as a directly accessible SQL source.

On the Fabric side, there are a few options worth looking at:

Fabric currently has a Dynamics 365 Business Central connector available as a Dataflow Gen2 source.
For more controlled/custom extraction, the standard Business Central REST APIs are still the preferred interface.
There is also a newer option called BC2Fab, which Microsoft highlighted in the Fabric What's New updates. It is a partner Fabric workload that replicates Business Central data into OneLake using Open Mirroring and incremental change detection, so it may be worth evaluating if your goal is ongoing analytical replication rather than repeatedly extracting through APIs.

If this is Business Central on-premises, then direct SQL access is technically another option because you control the underlying SQL Server. I would still compare that against the API approach before coupling your Fabric model directly to the physical BC database schema.

So the first question I would ask is whether your environment is Business Central Online or on-premises, and roughly how much data / how many tables you need to bring into Fabric.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Re: SharePoint Shortcut showing no Content

ShivekMaharaj — Sat, 12 Sep 2026 16:54:29 GMT

Hi mkjit256,

That extra URL helps quite a lot.

I would not move the files into Shared Documents yet. Microsoft's current OneDrive/SharePoint shortcut documentation does not state that shortcuts are limited to the default Documents library. In fact, Microsoft recommends creating the shortcut at the most specific folder level rather than at the document-library root.

From the URL you posted, it looks like:

BIA is the SharePoint site
Operations is the display name of the document library
Ops is probably the library's internal URL name
PowerBI Data/Output is the folder you actually want

So the target path appears to be something like:

/sites/BIA/Ops/PowerBI Data/Output

The fact that the Fabric picker only exposes Shared Documents therefore looks more like a library-enumeration issue than proof that your custom library is unsupported.

One useful isolation test would be to create a fresh connection against the exact site URL:

https://companyname.sharepoint.com/sites/BIA

using the same organizational account that can browse the Ops library in SharePoint, then check whether Ops appears in the shortcut browser.

If it still only shows Shared Documents, I would capture that as the repro.

There is also a OneLake Shortcuts REST API where a SharePoint shortcut is represented by a location and subpath. That may be worth testing as a workaround because the API lets you specify the SharePoint subpath directly rather than relying on the folder picker.

I would use the library's URL/internal name (Ops) rather than its display name (Operations) when testing the path.

Also confirm that BIA is a normal SharePoint site rather than a subsite or hub site, because Microsoft currently lists SharePoint subsites and hub sites as unsupported for OneLake shortcuts.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Re: Business Central Data to Fabric

NandanHegde — Sat, 12 Sep 2026 14:52:03 GMT

You cannot directly access the underlying database
similar thread : Accessing BC Data

Re: Business Central Data to Fabric

Olufemi7 — Sat, 12 Sep 2026 14:17:35 GMT

Hello Saum02,
If you're on BC Online, I will look at the Business Central APIs first rather than building the Fabric connection around the underlying SQL database.

Microsoft recommends the REST API stack for Business Central integrations.For BC Online, I will start with the Business Central APIs rather than building the Fabric solution around the underlying SQL database.

If this is an on-premises deployment, SQL access is another route to consider, depending on what you need to extract.

Is your BC environment Online or on-premises?

Microsoft docs:
Integration overview for Business Central (for architects and developers)
Introduction to Microsoft Fabric and Business Central

Re: SharePoint Shortcut showing no Content

DreamITLearn — Sat, 12 Sep 2026 10:13:10 GMT

Hey mkjit256 ,

this is usually one of these:

Delegated vs. app permissions – If your SharePoint connection is using an app-only/service principal auth method rather than your own delegated login, it can browse the site and see the folder structure but fail to enumerate files if permissions weren't explicitly granted at the library level, not just the site level.
Broken/synced files showing as empty – If the "Shared Documents" folder has files synced via OneDrive client or has some in a "pending" state, the shortcut browser sometimes shows nothing even though the web UI displays content, this is a caching mismatch between SharePoint's Graph API response and what you see in browser.
URL trailing slash or casing – Since you're using the sites/sitename pattern, double check there's no trailing slash and the site name casing matches exactly. Fabric's shortcut connector can silently fail to traverse if there's a mismatch.
Folder-level unique permissions – If someone broke inheritance on subfolders inside Shared Documents (common in larger orgs), the connector account may see the parent folder exists but can't read into it, this shows as "No Content" rather than an access-denied error, which is misleading.

Fastest way to isolate it: try creating the shortcut against a brand new, simple test folder with no nested subfolders or broken permissions. If that works, it confirms it's a permissions/inheritance issue on your actual folder, not a connector bug.

Re: Fabric SQL Database vs Warehouse

EduardoCastro — Fri, 11 Sep 2026 20:32:27 GMT

Based on the requirements for building a metadata-driven ETL framework in Fabric, the Fabric SQL Database is the more appropriate choice for the metadata storage layer. It is better suited for OLTP (transactional) needs, while the Warehouse is specifically designed for OLAP analytics. Metadata and logs are OLTP — constant small inserts, updates (status flags, watermarks), and deletes.

Re: SharePoint Shortcut showing no Content

mkjit256 — Fri, 11 Sep 2026 13:19:03 GMT

Hi v-aatheeque you might have missed my comment yesterday. Sorry seems that i didn't reply to you but placed my comment as a general reply.

Business Central Data to Fabric

Saum02 — Fri, 11 Sep 2026 11:18:28 GMT

Hii All,

Has anyone explored the BC connection to it's underlying SQL database? Is there limitation to this approach? I need to get data from BC to fabric, is SQL database of BC worth exploring?

Re: SharePoint Shortcut showing no Content

v-aatheeque — Fri, 11 Sep 2026 10:38:24 GMT

Hi mkjit256

Have you had a chance to look through the responses shared earlier? If anything is still unclear, we’ll be happy to provide additional support.

Re: Best Practices (?) for invoking Notebooks through Notebooks

v-achippa — Fri, 11 Sep 2026 10:34:45 GMT

Hi T-Eichinger,

Thank you for the response and confirming that the issue is resolved now. Thank you for being part of Microsoft Fabric Community.

Thanks and regards,
Anjan Kumar Chippa

Re: Fabric SQL Database vs Warehouse

v-aatheeque — Fri, 11 Sep 2026 10:29:49 GMT

Hi VaibhavTiwari

Have you had a chance to look through the responses shared earlier? If anything is still unclear, we’ll be happy to provide additional support.

Re: GPU based SQL - Who is excited ?

v-abhinavmu — Fri, 11 Sep 2026 06:30:59 GMT

Hi ipkus,

Just following up to check if you had a chance to review the information provided. Please let us know if you have any questions.

Thank you.

Re: FTL4 Trial Spark failing - InvalidRequestClusterFromFabricDenyList - Cluster Cancelled before Ready

v-achippa — Fri, 11 Sep 2026 06:30:43 GMT

Hi pritwade,

Thank you for testing on a separate trial capacity. Since spark also fails in a new workspace and now reports session personalization failure with Livy ending as dead, this points to a fabric spark backend provisioning issue rather than notebook or pool configuration.

There is no customer side fix for this. Since you are unable to open a support request from the Trial, please ask your tenant’s Fabric/Microsoft 365 administrator to open a support case through the Fabric Support Center, including the Trace ID, Workspace ID, Pool ID, region, timestamp, and both error messages so the spark backend team can investigate.

https://msit.powerbi.com/admin-portal/supportCenter

Create a Fabric and Power BI Support Ticket - Power BI | Microsoft Learn

Thanks and regards,
Anjan Kumar Chippa

Re: Current or upcoming DP-700 exam vouchers

v-abhinavmu — Fri, 11 Sep 2026 06:18:04 GMT

Hi UshaSriG,

May I check if this issue has been resolved? If not, Please feel free to contact us if you have any further questions.

Thank you

Re: Background CU used in Fabric Capacity Metrics

v-saisrao-msft — Fri, 11 Sep 2026 05:08:28 GMT

Hi dinhhoa,

Have you had a chance to review the solution we shared earlier? If the issue persists, feel free to reply so we can help further.

Thank you.

Re: Fabric Copy Activity fails in Query mode but succeeds in Table mode through an on-premises gateway

alisaleh — Fri, 11 Sep 2026 01:05:05 GMT

Hi GilbertQ ,

Thanks for your response. The SQL Query does actually run and I can preview it next to the box where the TSQL query is being populated.

Regarding the table vs Query , both are using the same connection and the table option work from both Lakehouse type connection and SQL end point connection , yet both fail on the TSQL option.

Re: Bronze data contracts for REST/API sources — workaround for Get Metadata's connector gap

GilbertQ — Thu, 10 Sep 2026 22:46:58 GMT

Hi damo_mac85

What I would probably do if running data sources from REST API is they typically get returned as JSON files. I would then store the JSON file. In the file section of the Lakehouse and then do a schema comparison. That way if it passes then simply take the JSON file and write it directly to the delta table.

Re: Fabric Copy Activity fails in Query mode but succeeds in Table mode through an on-premises gateway

GilbertQ — Thu, 10 Sep 2026 22:45:19 GMT

Hi alisaleh

From my understanding, when using a query it is using the SQL endpoint which is different to the table which is using the table directly. What I would recommend doing is to take your query and run it by the SQL endpoint to make sure it runs successfully and there are no issues. Once that is done, I would then make sure to reconfigure your copy activity using a Sequel server source and the SQL endpoint details to see if that works.