All Data Engineering posts

Re: SharePoint Shortcut showing no Content

mkjit256 — Fri, 11 Sep 2026 13:19:03 GMT

Hi v-aatheeque you might have missed my comment yesterday. Sorry seems that i didn't reply to you but placed my comment as a general reply.

Business Central Data to Fabric

Saum02 — Fri, 11 Sep 2026 11:18:28 GMT

Hii All,

Has anyone explored the BC connection to it's underlying SQL database? Is there limitation to this approach? I need to get data from BC to fabric, is SQL database of BC worth exploring?

Re: SharePoint Shortcut showing no Content

v-aatheeque — Fri, 11 Sep 2026 10:38:24 GMT

Hi mkjit256

Have you had a chance to look through the responses shared earlier? If anything is still unclear, we’ll be happy to provide additional support.

Re: Best Practices (?) for invoking Notebooks through Notebooks

v-achippa — Fri, 11 Sep 2026 10:34:45 GMT

Hi T-Eichinger,

Thank you for the response and confirming that the issue is resolved now. Thank you for being part of Microsoft Fabric Community.

Thanks and regards,
Anjan Kumar Chippa

Re: Fabric SQL Database vs Warehouse

v-aatheeque — Fri, 11 Sep 2026 10:29:49 GMT

Hi VaibhavTiwari

Have you had a chance to look through the responses shared earlier? If anything is still unclear, we’ll be happy to provide additional support.

Re: GPU based SQL - Who is excited ?

v-abhinavmu — Fri, 11 Sep 2026 06:30:59 GMT

Hi ipkus,

Just following up to check if you had a chance to review the information provided. Please let us know if you have any questions.

Thank you.

Re: FTL4 Trial Spark failing - InvalidRequestClusterFromFabricDenyList - Cluster Cancelled before Ready

v-achippa — Fri, 11 Sep 2026 06:30:43 GMT

Hi pritwade,

Thank you for testing on a separate trial capacity. Since spark also fails in a new workspace and now reports session personalization failure with Livy ending as dead, this points to a fabric spark backend provisioning issue rather than notebook or pool configuration.

There is no customer side fix for this. Since you are unable to open a support request from the Trial, please ask your tenant’s Fabric/Microsoft 365 administrator to open a support case through the Fabric Support Center, including the Trace ID, Workspace ID, Pool ID, region, timestamp, and both error messages so the spark backend team can investigate.

https://msit.powerbi.com/admin-portal/supportCenter

Create a Fabric and Power BI Support Ticket - Power BI | Microsoft Learn

Thanks and regards,
Anjan Kumar Chippa

Re: Current or upcoming DP-700 exam vouchers

v-abhinavmu — Fri, 11 Sep 2026 06:18:04 GMT

Hi UshaSriG,

May I check if this issue has been resolved? If not, Please feel free to contact us if you have any further questions.

Thank you

Re: Background CU used in Fabric Capacity Metrics

v-saisrao-msft — Fri, 11 Sep 2026 05:08:28 GMT

Hi dinhhoa,

Have you had a chance to review the solution we shared earlier? If the issue persists, feel free to reply so we can help further.

Thank you.

Re: Fabric Copy Activity fails in Query mode but succeeds in Table mode through an on-premises gateway

alisaleh — Fri, 11 Sep 2026 01:05:05 GMT

Hi GilbertQ ,

Thanks for your response. The SQL Query does actually run and I can preview it next to the box where the TSQL query is being populated.

Regarding the table vs Query , both are using the same connection and the table option work from both Lakehouse type connection and SQL end point connection , yet both fail on the TSQL option.

Re: Bronze data contracts for REST/API sources — workaround for Get Metadata's connector gap

GilbertQ — Thu, 10 Sep 2026 22:46:58 GMT

Hi damo_mac85

What I would probably do if running data sources from REST API is they typically get returned as JSON files. I would then store the JSON file. In the file section of the Lakehouse and then do a schema comparison. That way if it passes then simply take the JSON file and write it directly to the delta table.

Re: Fabric Copy Activity fails in Query mode but succeeds in Table mode through an on-premises gateway

GilbertQ — Thu, 10 Sep 2026 22:45:19 GMT

Hi alisaleh

From my understanding, when using a query it is using the SQL endpoint which is different to the table which is using the table directly. What I would recommend doing is to take your query and run it by the SQL endpoint to make sure it runs successfully and there are no issues. Once that is done, I would then make sure to reconfigure your copy activity using a Sequel server source and the SQL endpoint details to see if that works.

Fabric Copy Activity fails in Query mode but succeeds in Table mode through an on-premises gateway

alisaleh — Thu, 10 Sep 2026 20:25:01 GMT

Hi Fabric Community,

I am using a Microsoft Fabric pipeline Copy Activity to copy data from a Fabric Lakehouse to an on-premises SQL Server database through a dedicated on-premises data gateway.

I tested the Copy Activity using both of the following source connection types:

Lakehouse connection
Lakehouse SQL analytics endpoint connection

With both connection types, the behavior is the same:

Lookup Activity: Successful
Script Activity: Successful
Copy Activity using Table mode: Successful
Copy Activity using Query mode: Fails

When the Copy Activity source is configured with Use query = Query, it fails with the following error:

ErrorCode=SqlFailedToConnect,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Cannot connect to SQL Database. Please contact SQL server team for further support. Server: 'xxxxxxxx.datawarehouse.fabric.microsoft.com', Database: 'LH_XYZ', User: ''. Check the connection configuration is correct, and make sure the SQL Database firewall allows the Data Factory runtime to access.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server),Source=.Net SqlClient Data Provider,SqlErrorNumber=53,Class=20,ErrorCode=-2146232060,State=0,Errors=[{Class=20,Number=53,State=0,Message=A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server),},],''Type=System.ComponentModel.Win32Exception,Message=The network path was not found,Source=,'

However, when I change the source configuration from Query to Table and select the source table directly, the Copy Activity completes successfully.

Has anyone encountered this behavior?

Does Query mode use a different runtime, connector path, or metadata-discovery process when the Copy Activity runs through an on-premises gateway?

Are there any known limitations or additional gateway requirements for using a custom query as the source?

Any guidance on how to diagnose or resolve this would be appreciated.

Bronze data contracts for REST/API sources — workaround for Get Metadata's connector gap

damo_mac85 — Thu, 10 Sep 2026 20:15:32 GMT

For SQL sources, our pattern is: Copy activity lands the table to Parquet, Get Metadata pulls the source structure, a Notebook compares that structure to a JSON contract, and if it matches, a Copy activity promotes Parquet to Delta — if not, we notify instead.

Wanted the same for REST/API sources, but Get Metadata doesn't support REST connectors — so there's no direct "get source structure" step for an API.

Workaround: since Parquet stores its own schema, we skip Get Metadata for API sources and just read it back off the landed file:

Copy: API → Parquet (raw landing)

Notebook: spark.read.parquet(path).schema vs. JSON contract

Match → Copy Parquet to Delta. Mismatch → notify,

Questions for the community:

Anyone found a more native way to get REST schema before landing data?

Is anyone doing a pre-flight sample call to validate before pulling the full payload?

Any word on Get Metadata adding REST support?

Curious how others doing bronze contracts in Fabric have handled this.

Re: Fabric SQL Database vs Warehouse

ipkus — Fri, 11 Sep 2026 14:28:02 GMT

Yeah, either approach can work. We use a Fabric Warehouse for both metadata storage and pipeline logging, and we haven't experienced any performance issues.

In most metadata-driven ETL frameworks, the volume of metadata and log records is relatively small compared to fact and reporting data, so it's unlikely to stress a Warehouse. The Warehouse also provides a familiar SQL experience, making it easy to manage configuration tables, execution logs, audit history, and monitoring queries in a single place.

While SQL Database is often associated with OLTP workloads, I would choose the platform that best aligns with your team's skills and overall architecture. For metadata and pipeline logs, a Warehouse is generally more than capable and keeps everything within the Fabric ecosystem.

If this helped, feel free to give it a kudos or mark it as solution, helps others find it too.

Thanks!

Re: Variable Library

ipkus — Thu, 10 Sep 2026 18:32:54 GMT

Try storing the GUID as ENV Variables. Set ENV variables for each environment ( ex DEV , TST , PRD ) and use them as parameters in pipeline or notebooks.

When you move things from one env to next the GUID are automatically applied.

Re: Direct Lake vS Direct Query

ipkus — Thu, 10 Sep 2026 18:31:07 GMT

I think the article also summarizes the points shared in various answers.

Re: Materialized view vs delta tables in gold layer

ipkus — Thu, 10 Sep 2026 18:29:22 GMT

Materialized Views do physically persist data, and under the covers they are stored as Delta data as well. The key distinction is not the storage format, but the purpose and management of the object.

A Gold Delta table is typically your business-ready, reusable source of truth that can support many different analytical use cases. A Materialized View, on the other hand, is usually created to optimize a specific query pattern, such as pre-aggregating Sales by Region and Month. The Materialized View is automatically maintained and refreshed from its underlying tables.

So while both ultimately store data in Delta format, the recommended pattern is still:

Silver → Gold Delta Tables (source of truth) → Materialized Views (performance optimization for specific workloads)

This provides both reusability and performance without duplicating business logic across multiple datasets.

Also, consider creating Business views as well. MLV might start to capacity overhead if done across the board.

Re: Best Practices (?) for invoking Notebooks through Notebooks

T-Eichinger — Thu, 10 Sep 2026 13:28:03 GMT

Dear GilbertQ ,

thank you very much!

I just tested notebook execution with ABFSS path references and it worked!

Best wishes from Berlin,

Tobias

v-achippa , we can close this thread :)

Re: Materialized view vs delta tables in gold layer

v-sathmakuri — Thu, 10 Sep 2026 12:52:15 GMT

Hi ahmedshalabyy12 ,

For example, if Silver has millions of sales records:

Gold Delta table: Store the cleansed, business ready Sales data. This is your source of truth and can be reused by Power BI, notebooks and other pipelines.
Materialized view: If Power BI frequently needs Sales by Region and Month, create an MV that pre-computes that aggregation for faster queries.

So, the approach would be : Silver -> Gold Delta table -> Materialized views for specific performance needs

In short, use Delta tables for the core Gold data and materialized views when you need to optimize frequently used queries/aggregations.

Thanks!!