Data Engineering topics

Materialized view vs delta tables in gold layer

ahmedshalabyy12 — Wed, 09 Sep 2026 08:06:10 GMT

Hello,

I asked ChatGPT this question, but I got a useless answer from the AI, and it changed its opinion twice during the session. So I decided to ask the experts: from your previous experience, which is better
Materialized views or delta tables in the gold layer? Both store the physical data, and both can do incremental refresh. But when and why, that's my question.

Thank you

Variable Library

Ira_27 — Tue, 08 Sep 2026 22:23:32 GMT

I am trying to use variable library in Lookup task in Fabric pipeline but choosing a GUID for a WH doesnt seem to work. As soon as i use dynamic content in connection it wipes off all the settings and changes to below screenshot.

Has anyone used variable library including connection reference and item reference in LookUp, Notebook tasks?

Background CU used in Fabric Capacity Metrics

dinhhoa — Tue, 08 Sep 2026 16:11:30 GMT

I'm trying to manage the CU usages to lower my billing.

My system has 2 main orchestrators to process data for my 2 reports. One will run every 15min and other is 8h. Both will use copy activity to retrieve data from 42 data sources (using ForEach) and then using notebook to process later on.

I was previously using an F62 capacity. Based on my observations in the Job tab under Spark Settings, my max workload was only taking around 88/192 CUs (when there was 2 15m orche and 1 8h orche running in the same time), so I decided to scale down to F32.

After the downgrade, my system became unresponsive and started hitting capacity limits. My Power BI reports couldn't load, and I was forced to scale the capacity back up to F62.

Then, I discovered the Fabric Capacity Metrics app and noticed that my Background workload was consuming nearly 80% of the available CUs. What's strange is that many pipelines, activity, notebooks, and other jobs that had already completed more than 12 hours earlier still appeared to be consuming a small amount of capacity in the background.

Does anyone know how to troubleshoot or resolve these background activities? is there a way to identify or clear them?

SharePoint Shortcut showing no Content

mkjit256 — Tue, 08 Sep 2026 14:06:17 GMT

Hello I am trying to set a shortcut in Lakehouse (tried both the create table shortcut, and create shortcut from files section) to a SharePoint folder, I can see the site, and also the connection authenticate correctly, however when I click next I am only present by one folder, which is the Shared Document folder that show no content though i am sure that there is files and folders as viewed in browser.

I am using the patter of https://tenant.sharepoint.com/sites/sitename

Best Practices (?) for invoking Notebooks through Notebooks

T-Eichinger — Mon, 07 Sep 2026 12:19:15 GMT

Dear community,

we are trying to make "post deployment scripts" work by invoking child notebooks through a parent notebook.

ParentNotebook (default lakehouse: lh1)

--- invokes ChildNotebook1 (default lakehouse: lh1)

--- invokes ChildNotebook2 (default lakehouse: lh2)

Use case: Rename a column in lh1.

Parent notebook drops table tbl1 in lakehouse lh1 to avoid schema collision upon recalculation of the table.
Parent notebook then invokes ChildNotebook1 that generates table tbl1 in lh1 (with the new column name).

Now the catch. We have another lakehouse lh2 for which we want to use the same post deployment notebook. The child generates a table tbl2. BUT, since

default lakehouse of ParentNotebook = lh1!= lh2 = default lakehouse of ChildNotebook2

I receive an error

AnalysisException
Couldn't find a catalog to handle the identifier lh2.dbo.tbl2.

Issue: When starting a Spark session, the sessions metastore is initialized with the default lakehouse's data catalog and cannot be altered down the line. I assume that this is due to data security.

Current workaround:

Run the ParentNotebook once with default lakehouse = lh1.
Re-run the notebook with default lakehouse = lh2.

but this is somewhat cumbersome and manual. Are there any best practices?

NOTE: changes in the tables lh1 unfortunately affect changes in the tables in lh2, which is why splitting post deployment into two scripts is also not ideal.

An online search has not yielded any satisfying results. Please feel free to comment and help us out.

Thank you very much!

Tobias

Azure free account - where to see 27$ spent of 200$ free credit and other questions

JaimeSG — Sat, 05 Sep 2026 13:02:42 GMT

Hi All,

sorry to disturb but I just opened a free account of 1 year 2 days ago and then I am too unexperience using Azure.

I tried to install a free database but I used "Central Spain" as server and then failed. I realised it was because "Central Spain" can not be use in free trial. Then I did a new installation using "Central France" and the deploy of the new database went OK.

After that I have simplely run very simple queries using the sample database, things like (I am not going to put the exact commands, but it was something like that I write):

- select * from table_1 (not even any kind of join)

- $myVar = 1

select coalesces($mivVar,2) as coalesce_value

and I have things in the next screencapture that I do not understand and that is why I write this post to see if anyone of you could please help me to understand.

Question 1:

- In orange arrow (right part) it seems I have consumed 28,27$ of my free 200$ credit but If I go to the "Azure subscription 1" details, it says "your remaining $200 of free credit expires in ...".(yellow arrow at the left) I do not understand "remaining" in "your remaining $200" .

Do I have 171,73$ (orange arrow) or 200$ as remaining credit?

Question 2: Where can I see where I have spent the 28,27$? If you see in my subsription details on the left, it seems I have not consumed any $, so I do not kwow where those 28,27$ were employed, and I doubt that because running around 10 simple queries as the ones I explained I ran it could be that money.

Question 3: In left wodn part with purple arrows it seems my subscriptons has 2 databases but if I do go recource manager window, botton right part in screenshot, I only see 1 database. How is this possible?

Many thanks to all in advance and sorry for the inconveniences.

Sempy Labs + Azure DevOps

Charline_74 — Fri, 04 Sep 2026 15:48:09 GMT

Do you know if it's possible to run a Sempy Labs module via Azure DevOps—for example, as an action in an Azure DevOps pipeline? If so, do you know how? Do you have any examples?

Thanks in Advance,

Charline BONIER

Unexpected error in fabric pyspark notebook

KJ14 — Fri, 04 Sep 2026 10:59:26 GMT

Hi Everyone,

I am using Fabric pyspark notebook .It was as working fine until last month but when i am trying to run it now but showing error like F is not defined but I have already imported functions as F one time it runs successfully next time it gives error even if function or variable is already defined .Can anyone help me.

How do I unzip a .gz file?

abhidotnet — Thu, 03 Sep 2026 20:55:53 GMT

I used a data pipeline to make a web call (http) and get a file. 
The file has been downloaded to the Files area in the lakehouse. How can I uncompress this file? 

I am using a PySpark notebook to unzip this file. The file itself is good, since I was able to download the file and uncompress it on my windows machine. 

I tried this code, but that fails.

df = spark.read.format("json").option("multiLine", "false").load("Files/bronze/github-events-2025-01-15-12.json.gz")
# Display a preview
display(df.limit(5))
# Save as Delta table
df.write.mode("overwrite").format("delta").saveAsTable("github_events_bronze")
print(f"Successfully loaded {df.count()} records")

GPU based SQL - Who is excited ?

ipkus — Thu, 03 Sep 2026 17:12:08 GMT

Shared some perspective on GPU based SQL. Do take a look

https://medium.com/@rishabh.gulati_94109/microsoft-fabric-gpu-powered-sql-the-next-big-shift-in-analytics-d240e3227dc0

Fabric SQL Database vs Warehouse

VaibhavTiwari — Thu, 03 Sep 2026 09:54:57 GMT

I am building a Metadata driven ETL Framework in Fabric and I am confused on where I should create my metadata storage and pipeline logs. As warehouse specializes in OLAP requirements, by definition I am leaning towards SQL Database which is suggested for OTLP requirements.

Does Fabric Managed Private Endpoint MPE still require F64, or is it supported on all F SKUs?

KarolineSO — Thu, 03 Sep 2026 08:24:32 GMT

I am planning to build a secure Fabric ingestion solution that reads from private AWS data sources using notebooks, while keeping the required Fabric capacity, and therefore cost, as low as possible.

I’m seeing conflicting information regarding the capacity requirements for Managed Private Endpoints in Microsoft Fabric. The current Microsoft Learn documentation appears to state that Managed Private Endpoints require a capacity of F64 or higher

However, Microsoft announced in August 2024 that Managed Private Endpoints became available on any purchased Fabric F capacity, and several later articles still reference this change, not able to add links but some are from current year.

In a previous project, we used this setup successfully on F16, so I would have expected a change back to F64+ to have generated quite a bit of discussion online.

Can anyone confirm what the current requirement in 2026 is?

Can a Spark notebook on an F16 capacity use a Managed Private Endpoint?
Is the F64 requirement in the current Learn documentation outdated, or does it apply only to certain workloads/scenarios?

I’m particularly interested in confirmation from someone who has this working on F16, or another SKU below F64.

Best Practice for Ingesting External APIs and CSV Data into Microsoft Fabric

FabricEnjoyer — Thu, 03 Sep 2026 07:03:08 GMT

Hi everyone,

I'm looking for guidance on the recommended approach for ingesting external data into Microsoft Fabric.

From sources, including WooCommerce, Zoho, and third-party service providers such as Bobgo that expose data through REST APIs. We also receive data in CSV files from various external sources.

What is considered best practice in Fabric for bringing this type of data into the platform?

For example:

Use a Notebook (Python/PySpark) to call APIs and land data in a Lakehouse?
Use a Dataflow Gen2 to consume API and CSV data and load it into a Lakehouse?
Use a Data Pipeline for orchestration and scheduling?
Is Lakehouse the recommended landing destination?
Is there a preferred medallion architecture (Bronze → Silver → Gold) for these types of sources?

I'm particularly interested in understanding:

What ingestion method is preferred for API-based sources.
What ingestion method is preferred for CSV files that arrive on a schedule or are uploaded manually.
When to choose Notebooks versus Dataflow Gen2.
Whether Lakehouse is the recommended destination for raw ingestion.
How others are handling authentication, pagination, incremental loads, retries, and error handling for APIs.
How others are managing schema drift and changing file structures for CSV-based sources.
Any real-world architecture patterns, lessons learned, or recommendations for production workloads.

We're looking to establish a standard approach for onboarding new external data sources into Fabric, so any guidance or examples would be greatly appreciated.

Thanks in advance!

Current or upcoming DP-700 exam vouchers

UshaSriG — Wed, 02 Sep 2026 16:31:39 GMT

Hello, I'm preparing for the DP-700 certification. The previous free-voucher campaign ended on April 20, 2026. Are there any current or upcoming DP-700 voucher campaigns, learning sessions or discount opportunities? Thank you.

FTL4 Trial Spark failing - InvalidRequestClusterFromFabricDenyList - Cluster Cancelled before Ready

pritwade — Wed, 02 Sep 2026 04:12:07 GMT

FTL4 Trial, Canada Central. Spark sessions cannot start in multiple workspaces. Both Starter Pool and custom Small 1-node pool fail. Error is InvalidRequestClusterFromFabricDenyList and cluster is cancelled before reaching Ready. Please advise whether the Fabric Trial capacity/Spark backend needs to be reprovisioned or removed from the deny list.

Unable to open Fabric Data App connected to a semantic model

Aparnaa_MS — Tue, 01 Sep 2026 09:58:13 GMT

Hi Community,

I have created and deployed a Data App in Fabric Apps and connected it to an existing Semantic model.

The deployment completes successfully. However, when I select Open from the deployed app page, the app does not load and displays the following message:

"Can’t open this app outside Fabric
Opening apps connected to semantic models outside of the Fabric portal is not supported at this time."

I understand from the message that Fabric Apps connected to semantic models cannot currently be opened in a separate browser window. Microsoft documentation also lists this as a known limitation and mentions that selecting Open can cause the visual queries to fail.

Can anyone confirm:

Is there a supported way to open or preview the deployed Data App inside the Fabric portal?
If the Open button launches the app outside Fabric, what is the correct way to test the deployed app?

The semantic model is hosted on fabric capacity.

Any guidance or workaround would be helpful.

Thanks!

Direct Lake vS Direct Query

Sparkdata — Mon, 31 Aug 2026 21:14:30 GMT

Came across this, has some deep insights.

i personally find Direct Query more flexible, what do you all prefer ?

https://medium.com/@rishabh.gulati_94109/microsoft-fabric-direct-lake-is-fast-good-data-architecture-is-faster-340276de2e6b

Semantics as Code: Looking for Collaborators on Open Enterprise Business Semantics

vkondepati — Mon, 31 Aug 2026 16:27:58 GMT

Infrastructure became code.

Pipelines became code.

Policies became code.

Why is enterprise business meaning still scattered across catalogs, BI models, spreadsheets, wikis, and tribal knowledge?

I’ve been working on Semantics as Code, an open-source, vendor-neutral approach for defining enterprise business meaning as version-controlled, testable, and deployable artifacts.

The idea is to treat business semantics with the same engineering discipline we apply to infrastructure and data pipelines.

Business entities, metrics, relationships, glossary terms, ownership, governance metadata, quality expectations, and AI context can be defined as code, validated through CI/CD, reviewed through Git, and generated for downstream data and AI platforms.

For example, instead of allowing every dashboard, data product, or AI agent to independently interpret what Revenue, Customer, or Active Customer means, we can establish governed semantic definitions that are reusable across the enterprise.

Why this becomes especially important with AI agents

As enterprises adopt agentic AI, I believe we face a problem beyond traditional data governance.

Data governance can answer:

“Can this agent access this data?”

Policy governance can answer:

“Is this agent allowed to perform this action?”

But we also need to answer:

“What does this business concept mean, and which definition should this agent use in this context?”

I’m exploring this as Meaning Governance—using Semantics as Code as a foundation for providing authoritative, governed business context to AI agents.

The open project currently supports semantic definitions for entities, metrics, relationships, glossary terms, quality expectations, governance metadata, and AI context. The reference implementation also includes generation targets for platforms and technologies including Databricks Metric Views, dbt, OpenMetadata, knowledge graphs, and AI context artifacts.

Looking for collaborators

I’d love to collaborate with people in the Databricks community interested in:

Databricks Metric Views and semantic layers

Unity Catalog and data governance

Agentic AI / AI agents

Enterprise metadata and business glossaries

Semantic models and knowledge graphs

Semantic interoperability

Data contracts and data quality

Governance-as-code / policy-as-code

Building Databricks adapters and real-world examples

I’m particularly interested in exploring how Semantics as Code + Databricks + AI agents could work together to create a governed semantic foundation where business meaning is portable, testable, traceable, and consumable by both humans and autonomous agents.

The project is open source, and contributions, architectural feedback, use cases, criticism, and research collaboration are all welcome.

Project documentation:

https://vkondepati.github.io/semantics-as-code/

GitHub:

https://github.com/vkondepati/semantics-as-code

If this problem resonates with you, I’d love to connect and collaborate.

Define once. Govern everywhere.

Fabric UDFs Deployment ISSUEs

Hansie151 — Mon, 31 Aug 2026 13:31:58 GMT

When using python packages the UFDs work in testing mode, but when published and running in run mode I get the following error:

Error:

{ "functionName": "hello_fabric", "invocationId": "00000000-0000-0000-0000-000000000000", "status": "Failed", "errors": [ { "errorCode": "WorkloadException", "subErrorCode": "NotFound", "message": "User data function: 'hello_fabric' invocation failed." } ] }

I have pinpointed that this only happens when adding the bigquery auth package and also only in prod/run mode. I have no idea how to fix it as it was working for the past month but after small change to my functions the publishing always show this error.

Example function:

import datetime import fabric.functions as fn import logging import json from google.cloud import bigquery from google.oauth2.credentials import Credentials udf = fn.UserDataFunctions() def get_bigquery_client(var_lib: fn.FabricVariablesClient, auth_type: str = "SERVICE") -> bigquery.Client: variables = var_lib.getVariables() auth_type_upper = auth_type.upper() if auth_type_upper == "PERSONAL": json_string = variables.get("GCP_PERSONAL_API_CREDS_JSON") or "" if not json_string: raise ValueError("auth_type is 'PERSONAL' but GCP_PERSONAL_API_CREDS_JSON is missing or empty.") credentials_info = json.loads(json_string) project_id = credentials_info.get("quota_project_id") user_creds = Credentials.from_authorized_user_info(credentials_info) return bigquery.Client(credentials=user_creds, project=project_id) if auth_type_upper == "SERVICE": json_string = variables.get("GCP_API_CREDS_JSON") or "" if not json_string: raise ValueError("auth_type is 'SERVICE' but GCP_API_CREDS_JSON is missing or empty.") credentials_info = json.loads(json_string) return bigquery.Client.from_service_account_info(credentials_info) raise ValueError(f"Invalid auth_type '{auth_type}'. Must be 'SERVICE' or 'PERSONAL'.") @udf.connection(argName="varLib", alias="apivariables") @udf.function() def hello_fabric(varLib: fn.FabricVariablesClient,name: str) -> str: client = get_bigquery_client(varLib, "PERSONAL") logging.info('Python UDF trigger function processed a request.') return f"{varLib} {client} Welcome to Fabric Functions, {name}, at {datetime.datetime.now()}!"

Any help would be appreciated!

Looking for dbt-Like Lineage-Aware Refreshes in Microsoft Fabric

tsingh — Mon, 31 Aug 2026 12:54:14 GMT

Hi all,

I have worked with `dbt-databricks` in previous projects, and one of the features I really appreciate is its built-in lineage management and orchestration. It allows you to run a specific part of a pipeline independently while automatically resolving and executing the required dependencies based on the DAG and model relationships.

I'm curious whether a similar capability exists in Microsoft Fabric using the currently available orchestration options such as Pipelines, Notebooks, Dataflows, or other Fabric-native approaches. I am familiar with creating DAG-like workflows using pipeline activities and dependency conditions (success/failure), but what I'm looking for is something more lineage-driven.

For example, in an end-to-end data engineering solution spanning ingestion → transformation → data marts → reporting, is there a way to selectively refresh a specific table or object and automatically execute only its relevant upstream or downstream dependencies, without impacting unrelated objects in the pipeline?

I'd love to hear how others are approaching this in Fabric and whether there are any recommended patterns, tools, or best practices to achieve similar behavior.