Data Engineering topics

FTL4 Trial Spark failing - InvalidRequestClusterFromFabricDenyList - Cluster Cancelled before Ready

pritwade — Wed, 02 Sep 2026 04:12:07 GMT

FTL4 Trial, Canada Central. Spark sessions cannot start in multiple workspaces. Both Starter Pool and custom Small 1-node pool fail. Error is InvalidRequestClusterFromFabricDenyList and cluster is cancelled before reaching Ready. Please advise whether the Fabric Trial capacity/Spark backend needs to be reprovisioned or removed from the deny list.

Unable to open Fabric Data App connected to a semantic model

Aparnaa_MS — Tue, 01 Sep 2026 09:58:13 GMT

Hi Community,

I have created and deployed a Data App in Fabric Apps and connected it to an existing Semantic model.

The deployment completes successfully. However, when I select Open from the deployed app page, the app does not load and displays the following message:

"Can’t open this app outside Fabric
Opening apps connected to semantic models outside of the Fabric portal is not supported at this time."

I understand from the message that Fabric Apps connected to semantic models cannot currently be opened in a separate browser window. Microsoft documentation also lists this as a known limitation and mentions that selecting Open can cause the visual queries to fail.

Can anyone confirm:

Is there a supported way to open or preview the deployed Data App inside the Fabric portal?
If the Open button launches the app outside Fabric, what is the correct way to test the deployed app?

The semantic model is hosted on fabric capacity.

Any guidance or workaround would be helpful.

Thanks!

Direct Lake vS Direct Query

Sparkdata — Mon, 31 Aug 2026 21:14:30 GMT

Came across this, has some deep insights.

i personally find Direct Query more flexible, what do you all prefer ?

https://medium.com/@rishabh.gulati_94109/microsoft-fabric-direct-lake-is-fast-good-data-architecture-is-faster-340276de2e6b

Semantics as Code: Looking for Collaborators on Open Enterprise Business Semantics

vkondepati — Mon, 31 Aug 2026 16:27:58 GMT

Infrastructure became code.

Pipelines became code.

Policies became code.

Why is enterprise business meaning still scattered across catalogs, BI models, spreadsheets, wikis, and tribal knowledge?

I’ve been working on Semantics as Code, an open-source, vendor-neutral approach for defining enterprise business meaning as version-controlled, testable, and deployable artifacts.

The idea is to treat business semantics with the same engineering discipline we apply to infrastructure and data pipelines.

Business entities, metrics, relationships, glossary terms, ownership, governance metadata, quality expectations, and AI context can be defined as code, validated through CI/CD, reviewed through Git, and generated for downstream data and AI platforms.

For example, instead of allowing every dashboard, data product, or AI agent to independently interpret what Revenue, Customer, or Active Customer means, we can establish governed semantic definitions that are reusable across the enterprise.

Why this becomes especially important with AI agents

As enterprises adopt agentic AI, I believe we face a problem beyond traditional data governance.

Data governance can answer:

“Can this agent access this data?”

Policy governance can answer:

“Is this agent allowed to perform this action?”

But we also need to answer:

“What does this business concept mean, and which definition should this agent use in this context?”

I’m exploring this as Meaning Governance—using Semantics as Code as a foundation for providing authoritative, governed business context to AI agents.

The open project currently supports semantic definitions for entities, metrics, relationships, glossary terms, quality expectations, governance metadata, and AI context. The reference implementation also includes generation targets for platforms and technologies including Databricks Metric Views, dbt, OpenMetadata, knowledge graphs, and AI context artifacts.

Looking for collaborators

I’d love to collaborate with people in the Databricks community interested in:

Databricks Metric Views and semantic layers

Unity Catalog and data governance

Agentic AI / AI agents

Enterprise metadata and business glossaries

Semantic models and knowledge graphs

Semantic interoperability

Data contracts and data quality

Governance-as-code / policy-as-code

Building Databricks adapters and real-world examples

I’m particularly interested in exploring how Semantics as Code + Databricks + AI agents could work together to create a governed semantic foundation where business meaning is portable, testable, traceable, and consumable by both humans and autonomous agents.

The project is open source, and contributions, architectural feedback, use cases, criticism, and research collaboration are all welcome.

Project documentation:

https://vkondepati.github.io/semantics-as-code/

GitHub:

https://github.com/vkondepati/semantics-as-code

If this problem resonates with you, I’d love to connect and collaborate.

Define once. Govern everywhere.

Fabric UDFs Deployment ISSUEs

Hansie151 — Mon, 31 Aug 2026 13:31:58 GMT

When using python packages the UFDs work in testing mode, but when published and running in run mode I get the following error:

Error:

{ "functionName": "hello_fabric", "invocationId": "00000000-0000-0000-0000-000000000000", "status": "Failed", "errors": [ { "errorCode": "WorkloadException", "subErrorCode": "NotFound", "message": "User data function: 'hello_fabric' invocation failed." } ] }

I have pinpointed that this only happens when adding the bigquery auth package and also only in prod/run mode. I have no idea how to fix it as it was working for the past month but after small change to my functions the publishing always show this error.

Example function:

import datetime import fabric.functions as fn import logging import json from google.cloud import bigquery from google.oauth2.credentials import Credentials udf = fn.UserDataFunctions() def get_bigquery_client(var_lib: fn.FabricVariablesClient, auth_type: str = "SERVICE") -> bigquery.Client: variables = var_lib.getVariables() auth_type_upper = auth_type.upper() if auth_type_upper == "PERSONAL": json_string = variables.get("GCP_PERSONAL_API_CREDS_JSON") or "" if not json_string: raise ValueError("auth_type is 'PERSONAL' but GCP_PERSONAL_API_CREDS_JSON is missing or empty.") credentials_info = json.loads(json_string) project_id = credentials_info.get("quota_project_id") user_creds = Credentials.from_authorized_user_info(credentials_info) return bigquery.Client(credentials=user_creds, project=project_id) if auth_type_upper == "SERVICE": json_string = variables.get("GCP_API_CREDS_JSON") or "" if not json_string: raise ValueError("auth_type is 'SERVICE' but GCP_API_CREDS_JSON is missing or empty.") credentials_info = json.loads(json_string) return bigquery.Client.from_service_account_info(credentials_info) raise ValueError(f"Invalid auth_type '{auth_type}'. Must be 'SERVICE' or 'PERSONAL'.") @udf.connection(argName="varLib", alias="apivariables") @udf.function() def hello_fabric(varLib: fn.FabricVariablesClient,name: str) -> str: client = get_bigquery_client(varLib, "PERSONAL") logging.info('Python UDF trigger function processed a request.') return f"{varLib} {client} Welcome to Fabric Functions, {name}, at {datetime.datetime.now()}!"

Any help would be appreciated!

Looking for dbt-Like Lineage-Aware Refreshes in Microsoft Fabric

tsingh — Mon, 31 Aug 2026 12:54:14 GMT

Hi all,

I have worked with `dbt-databricks` in previous projects, and one of the features I really appreciate is its built-in lineage management and orchestration. It allows you to run a specific part of a pipeline independently while automatically resolving and executing the required dependencies based on the DAG and model relationships.

I'm curious whether a similar capability exists in Microsoft Fabric using the currently available orchestration options such as Pipelines, Notebooks, Dataflows, or other Fabric-native approaches. I am familiar with creating DAG-like workflows using pipeline activities and dependency conditions (success/failure), but what I'm looking for is something more lineage-driven.

For example, in an end-to-end data engineering solution spanning ingestion → transformation → data marts → reporting, is there a way to selectively refresh a specific table or object and automatically execute only its relevant upstream or downstream dependencies, without impacting unrelated objects in the pipeline?

I'd love to hear how others are approaching this in Fabric and whether there are any recommended patterns, tools, or best practices to achieve similar behavior.

ANY VOUCHER FOR DP-700 OR DP-600

jayavardhan3557 — Fri, 28 Aug 2026 10:23:35 GMT

Please i need a voucher for DP-700 or DP-600. I would appreciate if any of the mods are giving out free vouchers.

Issue with Fabric Apps To-do template

Aparnaa_MS — Fri, 28 Aug 2026 04:37:18 GMT

Hi Everyone,

I am using the To-do app template in Fabric Apps. The deployed app opens successfully sometimes, but at other times authentication does not complete and the app remains stuck on the authentication page.

There are no configuration changes between the successful and failed attempts.

Has anyone experienced this intermittent behaviour? Are there any authentication settings, permissions, redirect configurations, or logs that I should check?

Thanks in advance for your help.

When to Choose Views vs. Materialized Lake Views

ipkus — Thu, 27 Aug 2026 15:41:58 GMT

This is a question that comes up often as teams start designing their Fabric architecture. The decision looks simple, but it can directly impact Power BI performance, Direct Lake vs. DirectQuery, data freshness, and overall development complexity.

I came across this article which covers this topic. What do you all think, is there a preference ?

https://medium.com/@rishabh.gulati_94109/microsoft-fabric-balancing-speed-and-complexity-when-to-choose-views-vs-materialized-lake-views-b43b9d0d579b

Direct OneLake / ADLS Gen2 Access to Lakehouse & Open Mirroring Tables

FabricEnjoyer — Thu, 27 Aug 2026 13:32:50 GMT

Hi Fabric Community,

I'm currently exploring whether it's possible for external users to connect directly to Fabric data at the OneLake storage layer using the Azure Data Lake Storage Gen2 connector in Power BI, rather than accessing the data through the SQL Endpoint.

The reason for this investigation is that the recent SQL endpoint metering changes have significantly increased costs in some of our environments. We're looking at alternatives where clients could build and refresh their own semantic models directly against the underlying OneLake data, thereby avoiding SQL endpoint usage where possible.

What I've Tried

I granted a guest user access to the workspace and attempted to connect to both:

Lakehouse tables
Open Mirroring tables

using the Azure Data Lake Storage Gen2 connector in Power BI Desktop.

I copied the ABFS(https path rather since the abfs did not work) path from the table properties and connected successfully. However, instead of seeing the Delta table itself, Power BI only exposes the underlying files and folders (Parquet and JSON files), as shown below:

While I can browse the files, I have not been able to get Power BI to recognise the table as a Delta table automatically.

Goal

The objective is to allow clients to:

Connect directly to OneLake storage using their guest account.
Build Import semantic models in their own tenant.
Refresh those semantic models without querying our Fabric SQL Endpoint.
Reduce or eliminate SQL Endpoint compute costs resulting from semantic model refreshes and user queries.

Questions

Has anyone successfully connected to Lakehouse tables through the ADLS Gen2 connector and had Power BI recognise the Delta table structure automatically?
Is this supported for Open Mirroring tables as well?
Is there a recommended approach for reading Delta tables directly from OneLake in Power BI Desktop?
Are there any limitations or permissions that would prevent external B2B users from accessing Delta tables in this manner?
Has anyone implemented a similar architecture to avoid SQL Endpoint consumption after the recent metering changes?

I also came across the Fabric connector that exposes Lakehouse tables directly through OneLake Data Hub and wanted to understand whether this could be a viable option as well.

However, it appears that this method creates the semantic model in our tenant, alongside the Lakehouse where the data is stored. If so, I assume refreshes and queries would still consume resources within our Fabric environment.

Has anyone tested this approach and compared the cost implications against direct access to the underlying Delta tables through ADLS Gen2?

Here are a few concepts I am currently exploring:

Copying the materialized/gold tables into the client's tenant, allowing them to build and manage their own semantic models and reports entirely within their environment.
Creating and maintaining the semantic model in our tenant, with the client connecting their reports directly to that shared semantic model.
Creating shortcuts from our gold tables into the client's tenant, enabling them to build semantic models and reports against the shortcut data in their own Fabric environment. I'm not entirely sure whether this approach is supported across tenants, but it is something I'm currently investigating.

At this stage, I'm evaluating the pros and cons of each option, particularly with regard to performance, governance, data ownership, and the impact of the recent SQL endpoint metering changes.

It is important to note, however, that these clients do not currently have Microsoft Fabric licenses or capacities in their own tenants. They only have Power BI Pro licenses available. Any potential solution would therefore need to operate within those licensing constraints and ideally avoid requiring the client to purchase or maintain Fabric capacity in their environment.

Any guidance, best practices, or pointers would be greatly appreciated.

Thanks in advance!

Kind regards,

SessionStateError

kamal2 — Thu, 27 Aug 2026 12:34:09 GMT

Hi all,

During the pipeline run on Sunday, the notebook failed with the following error:

SessionStateError: Livy session has failed. Session state: Dead. Session yields "Uncertain" result.
The same pipeline completed successfully on Monday without any manual changes or rerun.

What typically causes a Livy session to enter the Dead state?

Since Monday's run completed successfully, can we confirm that the failed Sunday load was automatically recovered?

How to overcome this issue? please guide me i want a solution

DataFlow Gen2 Issue

LUISS252 — Thu, 27 Aug 2026 00:00:56 GMT

Hi,

Could someone please help me?

I'm trying to load data from my on-premises SQL Server database into a Lakehouse in Microsoft Fabric.

Step 1: Load data into Power Query (Dataflow Gen2) from SQL Server on-premises.
Result: Success.

Step 2: Load the data into my Lakehouse. This is where I'm encountering the problem.

When I try to connect to the Lakehouse destination, I receive the following error:

Additional information:

The SQL Server connection is working correctly.
The On-premises Data Gateway status is Online.
I can access and write to the Lakehouse successfully from Fabric Notebooks.
The issue occurs only when trying to use the Lakehouse as the destination in Dataflow Gen2.

Has anyone experienced a similar issue or found a solution?

Thank you.

Multiple developers working on the same Fabric item

bao_phan — Wed, 26 Aug 2026 15:02:39 GMT

Hi everyone,

I’m looking for some advice on the best way in Microsoft Fabric when multiple developers need to work on the same item, such as a Pipeline, Notebook, or other Fabric artifact.

For example, we have 2 devs working on the same notebook at the same time. Each developer may make changes to different parts of the item, and eventually we need to combine those changes and deploy the final version.

I’m wondering how do you normally handle multiple developers working on the same Fabric item?

I read Microsoft Docs and know that they advise to create seperate workspace for each dev (acting as branch like we have in Git). Beside that options, do we have anything else ?

Thanks!

Is a table ordinary, or special (= from a shortcut, a system table or materialized lake view table)?

Ben-Dev — Tue, 25 Aug 2026 19:40:20 GMT

Hi,

In Python/PySpark/Spark SQL, is there a way to determine what kind of lakehouse table a table is—that is whether the table is a) an ordinary table, b) a materialized lake view table, c) a table from a shortcut, or d) a system table (e.g. like sys_dq_metrics)?

In PySpark, I can use `spark.catalog.listTables()` to get details about the tables in a lakehouse. In Spark SQL, I can use `SHOW TABLES` to get the table list then `DESCRIBE DETAIL some_table_name` on each to get the particular table's details. However, in both cases, these details returned don't seem to give me an easy way to differentiate between ordinary tables and special tables (materialized views, shortcust, system tables).

I'd really like to be able to differentiate between the two....

Any ideas?

Thanks!

Safest approach for handling schema changes

Selcii-16 — Tue, 25 Aug 2026 13:03:31 GMT

What is the safest approach for handling schema changes in source tables when downstream Fabric pipelines, Lakehouse tables, and Power BI semantic models depend on them?

For example, if a source system adds a new column, renames an existing column, changes a data type, or removes a column, what is the recommended way to manage these changes without breaking downstream pipelines and reports?

Would you recommend using schema validation, a staging layer, versioned schemas, or some other approach in Microsoft Fabric?

Architecture Best Practice

FabricEnjoyer — Mon, 24 Aug 2026 13:19:05 GMT

Current Architecture and Challenges

We currently operate approximately 32 Fabric workspaces, each with its own Open Mirroring Database (OMDB).

Users access data in several ways:

Querying the SQL endpoint directly from tools such as Excel.
Creating and querying SQL views within the SQL endpoint.
Connecting semantic models to SQL views hosted on the SQL endpoint.
Building reports against semantic models that ultimately source data from the SQL endpoint.

Since the recent changes to the Fabric metering model, we have experienced a significant increase in capacity consumption. This appears to be driven by a combination of:

Users refreshing Excel workbooks multiple times per day against SQL endpoints.
Reports and ad hoc queries being executed directly against SQL endpoints.
Semantic models being refreshed several times per day, in some cases up to 8 refreshes daily.
Multiple workspaces generating concurrent query workloads against OMDB SQL endpoints.

Current Architecture:

Would appreciate any advice on approaches that could help reduce both user-driven and system-driven SQL Endpoint CU consumption under the new metering model, while still maintaining a good user experience.

Ideally, we're looking for solutions that:

Reduce capacity usage and associated costs.
Minimize the impact of ad hoc user queries, Excel refreshes, and semantic model refreshes.
Avoid introducing significant performance degradation or noticeably slower refresh times.
Scale effectively across multiple workspaces and customers.
Allow us to continue providing a responsive reporting experience without users experiencing delays or timeouts.

We would be particularly interested in feedback on:

Whether SQL Pools could help reduce overall CU consumption in our scenario, and if so, what types of workloads would benefit most.
Recommended architectural patterns for separating reporting workloads from Open Mirroring Database SQL Endpoints.
Best practices for serving Power BI, Excel, and other analytical workloads in a way that minimizes Fabric capacity consumption.
Any real-world experiences or lessons learned following the recent Fabric metering changes.

Any guidance or recommendations would be greatly appreciated.

Fabric administrator’s perspective

reddyr2502 — Fri, 21 Aug 2026 02:12:53 GMT

Hi,

I am the only person responsible for managing Microsoft Fabric in my company, including administration, data engineering, workspace management, security, governance, and reporting.

What initial setup steps and preventive measures should I complete from a Fabric administrator’s perspective? For example, I would like to enable item recovery and configure other important settings that protect the environment from accidental deletion, data loss, security issues, and operational failures.

Could you provide a practical checklist covering tenant settings, workspace governance, access control, backup and recovery, monitoring, auditing, capacity management, deployment practices, and business continuity?

Code organization for Bronze layer

reddyr2502 — Fri, 21 Aug 2026 02:10:19 GMT

Hi,

I am the only person managing data analytics at a startup. We use a single Microsoft Fabric workspace for our Bronze, Silver, and Gold data layers.

For the Bronze layer, we ingest data from more than 10 source applications using a metadata-driven framework. Each application has different connection requirements and may return data in formats such as CSV, simple JSON, or deeply nested JSON.

What is the best way to organize the following components?

Reusable ingestion code
Application-specific connection configurations
Metadata tables and configuration files
Source tables and extraction settings
File-format-specific processing
JSON flattening rules for both simple and nested structures
Pipelines, notebooks, and supporting utilities

I would like the framework to use metadata and configuration files wherever possible, while still supporting application-specific logic when required. Please review the ER diagram for metadata driven tables. Could you recommend a simple, scalable folder structure and an overall design for organizing the Bronze-layer ingestion framework in Microsoft Fabric?

Folders Structures in workspace

reddyr2502 — Fri, 21 Aug 2026 02:05:39 GMT

Hi,

I am the only person managing data analytics at a startup. We currently use a single Microsoft Fabric workspace for our Bronze, Silver, and Gold data layers.

What is the best way to organize and clearly distinguish pipelines, notebooks, dataflows, and other artifacts for each layer?

I am considering creating separate top-level folders for Bronze, Silver, and Gold. Could you recommend a practical folder and subfolder structure for each layer that is simple to manage now and scalable for future growth?

How to create external ai agent from co-pilot and how to connect it to Fabric.

Ka13 — Fri, 21 Aug 2026 01:29:43 GMT

How to create external ai agent from co-pilot and how to connect to Fabric.

I perform the below steps, can you please suggest.

In Azure Portal , from App Registration, created new App
Created new Security group from Microsoft Entra Id and assigned above created App.
But how to authenticate the App Registration with Fabric ?
How the agent can interact with Fabric to list the Lakehouse in the Fabric?