Best Practices (?) for invoking Notebooks through Notebooks
Dear community, we are trying to make "post deployment scripts" work by invoking child notebooks through a parent notebook. ParentNotebook (default lakehouse: lh1) --- invokes ChildNotebook1 (default lakehouse: lh1) --- invokes ChildNotebook2 (default lakehouse: lh2) Use case: Rename a column in lh1. Parent notebook drops table tbl1 in lakehouse lh1 to avoid schema collision upon recalculation of the table. Parent notebook then invokes ChildNotebook1 that generates table tbl1 in lh1 (with the new column name). Now the catch. We have another lakehouse lh2 for which we want to use the same post deployment notebook. The child generates a table tbl2. BUT, since default lakehouse of ParentNotebook = lh1!= lh2 = default lakehouse of ChildNotebook2 I receive an error AnalysisException Couldn't find a catalog to handle the identifier lh2.dbo.tbl2. Issue: When starting a Spark session, the sessions metastore is initialized with the default lakehouse's data catalog and cannot be altered down the line. I assume that this is due to data security. Current workaround: Run the ParentNotebook once with default lakehouse = lh1. Re-run the notebook with default lakehouse = lh2. but this is somewhat cumbersome and manual. Are there any best practices? NOTE: changes in the tables lh1 unfortunately affect changes in the tables in lh2, which is why splitting post deployment into two scripts is also not ideal. An online search has not yielded any satisfying results. Please feel free to comment and help us out. Thank you very much! Tobias
Best Practice for Ingesting External APIs and CSV Data into Microsoft Fabric
Hi everyone, I'm looking for guidance on the recommended approach for ingesting external data into Microsoft Fabric. From sources, including WooCommerce, Zoho, and third-party service providers such as Bobgo that expose data through REST APIs. We also receive data in CSV files from various external sources. What is considered best practice in Fabric for bringing this type of data into the platform? For example: Use a Notebook (Python/PySpark) to call APIs and land data in a Lakehouse? Use a Dataflow Gen2 to consume API and CSV data and load it into a Lakehouse? Use a Data Pipeline for orchestration and scheduling? Is Lakehouse the recommended landing destination? Is there a preferred medallion architecture (Bronze → Silver → Gold) for these types of sources? I'm particularly interested in understanding: What ingestion method is preferred for API-based sources. What ingestion method is preferred for CSV files that arrive on a schedule or are uploaded manually. When to choose Notebooks versus Dataflow Gen2. Whether Lakehouse is the recommended destination for raw ingestion. How others are handling authentication, pagination, incremental loads, retries, and error handling for APIs. How others are managing schema drift and changing file structures for CSV-based sources. Any real-world architecture patterns, lessons learned, or recommendations for production workloads. We're looking to establish a standard approach for onboarding new external data sources into Fabric, so any guidance or examples would be greatly appreciated. Thanks in advance!
FTL4 Trial Spark failing - InvalidRequestClusterFromFabricDenyList - Cluster Cancelled before Ready
FTL4 Trial, Canada Central. Spark sessions cannot start in multiple workspaces. Both Starter Pool and custom Small 1-node pool fail. Error is InvalidRequestClusterFromFabricDenyList and cluster is cancelled before reaching Ready. Please advise whether the Fabric Trial capacity/Spark backend needs to be reprovisioned or removed from the deny list.How do I unzip a .gz file?
I used a data pipeline to make a web call (http) and get a file. The file has been downloaded to the Files area in the lakehouse. How can I uncompress this file? I am using a PySpark notebook to unzip this file. The file itself is good, since I was able to download the file and uncompress it on my windows machine. I tried this code, but that fails. df = spark.read.format("json").option("multiLine", "false").load("Files/bronze/github-events-2025-01-15-12.json.gz") # Display a preview display(df.limit(5)) # Save as Delta table df.write.mode("overwrite").format("delta").saveAsTable("github_events_bronze") print(f"Successfully loaded {df.count()} records")Switching Lakehouses in a Microsoft Fabric Notebook
When working with Microsoft Fabric Notebooks, it is common to work with data that lives in more than one Lakehouse. Instead of creating a separate Notebook for every Lakehouse, Microsoft Fabric allows you to attach multiple Lakehouses to the same Notebook and switch between them when needed. In this article, I’ll demonstrate this using two Lakehouses: Sales and Marketing. The Sales Lakehouse contains a dbo.sales_analytics table, while the Marketing Lakehouse contains a dbo.marketing_analytics table. Connecting Multiple Lakehouses to a Notebook I started by creating a Fabric Notebook and attaching both the Sales and Marketing Lakehouses to it. Once both Lakehouses are connected, they become available within the Notebook environment. This means I can work with data from either Lakehouse without having to create separate Notebooks. For example, the Sales Lakehouse contains: dbo.sales_analytics while the Marketing Lakehouse contains: dbo.marketing_analytics The important part here is that both Lakehouses are connected to the same Notebook. Switching Between Lakehouses One useful feature of Fabric Notebooks is the ability to switch the active Lakehouse from within the Notebook interface. Marketing Lakehouse is the current Lakehouse with pin indicator. If I intend to switch, I can select the Sales Lakehouse as the active Lakehouse and work with the sales_analytics table. To switch, right-click on the Sales Lakehouse (or click on the ellipsis). Then, I can click Select as default lakehouse As seen below, the Sales Lakehouse has been promoted to the top of the Marketing Lakehouse. I can also switch the active Sales Lakehouse back to Marketing and work with the marketing_analytics table, all within the same Notebook. This can be particularly useful when building data engineering or analytics workflows where data is distributed across multiple Lakehouses. Working with the Sales Lakehouse After switching the active Lakehouse to Sales, I can access the sales data from the Notebook. For example: df = spark.sql("SELECT * FROM Sales.dbo.sales_analytics LIMIT 10") display(df) Because Sales is the active Lakehouse, the table can be referenced directly using its table name. I don't need to create another Notebook just to work with the Sales Lakehouse. Query Marketing Analytics Table Note, I don't necessary need to switch to the Marketing Lakehouse before I can query the inherent marketing_analytics table! For example, I can simply author the SparkSQL code shown before to read the data from the marketing_analytics df = spark.sql("SELECT * FROM Marketing.dbo.marketing_analytics LIMIT 10") display(df) Being able to switch Lakehouses makes the Notebook much more flexible because I can move between different Lakehouse environments without leaving the Notebook or creating additional Notebooks. Why Switching Lakehouses Is Useful At first, this might seem like a small feature, but it can be very useful when designing Fabric data engineering solutions. For example, imagine an organisation has separate Lakehouses for different business domains: Sales Lakehouse └── dbo.sales_analytics Marketing Lakehouse └── dbo.marketing_analytics A single Notebook can be connected to both Lakehouses. Depending on the task I'm performing, I can switch between them and work with the appropriate data. This can help reduce the number of Notebooks required in a Fabric workspace and make development more convenient. It can also be useful when demonstrating Fabric capabilities, developing data transformation logic, or working with multiple business-domain Lakehouses. One Notebook, Multiple Lakehouses The key takeaway is that a Fabric Notebook doesn't necessarily have to be tied to just one Lakehouse. By connecting multiple Lakehouses and switching the active Lakehouse when required, I can use a single Notebook to work with data across different Lakehouse environments. In this example, I started with the Sales Lakehouse, queried dbo.sales_analytics, switched to the Marketing Lakehouse, and then queried dbo.marketing_analytics — all from the same Fabric Notebook. This provides a simple and convenient way to work across multiple Lakehouses while keeping related development activities within a single Notebook.
Direct OneLake / ADLS Gen2 Access to Lakehouse & Open Mirroring Tables
Hi Fabric Community, I'm currently exploring whether it's possible for external users to connect directly to Fabric data at the OneLake storage layer using the Azure Data Lake Storage Gen2 connector in Power BI, rather than accessing the data through the SQL Endpoint. The reason for this investigation is that the recent SQL endpoint metering changes have significantly increased costs in some of our environments. We're looking at alternatives where clients could build and refresh their own semantic models directly against the underlying OneLake data, thereby avoiding SQL endpoint usage where possible. What I've Tried I granted a guest user access to the workspace and attempted to connect to both: Lakehouse tables Open Mirroring tables using the Azure Data Lake Storage Gen2 connector in Power BI Desktop. I copied the ABFS(https path rather since the abfs did not work) path from the table properties and connected successfully. However, instead of seeing the Delta table itself, Power BI only exposes the underlying files and folders (Parquet and JSON files), as shown below: While I can browse the files, I have not been able to get Power BI to recognise the table as a Delta table automatically. Goal The objective is to allow clients to: Connect directly to OneLake storage using their guest account. Build Import semantic models in their own tenant. Refresh those semantic models without querying our Fabric SQL Endpoint. Reduce or eliminate SQL Endpoint compute costs resulting from semantic model refreshes and user queries. Questions Has anyone successfully connected to Lakehouse tables through the ADLS Gen2 connector and had Power BI recognise the Delta table structure automatically? Is this supported for Open Mirroring tables as well? Is there a recommended approach for reading Delta tables directly from OneLake in Power BI Desktop? Are there any limitations or permissions that would prevent external B2B users from accessing Delta tables in this manner? Has anyone implemented a similar architecture to avoid SQL Endpoint consumption after the recent metering changes? I also came across the Fabric connector that exposes Lakehouse tables directly through OneLake Data Hub and wanted to understand whether this could be a viable option as well. However, it appears that this method creates the semantic model in our tenant, alongside the Lakehouse where the data is stored. If so, I assume refreshes and queries would still consume resources within our Fabric environment. Has anyone tested this approach and compared the cost implications against direct access to the underlying Delta tables through ADLS Gen2? Here are a few concepts I am currently exploring: Copying the materialized/gold tables into the client's tenant, allowing them to build and manage their own semantic models and reports entirely within their environment. Creating and maintaining the semantic model in our tenant, with the client connecting their reports directly to that shared semantic model. Creating shortcuts from our gold tables into the client's tenant, enabling them to build semantic models and reports against the shortcut data in their own Fabric environment. I'm not entirely sure whether this approach is supported across tenants, but it is something I'm currently investigating. At this stage, I'm evaluating the pros and cons of each option, particularly with regard to performance, governance, data ownership, and the impact of the recent SQL endpoint metering changes. It is important to note, however, that these clients do not currently have Microsoft Fabric licenses or capacities in their own tenants. They only have Power BI Pro licenses available. Any potential solution would therefore need to operate within those licensing constraints and ideally avoid requiring the client to purchase or maintain Fabric capacity in their environment. Any guidance, best practices, or pointers would be greatly appreciated. Thanks in advance! Kind regards,DataFlow Gen2 Issue
Hi, Could someone please help me? I'm trying to load data from my on-premises SQL Server database into a Lakehouse in Microsoft Fabric. Step 1: Load data into Power Query (Dataflow Gen2) from SQL Server on-premises. Result: Success. Step 2: Load the data into my Lakehouse. This is where I'm encountering the problem. When I try to connect to the Lakehouse destination, I receive the following error: Additional information: The SQL Server connection is working correctly. The On-premises Data Gateway status is Online. I can access and write to the Lakehouse successfully from Fabric Notebooks. The issue occurs only when trying to use the Lakehouse as the destination in Dataflow Gen2. Has anyone experienced a similar issue or found a solution? Thank you.
