Best Practices (?) for invoking Notebooks through Notebooks
Dear community, we are trying to make "post deployment scripts" work by invoking child notebooks through a parent notebook. ParentNotebook (default lakehouse: lh1) --- invokes ChildNotebook1 (default lakehouse: lh1) --- invokes ChildNotebook2 (default lakehouse: lh2) Use case: Rename a column in lh1. Parent notebook drops table tbl1 in lakehouse lh1 to avoid schema collision upon recalculation of the table. Parent notebook then invokes ChildNotebook1 that generates table tbl1 in lh1 (with the new column name). Now the catch. We have another lakehouse lh2 for which we want to use the same post deployment notebook. The child generates a table tbl2. BUT, since default lakehouse of ParentNotebook = lh1!= lh2 = default lakehouse of ChildNotebook2 I receive an error AnalysisException Couldn't find a catalog to handle the identifier lh2.dbo.tbl2. Issue: When starting a Spark session, the sessions metastore is initialized with the default lakehouse's data catalog and cannot be altered down the line. I assume that this is due to data security. Current workaround: Run the ParentNotebook once with default lakehouse = lh1. Re-run the notebook with default lakehouse = lh2. but this is somewhat cumbersome and manual. Are there any best practices? NOTE: changes in the tables lh1 unfortunately affect changes in the tables in lh2, which is why splitting post deployment into two scripts is also not ideal. An online search has not yielded any satisfying results. Please feel free to comment and help us out. Thank you very much! Tobias
Best Practice for Ingesting External APIs and CSV Data into Microsoft Fabric
Hi everyone, I'm looking for guidance on the recommended approach for ingesting external data into Microsoft Fabric. From sources, including WooCommerce, Zoho, and third-party service providers such as Bobgo that expose data through REST APIs. We also receive data in CSV files from various external sources. What is considered best practice in Fabric for bringing this type of data into the platform? For example: Use a Notebook (Python/PySpark) to call APIs and land data in a Lakehouse? Use a Dataflow Gen2 to consume API and CSV data and load it into a Lakehouse? Use a Data Pipeline for orchestration and scheduling? Is Lakehouse the recommended landing destination? Is there a preferred medallion architecture (Bronze → Silver → Gold) for these types of sources? I'm particularly interested in understanding: What ingestion method is preferred for API-based sources. What ingestion method is preferred for CSV files that arrive on a schedule or are uploaded manually. When to choose Notebooks versus Dataflow Gen2. Whether Lakehouse is the recommended destination for raw ingestion. How others are handling authentication, pagination, incremental loads, retries, and error handling for APIs. How others are managing schema drift and changing file structures for CSV-based sources. Any real-world architecture patterns, lessons learned, or recommendations for production workloads. We're looking to establish a standard approach for onboarding new external data sources into Fabric, so any guidance or examples would be greatly appreciated. Thanks in advance!
FTL4 Trial Spark failing - InvalidRequestClusterFromFabricDenyList - Cluster Cancelled before Ready
FTL4 Trial, Canada Central. Spark sessions cannot start in multiple workspaces. Both Starter Pool and custom Small 1-node pool fail. Error is InvalidRequestClusterFromFabricDenyList and cluster is cancelled before reaching Ready. Please advise whether the Fabric Trial capacity/Spark backend needs to be reprovisioned or removed from the deny list.How do I unzip a .gz file?
I used a data pipeline to make a web call (http) and get a file. The file has been downloaded to the Files area in the lakehouse. How can I uncompress this file? I am using a PySpark notebook to unzip this file. The file itself is good, since I was able to download the file and uncompress it on my windows machine. I tried this code, but that fails. df = spark.read.format("json").option("multiLine", "false").load("Files/bronze/github-events-2025-01-15-12.json.gz") # Display a preview display(df.limit(5)) # Save as Delta table df.write.mode("overwrite").format("delta").saveAsTable("github_events_bronze") print(f"Successfully loaded {df.count()} records")
Direct OneLake / ADLS Gen2 Access to Lakehouse & Open Mirroring Tables
Hi Fabric Community, I'm currently exploring whether it's possible for external users to connect directly to Fabric data at the OneLake storage layer using the Azure Data Lake Storage Gen2 connector in Power BI, rather than accessing the data through the SQL Endpoint. The reason for this investigation is that the recent SQL endpoint metering changes have significantly increased costs in some of our environments. We're looking at alternatives where clients could build and refresh their own semantic models directly against the underlying OneLake data, thereby avoiding SQL endpoint usage where possible. What I've Tried I granted a guest user access to the workspace and attempted to connect to both: Lakehouse tables Open Mirroring tables using the Azure Data Lake Storage Gen2 connector in Power BI Desktop. I copied the ABFS(https path rather since the abfs did not work) path from the table properties and connected successfully. However, instead of seeing the Delta table itself, Power BI only exposes the underlying files and folders (Parquet and JSON files), as shown below: While I can browse the files, I have not been able to get Power BI to recognise the table as a Delta table automatically. Goal The objective is to allow clients to: Connect directly to OneLake storage using their guest account. Build Import semantic models in their own tenant. Refresh those semantic models without querying our Fabric SQL Endpoint. Reduce or eliminate SQL Endpoint compute costs resulting from semantic model refreshes and user queries. Questions Has anyone successfully connected to Lakehouse tables through the ADLS Gen2 connector and had Power BI recognise the Delta table structure automatically? Is this supported for Open Mirroring tables as well? Is there a recommended approach for reading Delta tables directly from OneLake in Power BI Desktop? Are there any limitations or permissions that would prevent external B2B users from accessing Delta tables in this manner? Has anyone implemented a similar architecture to avoid SQL Endpoint consumption after the recent metering changes? I also came across the Fabric connector that exposes Lakehouse tables directly through OneLake Data Hub and wanted to understand whether this could be a viable option as well. However, it appears that this method creates the semantic model in our tenant, alongside the Lakehouse where the data is stored. If so, I assume refreshes and queries would still consume resources within our Fabric environment. Has anyone tested this approach and compared the cost implications against direct access to the underlying Delta tables through ADLS Gen2? Here are a few concepts I am currently exploring: Copying the materialized/gold tables into the client's tenant, allowing them to build and manage their own semantic models and reports entirely within their environment. Creating and maintaining the semantic model in our tenant, with the client connecting their reports directly to that shared semantic model. Creating shortcuts from our gold tables into the client's tenant, enabling them to build semantic models and reports against the shortcut data in their own Fabric environment. I'm not entirely sure whether this approach is supported across tenants, but it is something I'm currently investigating. At this stage, I'm evaluating the pros and cons of each option, particularly with regard to performance, governance, data ownership, and the impact of the recent SQL endpoint metering changes. It is important to note, however, that these clients do not currently have Microsoft Fabric licenses or capacities in their own tenants. They only have Power BI Pro licenses available. Any potential solution would therefore need to operate within those licensing constraints and ideally avoid requiring the client to purchase or maintain Fabric capacity in their environment. Any guidance, best practices, or pointers would be greatly appreciated. Thanks in advance! Kind regards,DataFlow Gen2 Issue
Hi, Could someone please help me? I'm trying to load data from my on-premises SQL Server database into a Lakehouse in Microsoft Fabric. Step 1: Load data into Power Query (Dataflow Gen2) from SQL Server on-premises. Result: Success. Step 2: Load the data into my Lakehouse. This is where I'm encountering the problem. When I try to connect to the Lakehouse destination, I receive the following error: Additional information: The SQL Server connection is working correctly. The On-premises Data Gateway status is Online. I can access and write to the Lakehouse successfully from Fabric Notebooks. The issue occurs only when trying to use the Lakehouse as the destination in Dataflow Gen2. Has anyone experienced a similar issue or found a solution? Thank you.
