Best Practices (?) for invoking Notebooks through Notebooks
Dear community, we are trying to make "post deployment scripts" work by invoking child notebooks through a parent notebook. ParentNotebook (default lakehouse: lh1) --- invokes ChildNotebook1 (default lakehouse: lh1) --- invokes ChildNotebook2 (default lakehouse: lh2) Use case: Rename a column in lh1. Parent notebook drops table tbl1 in lakehouse lh1 to avoid schema collision upon recalculation of the table. Parent notebook then invokes ChildNotebook1 that generates table tbl1 in lh1 (with the new column name). Now the catch. We have another lakehouse lh2 for which we want to use the same post deployment notebook. The child generates a table tbl2. BUT, since default lakehouse of ParentNotebook = lh1!= lh2 = default lakehouse of ChildNotebook2 I receive an error AnalysisException Couldn't find a catalog to handle the identifier lh2.dbo.tbl2. Issue: When starting a Spark session, the sessions metastore is initialized with the default lakehouse's data catalog and cannot be altered down the line. I assume that this is due to data security. Current workaround: Run the ParentNotebook once with default lakehouse = lh1. Re-run the notebook with default lakehouse = lh2. but this is somewhat cumbersome and manual. Are there any best practices? NOTE: changes in the tables lh1 unfortunately affect changes in the tables in lh2, which is why splitting post deployment into two scripts is also not ideal. An online search has not yielded any satisfying results. Please feel free to comment and help us out. Thank you very much! Tobias
Unexpected error in fabric pyspark notebook
Hi Everyone, I am using Fabric pyspark notebook .It was as working fine until last month but when i am trying to run it now but showing error like F is not defined but I have already imported functions as F one time it runs successfully next time it gives error even if function or variable is already defined .Can anyone help me.
Background CU used in Fabric Capacity Metrics
I'm trying to manage the CU usages to lower my billing. My system has 2 main orchestrators to process data for my 2 reports. One will run every 15min and other is 8h. Both will use copy activity to retrieve data from 42 data sources (using ForEach) and then using notebook to process later on. I was previously using an F62 capacity. Based on my observations in the Job tab under Spark Settings, my max workload was only taking around 88/192 CUs (when there was 2 15m orche and 1 8h orche running in the same time), so I decided to scale down to F32. After the downgrade, my system became unresponsive and started hitting capacity limits. My Power BI reports couldn't load, and I was forced to scale the capacity back up to F62. Then, I discovered the Fabric Capacity Metrics app and noticed that my Background workload was consuming nearly 80% of the available CUs. What's strange is that many pipelines, activity, notebooks, and other jobs that had already completed more than 12 hours earlier still appeared to be consuming a small amount of capacity in the background. Does anyone know how to troubleshoot or resolve these background activities? is there a way to identify or clear them?
Looking for dbt-Like Lineage-Aware Refreshes in Microsoft Fabric
Hi all, I have worked with `dbt-databricks` in previous projects, and one of the features I really appreciate is its built-in lineage management and orchestration. It allows you to run a specific part of a pipeline independently while automatically resolving and executing the required dependencies based on the DAG and model relationships. I'm curious whether a similar capability exists in Microsoft Fabric using the currently available orchestration options such as Pipelines, Notebooks, Dataflows, or other Fabric-native approaches. I am familiar with creating DAG-like workflows using pipeline activities and dependency conditions (success/failure), but what I'm looking for is something more lineage-driven. For example, in an end-to-end data engineering solution spanning ingestion → transformation → data marts → reporting, is there a way to selectively refresh a specific table or object and automatically execute only its relevant upstream or downstream dependencies, without impacting unrelated objects in the pipeline? I'd love to hear how others are approaching this in Fabric and whether there are any recommended patterns, tools, or best practices to achieve similar behavior.
Best Practice for Ingesting External APIs and CSV Data into Microsoft Fabric
Hi everyone, I'm looking for guidance on the recommended approach for ingesting external data into Microsoft Fabric. From sources, including WooCommerce, Zoho, and third-party service providers such as Bobgo that expose data through REST APIs. We also receive data in CSV files from various external sources. What is considered best practice in Fabric for bringing this type of data into the platform? For example: Use a Notebook (Python/PySpark) to call APIs and land data in a Lakehouse? Use a Dataflow Gen2 to consume API and CSV data and load it into a Lakehouse? Use a Data Pipeline for orchestration and scheduling? Is Lakehouse the recommended landing destination? Is there a preferred medallion architecture (Bronze → Silver → Gold) for these types of sources? I'm particularly interested in understanding: What ingestion method is preferred for API-based sources. What ingestion method is preferred for CSV files that arrive on a schedule or are uploaded manually. When to choose Notebooks versus Dataflow Gen2. Whether Lakehouse is the recommended destination for raw ingestion. How others are handling authentication, pagination, incremental loads, retries, and error handling for APIs. How others are managing schema drift and changing file structures for CSV-based sources. Any real-world architecture patterns, lessons learned, or recommendations for production workloads. We're looking to establish a standard approach for onboarding new external data sources into Fabric, so any guidance or examples would be greatly appreciated. Thanks in advance!Switching Lakehouses in a Microsoft Fabric Notebook
When working with Microsoft Fabric Notebooks, it is common to work with data that lives in more than one Lakehouse. Instead of creating a separate Notebook for every Lakehouse, Microsoft Fabric allows you to attach multiple Lakehouses to the same Notebook and switch between them when needed. In this article, I’ll demonstrate this using two Lakehouses: Sales and Marketing. The Sales Lakehouse contains a dbo.sales_analytics table, while the Marketing Lakehouse contains a dbo.marketing_analytics table. Connecting Multiple Lakehouses to a Notebook I started by creating a Fabric Notebook and attaching both the Sales and Marketing Lakehouses to it. Once both Lakehouses are connected, they become available within the Notebook environment. This means I can work with data from either Lakehouse without having to create separate Notebooks. For example, the Sales Lakehouse contains: dbo.sales_analytics while the Marketing Lakehouse contains: dbo.marketing_analytics The important part here is that both Lakehouses are connected to the same Notebook. Switching Between Lakehouses One useful feature of Fabric Notebooks is the ability to switch the active Lakehouse from within the Notebook interface. Marketing Lakehouse is the current Lakehouse with pin indicator. If I intend to switch, I can select the Sales Lakehouse as the active Lakehouse and work with the sales_analytics table. To switch, right-click on the Sales Lakehouse (or click on the ellipsis). Then, I can click Select as default lakehouse As seen below, the Sales Lakehouse has been promoted to the top of the Marketing Lakehouse. I can also switch the active Sales Lakehouse back to Marketing and work with the marketing_analytics table, all within the same Notebook. This can be particularly useful when building data engineering or analytics workflows where data is distributed across multiple Lakehouses. Working with the Sales Lakehouse After switching the active Lakehouse to Sales, I can access the sales data from the Notebook. For example: df = spark.sql("SELECT * FROM Sales.dbo.sales_analytics LIMIT 10") display(df) Because Sales is the active Lakehouse, the table can be referenced directly using its table name. I don't need to create another Notebook just to work with the Sales Lakehouse. Query Marketing Analytics Table Note, I don't necessary need to switch to the Marketing Lakehouse before I can query the inherent marketing_analytics table! For example, I can simply author the SparkSQL code shown before to read the data from the marketing_analytics df = spark.sql("SELECT * FROM Marketing.dbo.marketing_analytics LIMIT 10") display(df) Being able to switch Lakehouses makes the Notebook much more flexible because I can move between different Lakehouse environments without leaving the Notebook or creating additional Notebooks. Why Switching Lakehouses Is Useful At first, this might seem like a small feature, but it can be very useful when designing Fabric data engineering solutions. For example, imagine an organisation has separate Lakehouses for different business domains: Sales Lakehouse └── dbo.sales_analytics Marketing Lakehouse └── dbo.marketing_analytics A single Notebook can be connected to both Lakehouses. Depending on the task I'm performing, I can switch between them and work with the appropriate data. This can help reduce the number of Notebooks required in a Fabric workspace and make development more convenient. It can also be useful when demonstrating Fabric capabilities, developing data transformation logic, or working with multiple business-domain Lakehouses. One Notebook, Multiple Lakehouses The key takeaway is that a Fabric Notebook doesn't necessarily have to be tied to just one Lakehouse. By connecting multiple Lakehouses and switching the active Lakehouse when required, I can use a single Notebook to work with data across different Lakehouse environments. In this example, I started with the Sales Lakehouse, queried dbo.sales_analytics, switched to the Marketing Lakehouse, and then queried dbo.marketing_analytics — all from the same Fabric Notebook. This provides a simple and convenient way to work across multiple Lakehouses while keeping related development activities within a single Notebook.
