Looking for dbt-Like Lineage-Aware Refreshes in Microsoft Fabric
Hi all, I have worked with `dbt-databricks` in previous projects, and one of the features I really appreciate is its built-in lineage management and orchestration. It allows you to run a specific part of a pipeline independently while automatically resolving and executing the required dependencies based on the DAG and model relationships. I'm curious whether a similar capability exists in Microsoft Fabric using the currently available orchestration options such as Pipelines, Notebooks, Dataflows, or other Fabric-native approaches. I am familiar with creating DAG-like workflows using pipeline activities and dependency conditions (success/failure), but what I'm looking for is something more lineage-driven. For example, in an end-to-end data engineering solution spanning ingestion → transformation → data marts → reporting, is there a way to selectively refresh a specific table or object and automatically execute only its relevant upstream or downstream dependencies, without impacting unrelated objects in the pipeline? I'd love to hear how others are approaching this in Fabric and whether there are any recommended patterns, tools, or best practices to achieve similar behavior.
SessionStateError
Hi all, During the pipeline run on Sunday, the notebook failed with the following error: SessionStateError: Livy session has failed. Session state: Dead. Session yields "Uncertain" result. The same pipeline completed successfully on Monday without any manual changes or rerun. What typically causes a Livy session to enter the Dead state? Since Monday's run completed successfully, can we confirm that the failed Sunday load was automatically recovered? How to overcome this issue? please guide me i want a solutionHow do I unzip a .gz file?
I used a data pipeline to make a web call (http) and get a file. The file has been downloaded to the Files area in the lakehouse. How can I uncompress this file? I am using a PySpark notebook to unzip this file. The file itself is good, since I was able to download the file and uncompress it on my windows machine. I tried this code, but that fails. df = spark.read.format("json").option("multiLine", "false").load("Files/bronze/github-events-2025-01-15-12.json.gz") # Display a preview display(df.limit(5)) # Save as Delta table df.write.mode("overwrite").format("delta").saveAsTable("github_events_bronze") print(f"Successfully loaded {df.count()} records")
Fabric SQL Database vs Warehouse
I am building a Metadata driven ETL Framework in Fabric and I am confused on where I should create my metadata storage and pipeline logs. As warehouse specializes in OLAP requirements, by definition I am leaning towards SQL Database which is suggested for OTLP requirements.
Best Practice for Ingesting External APIs and CSV Data into Microsoft Fabric
Hi everyone, I'm looking for guidance on the recommended approach for ingesting external data into Microsoft Fabric. From sources, including WooCommerce, Zoho, and third-party service providers such as Bobgo that expose data through REST APIs. We also receive data in CSV files from various external sources. What is considered best practice in Fabric for bringing this type of data into the platform? For example: Use a Notebook (Python/PySpark) to call APIs and land data in a Lakehouse? Use a Dataflow Gen2 to consume API and CSV data and load it into a Lakehouse? Use a Data Pipeline for orchestration and scheduling? Is Lakehouse the recommended landing destination? Is there a preferred medallion architecture (Bronze → Silver → Gold) for these types of sources? I'm particularly interested in understanding: What ingestion method is preferred for API-based sources. What ingestion method is preferred for CSV files that arrive on a schedule or are uploaded manually. When to choose Notebooks versus Dataflow Gen2. Whether Lakehouse is the recommended destination for raw ingestion. How others are handling authentication, pagination, incremental loads, retries, and error handling for APIs. How others are managing schema drift and changing file structures for CSV-based sources. Any real-world architecture patterns, lessons learned, or recommendations for production workloads. We're looking to establish a standard approach for onboarding new external data sources into Fabric, so any guidance or examples would be greatly appreciated. Thanks in advance!
From ADF Inventory to a Fabric Operating Model: A Practical Migration Playbook (Part 2)
A practical guide to building a portable, metadata-driven ingestion framework for Microsoft Fabric. Learn how JSON configuration, Pipelines or Airflow orchestration, watermarks, retries, and audit tables work together to make data ingestion scalable and safe.
