Programmatic point-of-failure recovery for Fabric pipelines
The Fabric monitoring UI already supports Rerun → rerun from failed activity. That capability is only reachable by a human clicking in the portal. Please make point-of-failure recovery available to automation as well through whatever mechanism fits the platform best. When a long pipeline fails at a late step, anything automated can only restart it from the beginning. Re-running from step one re-executes hours of already-successful work, burns capacity, delays the SLA, and risks duplicate loads or re-triggered downstream refreshes where activities aren't perfectly idempotent. So today recovery either requires someone to open the portal just to make reruns resumable. A supported way to resume a failed pipeline run from its point of failure without a person in the portal. I'm deliberately not prescribing the mechanism. Any of these would solve it: A recovery option on the existing job scheduler API A "rerun from failure" action available to Activator rules on pipeline failure events A pipeline-level setting for automatic resume-on-retry A first-class checkpoint/resume capability that survives a plain re-run Something else the product team considers a better fit Thanks for your attention,
Unexpected error in fabric pyspark notebook
Hi Everyone, I am using Fabric pyspark notebook .It was as working fine until last month but when i am trying to run it now but showing error like F is not defined but I have already imported functions as F one time it runs successfully next time it gives error even if function or variable is already defined .Can anyone help me.
Best Practices (?) for invoking Notebooks through Notebooks
Dear community, we are trying to make "post deployment scripts" work by invoking child notebooks through a parent notebook. ParentNotebook (default lakehouse: lh1) --- invokes ChildNotebook1 (default lakehouse: lh1) --- invokes ChildNotebook2 (default lakehouse: lh2) Use case: Rename a column in lh1. Parent notebook drops table tbl1 in lakehouse lh1 to avoid schema collision upon recalculation of the table. Parent notebook then invokes ChildNotebook1 that generates table tbl1 in lh1 (with the new column name). Now the catch. We have another lakehouse lh2 for which we want to use the same post deployment notebook. The child generates a table tbl2. BUT, since default lakehouse of ParentNotebook = lh1!= lh2 = default lakehouse of ChildNotebook2 I receive an error AnalysisException Couldn't find a catalog to handle the identifier lh2.dbo.tbl2. Issue: When starting a Spark session, the sessions metastore is initialized with the default lakehouse's data catalog and cannot be altered down the line. I assume that this is due to data security. Current workaround: Run the ParentNotebook once with default lakehouse = lh1. Re-run the notebook with default lakehouse = lh2. but this is somewhat cumbersome and manual. Are there any best practices? NOTE: changes in the tables lh1 unfortunately affect changes in the tables in lh2, which is why splitting post deployment into two scripts is also not ideal. An online search has not yielded any satisfying results. Please feel free to comment and help us out. Thank you very much! Tobias
GraphQL update... mutation for Fabric SQL DB failing
Hi everybody, I want to update items in a Fabric SQL DB via GraphQL mutation query. But that fails with "InternalServerError" (and nothing else). What I tried: ❌ mutation update.... ✅ mutation create... ✅ mutation delete... ✅ (directly in the SQL DB): UPDATE dbo.product SET product_name = 'CHANGED' WHERE ROWID = '<UUID>' Did someone else here experience something similar? Is this a defect 🐞 in the GraphQL API / Fabric SQL DB / their combination? What was your solution? (And what was the issue exactly?) A workaround might be to create a stored proc and let that do the update. I'd prefer to be able to use a working update mutation though. So Any help highly appreciated! <3 --> Snippets below Best Martin (Snippets stripped down for readability) The failing mutation: mutation { updateproduct ( ROWID: "<UUID>", item: { product_name: "CHANGED" }) { ROWID product_name } } //// Definitions from the GraphQL schema: """Updates a product""" updateproduct ( """The ID of the item being updated.""" ROWID: UUID! """Input representing all the fields for updating product""" item: UpdateproductInput! ): product """Input type for updating product_mapping""" input UpdateproductInput { """Input for field ROWID on type UpdateproductInput""" ROWID: UUID """Input for field product_name on type UpdateproductInput""" product_name: String } The table in Fabric SQL DB (as by "Script as create") CREATE TABLE [dbo].[product]( [ROWID] [uniqueidentifier] NOT NULL, [product_name] [nvarchar](255) NOT NULL, ) ON [PRIMARY] ALTER TABLE [dbo].[product_mapping] ADD PRIMARY KEY CLUSTERED ( [ROWID] ASC ) WITH ( STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ONLINE = OFF, OPTIMIZE_FOR_SEQUENTIAL_KEY = OFF) ON [PRIMARY] ALTER TABLE [dbo].[product_mapping] ADD DEFAULT (newsequentialid()) FOR [ROWID]
FTL4 Trial Spark failing - InvalidRequestClusterFromFabricDenyList - Cluster Cancelled before Ready
FTL4 Trial, Canada Central. Spark sessions cannot start in multiple workspaces. Both Starter Pool and custom Small 1-node pool fail. Error is InvalidRequestClusterFromFabricDenyList and cluster is cancelled before reaching Ready. Please advise whether the Fabric Trial capacity/Spark backend needs to be reprovisioned or removed from the deny list.
Looking for dbt-Like Lineage-Aware Refreshes in Microsoft Fabric
Hi all, I have worked with `dbt-databricks` in previous projects, and one of the features I really appreciate is its built-in lineage management and orchestration. It allows you to run a specific part of a pipeline independently while automatically resolving and executing the required dependencies based on the DAG and model relationships. I'm curious whether a similar capability exists in Microsoft Fabric using the currently available orchestration options such as Pipelines, Notebooks, Dataflows, or other Fabric-native approaches. I am familiar with creating DAG-like workflows using pipeline activities and dependency conditions (success/failure), but what I'm looking for is something more lineage-driven. For example, in an end-to-end data engineering solution spanning ingestion → transformation → data marts → reporting, is there a way to selectively refresh a specific table or object and automatically execute only its relevant upstream or downstream dependencies, without impacting unrelated objects in the pipeline? I'd love to hear how others are approaching this in Fabric and whether there are any recommended patterns, tools, or best practices to achieve similar behavior.
SessionStateError
Hi all, During the pipeline run on Sunday, the notebook failed with the following error: SessionStateError: Livy session has failed. Session state: Dead. Session yields "Uncertain" result. The same pipeline completed successfully on Monday without any manual changes or rerun. What typically causes a Livy session to enter the Dead state? Since Monday's run completed successfully, can we confirm that the failed Sunday load was automatically recovered? How to overcome this issue? please guide me i want a solutionFabric Data Warehouse best practices for medallion architectures
Part three of a series on medallion architecture with Fabric Data Warehouse. Good medallion architecture is mostly operational discipline. In part one of this series, we chose the pattern, and in part two, we filled in the Bronze, Silver, and Gold layers. Now comes the part that usually determines whether the architecture holds up in production: the operating rules. Many medallion architectures look great on paper but become difficult to maintain as data volumes, business requirements, and consumers grow. A medallion pipeline is easy to explain and easy to demo. It is more challenging to keep clean over time. The challenges usually start small: row-by-row loads, report-specific logic in the wrong place, transformations that cannot be safely rerun, or Gold tables that slowly become another staging layer. Why this matters Most medallion problems are not caused by the names Bronze, Silver, and Gold. They happen because the pipeline stops behaving like a pipeline. Bronze starts cleaning. Silver starts serving dashboards. Gold starts compensating for upstream data quality. Before long, nobody knows where a rule belongs, and every change feels risky. The goal of best practices is not to add ceremony. The goal is to make the pipeline predictable: predictable loads, repeatable transformations, trusted outputs, and clear places to look when something breaks. Best practice 1: Batch the writes Fabric DW is built for set-based work. Treat ingestion and transformations as batches, not as a stream of tiny row-by-row operations. In Bronze, that usually means using COPY INTO, Fabric Pipelines, or other bulk-loading patterns to land data in raw tables. If you are loading from files, aim for fewer well-sized files instead of many tiny ones. When practical, files in the 100 MB to 1 GB range are a healthier starting point than a long tail of small files. In Silver and Gold, the same idea applies: prefer set-based T-SQL transformations, CTAS, INSERT...SELECT, and MERGE patterns over procedural row-at-a-time logic. Do this well Load Bronze in batches, and avoid trickle inserts when the source can be staged first. Add ingestion metadata, such as source file name and load timestamp, so every batch is traceable. Keep operational logging lightweight. If you need very high-write audit events, do not turn the warehouse into a single-row logging engine. Let Bronze preserve the batch; let Silver decide what is valid. Rule of thumb: if a load pattern creates a large number of tiny writes, fix the load pattern before tuning the query. Best practice 2: Make Silver rerunnable Silver is where the pipeline earns trust. That means Silver transformations need to be repeatable, testable, and safe to rerun. If an upstream source reloads, or a cleansing rule changes, you should know how to rebuild the affected Silver tables without guessing which reports need to be patched. This is where idempotent design matters: a transformation should produce the same result when run again against the same inputs. In Fabric DW, use CTAS when you want to materialize a clean table from a query, INSERT...SELECT for controlled incremental loads, and MERGE when late-arriving or changed data needs to update existing Silver rows. Do this well Design transformations so they can run again without duplicating or corrupting data. Use staging tables when the logic is complex. A few clear steps are easier to operate than one unreadable query. Put quality gates in Silver: required fields, valid formats, duplicate handling, and reason codes for rejected records. Choose precise data types and lengths. Silver is the right place to turn loose source data into reliable analytical data. Rule of thumb: if a report needs to clean the data again, Silver did not finish its job. Best practice 3: Shape Gold for consumption Gold is not just “the final table.” Gold is the business-facing serving layer. It should be modeled around how people ask questions, not around how the source systems store data. For some workloads, that means a star schema with fact and dimension tables. For others, it means a data mart, a wide reporting table, or a pre-aggregated summary. The pattern matters less than the principle: Gold should make the common analytical path simple, fast, and trustworthy. This is also where you should be careful not to let Gold become a junk drawer. If Gold is full of one-off fixes, report-specific exceptions, and raw technical fields, the layer is doing too much. Do this well Model the grain explicitly. A fact table without a clear grain becomes hard to explain and harder to debug. Pre-aggregate where the business repeatedly asks the same question. Hide technical fields that helped the pipeline but do not help the consumer. Keep Gold dependent on Silver by default. Direct Gold-to-Bronze dependencies should be rare and deliberate. Rule of thumb: Gold should answer the business question quickly without making the report author rediscover the pipeline. Best practice 4: Use Fabric DW defaults, but do not fight the engine Fabric DW gives you a SQL warehouse over Delta data in OneLake. That means you get transactional behavior, optimized storage patterns, and a managed engine that handles many physical decisions for you. The practical advice: do not bring every habit from traditional data warehousing with you. You do not need to micromanage distribution or indexing the same way you would in older platforms. Focus first on healthy data layout, set-based transformations, good table design, and predictable query patterns. At the same time, do not ignore the basics. Query performance still benefits from clean data types, useful statistics, well-shaped Gold tables, and avoiding unnecessary scans. Do this well Keep V-Order and platform optimizations on unless you have a measured reason to change them. Use the performance guidance for Fabric Data Warehouse before inventing custom tuning patterns. Check query behavior when a Gold table becomes critical to many reports. Treat advanced exceptions as exceptions. Most teams should start with the defaults and tune only when evidence says to tune. Rule of thumb: tune from evidence, not from habit. Best practice 5: Monitor by layer A medallion pipeline should be observable at each layer. If a dashboard is wrong or slow, you should be able to tell whether the issue started in Bronze ingestion, Silver transformation, or Gold serving. In Fabric DW, use Query Insights and the warehouse monitoring views to understand query behavior, expensive operations, and refresh patterns. Pair that with pipeline-level monitoring so you can see not only whether a job failed, but where the failure happened. Measure the pipeline in terms the team can act on: Bronze load duration, Silver transformation duration, rejected-record counts, Gold refresh duration, and Gold query performance. Do this well Track load and refresh duration by layer. Log the number of records received, accepted, rejected, and published. Watch critical Gold queries after refresh, especially the ones that feed executive dashboards or widely used semantic models. Keep operational alerts tied to business impact. A failed Gold refresh matters differently from a delayed Bronze load. Rule of thumb: if you cannot tell which layer failed, your monitoring is not layer-aware enough. Before moving a medallion pipeline into production, use the following checklist to verify that each layer is operating as intended. The best-practice checklist Area Bronze Silver Gold Write pattern Batch ingest with metadata Set-based transformations Scheduled refreshes Quality rule Preserve what arrived Validate, conform, and flag Expose trusted fields only Performance focus Avoid tiny writes Keep logic rerunnable Shape for common queries Takeaway Part two of this series was about one job per layer. Part three is about operating each job like it matters. Batch the writes. Make Silver rerunnable. Shape Gold for consumption. Use Fabric DW’s managed engine instead of fighting it. Monitor the pipeline by layer so failures are easy to locate and fixes happen in the right place. If you follow those rules, your medallion architecture becomes less fragile over time, not more. This post is part of our Medallion Architecture on Fabric Data Warehouse series: Choosing your medallion pattern in Fabric Data Warehouse Building the Bronze → Silver → Gold layers Fabric DW best practices for medallion architectures Securing and governing your layers Performance tuning your medallion pipeline Ready to go deeper? Explore the Microsoft Fabric Data Warehouse performance guidelines and ingestion guidance, then stay tuned for Part four of this series, where we’ll cover securing and governing your layers.
