SessionStateError
Hi all, During the pipeline run on Sunday, the notebook failed with the following error: SessionStateError: Livy session has failed. Session state: Dead. Session yields "Uncertain" result. The same pipeline completed successfully on Monday without any manual changes or rerun. What typically causes a Livy session to enter the Dead state? Since Monday's run completed successfully, can we confirm that the failed Sunday load was automatically recovered? How to overcome this issue? please guide me i want a solution
CU consumption spike when running 20 notebooks concurrently, scheduled daily (F-SKU capacity)
Hi everyone, I'm running into unpredictable Capacity Unit (CU) consumption and would appreciate some guidance from the community. Setup: Capacity type: F-SKU (Fabric capacity), Pay-as-you-go Schedule: Daily, automated trigger, planned to run continuously for 1 year Workload: 20 notebook files running concurrently in the same schedule Each notebook imports historical data with a date range of 3–10 years (daily granularity Looking at the Spark run duration chart in the Capacity Metrics App, most runs complete in a relatively short time taking 1-2 hours. Since this workload is meant to run daily, I want to make sure the setup is sustainable and cost-predictable before scaling it up long-term. Questions: What could cause this kind of duration spike when running notebooks with large historical data imports (3–10 years, daily data) concurrently? Does running 20 notebooks at the same time on a single F-SKU capacity cause resource contention that leads to this kind of unpredictable duration/CU spike? For a workload scheduled to run daily, are there best practices to keep CU consumption predictable and avoid recurring spikes — e.g., staggering start times, splitting the data import range, or adjusting Spark pool/session settings? Given the long-running nature of this schedule (1 year), would Reserved Capacity be more cost-effective than Pay-as-you-go for this kind of consistent daily workload? How can I proactively monitor which specific notebook/session is driving the spike, beyond what's shown in the Capacity Metrics App? Based on this workload pattern (20 concurrent notebooks, daily schedule, each importing 3–10 years of historical data), what F-SKU size would you recommend as a starting point? Is there a rule of thumb for estimating capacity size based on concurrent Spark sessions and data volume? Any insights on capacity sizing, concurrency limits, or scheduling strategy would be very helpful. Thanks in advance!
Upcoming DP-700 Free Voucher Opportunities
Hello everyone, I am currently preparing for the DP-700: Microsoft Fabric Data Engineer Associate certification. I noticed that the recent 100% voucher campaign has ended. Could you please let me know if there are any upcoming official Microsoft/Fabric Community events or campaigns offering free DP-700 exam vouchers? Thank you.
Are Spark Jobs Supposed to Retry on Failure by Default?
My spark job in Fabric has the retry policy disabled (even if I enable it I can't set Max retry attempts to 0, this is the most disabled it gets). But the first time my .py file hits a "raise Exception", the job just restarts (the next time, it doesn't restart). I'm also pretty confident that sometimes after it retries, the entire log for the first pass is just gone. I only know b/c I have some messages being sent to an outside table, and what I see in there doesn't match up with what's actually in the job's log. Any ideas?
