CU consumption spike when running 20 notebooks concurrently, scheduled daily (F-SKU capacity)
Hi everyone, I'm running into unpredictable Capacity Unit (CU) consumption and would appreciate some guidance from the community. Setup: Capacity type: F-SKU (Fabric capacity), Pay-as-you-go Schedule: Daily, automated trigger, planned to run continuously for 1 year Workload: 20 notebook files running concurrently in the same schedule Each notebook imports historical data with a date range of 3–10 years (daily granularity Looking at the Spark run duration chart in the Capacity Metrics App, most runs complete in a relatively short time taking 1-2 hours. Since this workload is meant to run daily, I want to make sure the setup is sustainable and cost-predictable before scaling it up long-term. Questions: What could cause this kind of duration spike when running notebooks with large historical data imports (3–10 years, daily data) concurrently? Does running 20 notebooks at the same time on a single F-SKU capacity cause resource contention that leads to this kind of unpredictable duration/CU spike? For a workload scheduled to run daily, are there best practices to keep CU consumption predictable and avoid recurring spikes — e.g., staggering start times, splitting the data import range, or adjusting Spark pool/session settings? Given the long-running nature of this schedule (1 year), would Reserved Capacity be more cost-effective than Pay-as-you-go for this kind of consistent daily workload? How can I proactively monitor which specific notebook/session is driving the spike, beyond what's shown in the Capacity Metrics App? Based on this workload pattern (20 concurrent notebooks, daily schedule, each importing 3–10 years of historical data), what F-SKU size would you recommend as a starting point? Is there a rule of thumb for estimating capacity size based on concurrent Spark sessions and data volume? Any insights on capacity sizing, concurrency limits, or scheduling strategy would be very helpful. Thanks in advance!
SessionStateError
Hi all, During the pipeline run on Sunday, the notebook failed with the following error: SessionStateError: Livy session has failed. Session state: Dead. Session yields "Uncertain" result. The same pipeline completed successfully on Monday without any manual changes or rerun. What typically causes a Livy session to enter the Dead state? Since Monday's run completed successfully, can we confirm that the failed Sunday load was automatically recovered? How to overcome this issue? please guide me i want a solution
Upcoming DP-700 Free Voucher Opportunities
Hello everyone, I am currently preparing for the DP-700: Microsoft Fabric Data Engineer Associate certification. I noticed that the recent 100% voucher campaign has ended. Could you please let me know if there are any upcoming official Microsoft/Fabric Community events or campaigns offering free DP-700 exam vouchers? Thank you.
Are Spark Jobs Supposed to Retry on Failure by Default?
My spark job in Fabric has the retry policy disabled (even if I enable it I can't set Max retry attempts to 0, this is the most disabled it gets). But the first time my .py file hits a "raise Exception", the job just restarts (the next time, it doesn't restart). I'm also pretty confident that sometimes after it retries, the entire log for the first pass is just gone. I only know b/c I have some messages being sent to an outside table, and what I see in there doesn't match up with what's actually in the job's log. Any ideas?
How to set the maximum number of application attempts?
Hi there, I'm wondering if there is a way to control the maximum number of application attempts for a spark job submitted via the Spark Job Definition? When an application fails, you can see it makes two attempts, then fails completely. I want to set this to only be a single attempt (it's a spark streaming job, and I want the whole application to fail after the first attempt). You can see an example of this below: I have tried to set spark.yarn.maxAttempts via the Spark environment: And through the job's environment tab in the history server you can see the setting has made it through to the app: However, in the same tab you can also see the ID is set to 2 on the second attempt, so it keeps making two attempts then failing completely: Are there any other settings that might be controlling this? Thanks
