Scaling up to down fabric capacity giving utilisation error.
Hello, I have upscale my fabric capacity to F32 from F8 because I want to process 400 gb of data so now I have completed my work on few minutes and then I have down scale fabric capacity from F32 to F8 but after scaling down it is giving error like your organization has used entire capacity no capacity available. How to resolve this i don't want to increase capacity to resolve this issue.
CU consumption spike when running 20 notebooks concurrently, scheduled daily (F-SKU capacity)
Hi everyone, I'm running into unpredictable Capacity Unit (CU) consumption and would appreciate some guidance from the community. Setup: Capacity type: F-SKU (Fabric capacity), Pay-as-you-go Schedule: Daily, automated trigger, planned to run continuously for 1 year Workload: 20 notebook files running concurrently in the same schedule Each notebook imports historical data with a date range of 3–10 years (daily granularity Looking at the Spark run duration chart in the Capacity Metrics App, most runs complete in a relatively short time taking 1-2 hours. Since this workload is meant to run daily, I want to make sure the setup is sustainable and cost-predictable before scaling it up long-term. Questions: What could cause this kind of duration spike when running notebooks with large historical data imports (3–10 years, daily data) concurrently? Does running 20 notebooks at the same time on a single F-SKU capacity cause resource contention that leads to this kind of unpredictable duration/CU spike? For a workload scheduled to run daily, are there best practices to keep CU consumption predictable and avoid recurring spikes — e.g., staggering start times, splitting the data import range, or adjusting Spark pool/session settings? Given the long-running nature of this schedule (1 year), would Reserved Capacity be more cost-effective than Pay-as-you-go for this kind of consistent daily workload? How can I proactively monitor which specific notebook/session is driving the spike, beyond what's shown in the Capacity Metrics App? Based on this workload pattern (20 concurrent notebooks, daily schedule, each importing 3–10 years of historical data), what F-SKU size would you recommend as a starting point? Is there a rule of thumb for estimating capacity size based on concurrent Spark sessions and data volume? Any insights on capacity sizing, concurrency limits, or scheduling strategy would be very helpful. Thanks in advance!
Upcoming DP-700 Free Voucher Opportunities
Hello everyone, I am currently preparing for the DP-700: Microsoft Fabric Data Engineer Associate certification. I noticed that the recent 100% voucher campaign has ended. Could you please let me know if there are any upcoming official Microsoft/Fabric Community events or campaigns offering free DP-700 exam vouchers? Thank you.
Are Spark Jobs Supposed to Retry on Failure by Default?
My spark job in Fabric has the retry policy disabled (even if I enable it I can't set Max retry attempts to 0, this is the most disabled it gets). But the first time my .py file hits a "raise Exception", the job just restarts (the next time, it doesn't restart). I'm also pretty confident that sometimes after it retries, the entire log for the first pass is just gone. I only know b/c I have some messages being sent to an outside table, and what I see in there doesn't match up with what's actually in the job's log. Any ideas?Choosing the Right Way to Run Python in Microsoft Fabric
Fabric gives us several ways to run Python, and at first they can look overlapping. In this post, I share the practical decision model I use to choose the right option based on execution mode, compute engine, and data access path. If you are code-first and want fewer wrong turns when moving from exploration to production, this guide is for you.
How to set the maximum number of application attempts?
Hi there, I'm wondering if there is a way to control the maximum number of application attempts for a spark job submitted via the Spark Job Definition? When an application fails, you can see it makes two attempts, then fails completely. I want to set this to only be a single attempt (it's a spark streaming job, and I want the whole application to fail after the first attempt). You can see an example of this below: I have tried to set spark.yarn.maxAttempts via the Spark environment: And through the job's environment tab in the history server you can see the setting has made it through to the app: However, in the same tab you can also see the ID is set to 2 on the second attempt, so it keeps making two attempts then failing completely: Are there any other settings that might be controlling this? Thanks
