User Configuration Issue - Fail to Update Tables in Pipeline

[vwynn23]

Im using a Notebook to run my sftp files but I receive this error message every morning, around 1am. When we rerun this in the morning around 7:30am-8:00am it runs fine. Has anyone else ran into this issue as well. Here is a screenshot of the error message as well as the pipeline that we created, it always fails on the VIP File, I'm assuming becuase thats the biggest file.  Any resolution helps thanks. 

[Anonymous]

Hi @vwynn23 
I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.
Thank you.

[Anonymous]

Hi @vwynn23 
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.

[Anonymous]

Hi @vwynn23 

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[Anonymous · Accepted solution]

Hi @vwynn23 

Yes, your options are to either reschedule the pipeline or increase capacity. Fabric doesn’t currently support automatic or time-based scaling (e.g., scale up for 2 hours and down again), so any capacity changes would need to be done manually via the Admin Portal or through PowerShell/REST API scripts.

If rescheduling works reliably, that’s usually the simpler option. But if large files are a regular load, increasing capacity could provide more consistent performance.

If this solution helps, please consider giving us Kudos and accepting it as the solution so that it may assist other members in the community
Thank you.

[vwynn23]

It looks like its isnt a capacity issue, I'm going to use a wait clause and run it at 3:00am instead of the inital 1:00am and hopefully that works. I will keep you posted. Thanks 

[Anonymous]

Hi @vwynn23 
Thank you for reaching out microsoft fabric community forum.

Based on your description and the screenshots provided, it appears that the notebook consistently fails around 1:00 AM when processing the VIP file—likely due to its size—yet succeeds when rerun later in the morning. This behavior is commonly observed when capacity constraints or transient resource throttling occur during scheduled runs, especially on shared or lower-tier capacities (such as F SKUs). To mitigate this, we recommend implementing retry logic in your pipeline to automatically handle transient failures. You can also consider adjusting the pipeline schedule to a later time (e.g., 2:00–3:00 AM) when system load may be lower. Additionally, reviewing your workspace's capacity utilization and testing the pipeline on a higher or dedicated capacity can help confirm if the issue is resource-related. For further guidance, please refer to the Optimize Fabric Capacity  documentation. 

If this solution helps, please consider giving us Kudos and accepting it as the solution so that it may assist other members in the community
Thank you.

[vwynn23]

Thanks for your response, so my options are to either increase the capacity or have it later schedule? If we were to increase the capacity is it possible to scale the capacity up for 2 hours automatically and then downsize again?