Fabric pipeline failing in concurrent runs

[girishtharwani2]

Hi,

I am running fabric pipeline which points to snowflake and when object count is more than 20, the fabric pipeline is failing and not even hitting snowflake.

Can someone please assist.

Thanks,

Girish

[girishtharwani2]

Same pipeline with 50 concurrency works in synapse however not in fabric.

Fabric also working on 128 capacity however still failing.

Anything else that I can try ?

Thanks,

Girish

[v-lgarikapat]

Hi @girishtharwani2 ,

Thanks for the detailed response

You can try the workaround below.

1.Throttle concurrency to 10–20 and gradually scale up. Fabric may not handle 50 concurrent threads the same way Synapse does.

 2.Add retry logic and increase timeout settings in the Copy activity or notebook.

 3.Split the pipeline into smaller batches using a dynamic ForEach controller.

4. Monitor Spark session logs and capacity metrics in the Fabric Monitoring Hub to identify queuing or memory pressure.

5.Temporarily disable High Concurrency Mode if enabledsome users report it causes failures even when not actively used

Configure high concurrency mode for notebooks in pipelines - Microsoft Fabric | Microsoft Learn

Known issue - Pipelines fail with missing capacity error - Microsoft Fabric | Microsoft Learn

If you're still facing the same issue,
I suggest raising a support ticket so the support team can assist you in resolving the problem.
Please refer to the link below for instructions on how to raise a support ticket:

How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

If this post helped resolve your issue, please consider marking it as the Accepted Solution.
This not only acknowledges the support provided but also helps other community members find relevant solutions more easily.

We appreciate your engagement and thank you for being an active part of the community.

Best regards,
LakshmiNarayana

[v-lgarikapat]

Hi @girishtharwani2 ,

Thanks for reaching out to the Microsoft fabric community forum.

@Srisakthi 

Thanks for your prompt response

@girishtharwani2 , 

Based the screenshot you shared 

It looks like your pipeline is failing due to a request being blocked by the upstream service. Based on the error message in the image you uploaded, the request is blocked until 6/17/2025 at 12:41:52 PM (UTC). This suggests that the issue might be related to rate limiting or a restriction imposed by the upstream service

    Below are a few troubleshooting steps to check

1. Concurrency Limits: Snowflake has concurrency restrictions. If your Fabric pipeline is making too many requests simultaneously, this could trigger a block.
2. Fabric Configuration: Ensure that your Fabric pipeline settings allow handling more than 20 objects in a batch. If there’s a setting limiting concurrency or object count, try adjusting it.
3. Retry Strategy: If the request is temporarily blocked, implement a retry mechanism to avoid overwhelming the service.
4. Check Snowflake Logs: Since your request isn’t even hitting Snowflake, verify if there are any settings preventing Fabric from reaching Snowflake under high load.
5. Service Limits & Quotas: Review the API/service quotas to ensure you aren’t exceeding limits.

Configure Snowflake in a copy activity - Microsoft Fabric | Microsoft Learn

Solved: Re: Unable to connect to Snowflake from Microsoft ... - Microsoft Fabric Community

Snowflake connector overview - Microsoft Fabric | Microsoft Learn

If this post helped resolve your issue, please consider the Accepted Solution. This not only acknowledges the support provided but also helps other community members find relevant solutions more easily.

We appreciate your engagement and thank you for being an active part of the community.

Best regards,
LakshmiNarayana.

[girishtharwani2]

I’ve tried all the suggested steps, but unfortunately, nothing has worked so far. I’ve set the object count to 40, as it's a business requirement.

Additionally, I'm not using any API in this process.

The retry strategy hasn’t been effective either, since the service typically remains unavailable for over 30 minutes once the error occurs.

Thanks,

Girish

[anon97242]

I have had this issue and it seems it is related to capacity size number of vcores. Try reducing concurrency to less than the number Vcores you have in your capacity. 2-3, and see if it resolves the issue. You may also have to wait some time before trying again.

[v-lgarikapat]

Hi @girishtharwani2 

Thanks for your prompt response

@anon97242  in addiotion to @girishtharwani2  Added a learning document which can help you resolve the issue

Concurrency limits and queueing in Apache Spark for Fabric - Microsoft Fabric | Microsoft Learn

(1) Solved: Error code 2108 : Data Factory - Microsoft Fabric Community

If you're still facing challenges, feel free to let us know we’ll be glad to assist you further.

Looking forward to your response.

Best Regards,

Lakshmi Narayana

[v-lgarikapat]

Hi @girishtharwani2  ,

If your issue has been resolved, please consider marking the most helpful reply as the accepted solution. This helps other community members who may encounter the same issue to find answers more efficiently.

If you're still facing challenges, feel free to let us know we’ll be glad to assist you further.

Looking forward to your response.

Best regards,
LakshmiNarayana.

[girishtharwani2]

We use F128 which has sufficient vcores.

[Srisakthi]

Hi @girishtharwani2 ,

Can you check if there is any rules applied at snowflake side. There could be some settings applied at Snowflake side. Like, MAX Concurrency for warehouse, max data size can be fetched .

Regards,

Srisakthi

[girishtharwani2]

There is no such rule built on snowflake.