Issue with data copy using pipeline from on premisses gateway

[aizhan_otp]

Hi everyone.

I need help with the following issue. I have data gateway on another machine. I use this for refreshing dashboard. I have task to create process on Fabrics. So I copy data from ODBC Connection (Postgre DataBase) to LakeHouse or similar Data WareHouse on Fabric. I created 2 pipelines to copy data from two different tables in the database.

When I try to copy data to create a data pipeline for the second table (which consists of 25 million rows from a query, with new data added daily), I get the following error: Web request timeout: A task was canceled. Activity ID: 980d3d38-7dad-432e-9531-8a3dbfa4823d Retry to get column mappings.

I don't understand reasons why I without copy data for 1st table, but on 2nd table I have problems. 

Could someone please assist me with this issue?

[V-yubandi-msft]

Hi @aizhan_otp ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

 

Thank you.

[V-yubandi-msft]

Hi @aizhan_otp ,

Thank you for engaging with the Microsoft Fabric Community. It seems the issue might be due to several factors. If your dataset is large, schema inference could be taking too long, or slow queries might be causing delays. Sometimes, data reads exceed timeout limits, or the on premises gateway lacks sufficient resources.

@Ilgar_Zarbali , has shared good  suggestions. Additionally, I have included a few more ideas that might help resolve the issue.

1. Your second table has 25 million rows, which might be causing query timeouts. Try filtering out unnecessary columns to reduce the load.
2. Check the gateway logs to identify performance bottlenecks and adjust timeout settings if needed.
3. Web requests might be timing out too soon. Increase the timeout settings in your pipeline configuration to handle large data transfers.
4. Ensure your on premises data gateway has enough CPU and memory, a stable internet connection, and the latest updates installed.

Helpful Reference :

Solved: Re: Understanding Timeout in Activities - Microsoft Fabric Community

Thnaks for your response @Ilgar_Zarbali .

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly

[Ilgar_Zarbali · Accepted solution]

I suggest trying these steps:

1. First, run the same query directly in a SQL tool to see how long it takes. If it's slow, try improving the query or adding indexes to speed it up.
2. Adjust the timeout settings in your gateway, or try using a temporary storage method (staging) during the copy.
3. Instead of copying everything at once, only copy new or changed data using a filter like `WHERE modified_date > last_loaded_date`.
4. Break the data into smaller parts using batching or splitting by columns if possible.
5. Consider using Dataflow Gen2 or a Data Pipeline with a Lakehouse as a temporary step, and turn on caching or staging options.
6. Lastly, increase the number of retry attempts and give it more time before it times out.