Forum Discussion

aizhan_otp's avatar
aizhan_otp
Frequent Visitor
1 year ago
Solved

Issue with data copy using pipeline from on premisses gateway

Hi everyone. I need help with the following issue. I have data gateway on another machine. I use this for refreshing dashboard. I have task to create process on Fabrics. So I copy data from ODBC C...
data pipeline
lakehouse
  • Ilgar_Zarbali's avatar
    Ilgar_Zarbali
    1 year ago

    I suggest trying these steps:

    1. First, run the same query directly in a SQL tool to see how long it takes. If it's slow, try improving the query or adding indexes to speed it up.
    2. Adjust the timeout settings in your gateway, or try using a temporary storage method (staging) during the copy.
    3. Instead of copying everything at once, only copy new or changed data using a filter like `WHERE modified_date > last_loaded_date`.
    4. Break the data into smaller parts using batching or splitting by columns if possible.
    5. Consider using Dataflow Gen2 or a Data Pipeline with a Lakehouse as a temporary step, and turn on caching or staging options.
    6. Lastly, increase the number of retry attempts and give it more time before it times out.
V-yubandi-msft's avatar
V-yubandi-msft
Icon for Community Support rankCommunity Support
1 year ago

Hi aizhan_otp ,

Thank you for engaging with the Microsoft Fabric Community. It seems the issue might be due to several factors. If your dataset is large, schema inference could be taking too long, or slow queries might be causing delays. Sometimes, data reads exceed timeout limits, or the on premises gateway lacks sufficient resources.

 

Ilgar_Zarbali , has shared good  suggestions. Additionally, I have included a few more ideas that might help resolve the issue.

  1. Your second table has 25 million rows, which might be causing query timeouts. Try filtering out unnecessary columns to reduce the load.
  2. Check the gateway logs to identify performance bottlenecks and adjust timeout settings if needed.
  3. Web requests might be timing out too soon. Increase the timeout settings in your pipeline configuration to handle large data transfers.
  4. Ensure your on premises data gateway has enough CPU and memory, a stable internet connection, and the latest updates installed.


Helpful Reference :

Solved: Re: Understanding Timeout in Activities - Microsoft Fabric Community

 

Thnaks for your response Ilgar_Zarbali .

 

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly