Forum Discussion
aizhan_otp
1 year agoFrequent Visitor
Issue with data copy using pipeline from on premisses gateway
Hi everyone. I need help with the following issue. I have data gateway on another machine. I use this for refreshing dashboard. I have task to create process on Fabrics. So I copy data from ODBC C...
- 1 year ago
I suggest trying these steps:
- First, run the same query directly in a SQL tool to see how long it takes. If it's slow, try improving the query or adding indexes to speed it up.
- Adjust the timeout settings in your gateway, or try using a temporary storage method (staging) during the copy.
- Instead of copying everything at once, only copy new or changed data using a filter like `WHERE modified_date > last_loaded_date`.
- Break the data into smaller parts using batching or splitting by columns if possible.
- Consider using Dataflow Gen2 or a Data Pipeline with a Lakehouse as a temporary step, and turn on caching or staging options.
- Lastly, increase the number of retry attempts and give it more time before it times out.
Ilgar_Zarbali
Super User
1 year agoI suggest trying these steps:
- First, run the same query directly in a SQL tool to see how long it takes. If it's slow, try improving the query or adding indexes to speed it up.
- Adjust the timeout settings in your gateway, or try using a temporary storage method (staging) during the copy.
- Instead of copying everything at once, only copy new or changed data using a filter like `WHERE modified_date > last_loaded_date`.
- Break the data into smaller parts using batching or splitting by columns if possible.
- Consider using Dataflow Gen2 or a Data Pipeline with a Lakehouse as a temporary step, and turn on caching or staging options.
- Lastly, increase the number of retry attempts and give it more time before it times out.