Forum Discussion
aizhan_otp
1 year agoFrequent Visitor
Issue with data copy using pipeline from on premisses gateway
Hi everyone. I need help with the following issue. I have data gateway on another machine. I use this for refreshing dashboard. I have task to create process on Fabrics. So I copy data from ODBC C...
- 1 year ago
I suggest trying these steps:
- First, run the same query directly in a SQL tool to see how long it takes. If it's slow, try improving the query or adding indexes to speed it up.
- Adjust the timeout settings in your gateway, or try using a temporary storage method (staging) during the copy.
- Instead of copying everything at once, only copy new or changed data using a filter like `WHERE modified_date > last_loaded_date`.
- Break the data into smaller parts using batching or splitting by columns if possible.
- Consider using Dataflow Gen2 or a Data Pipeline with a Lakehouse as a temporary step, and turn on caching or staging options.
- Lastly, increase the number of retry attempts and give it more time before it times out.
V-yubandi-msft
Community Support
1 year agoHi aizhan_otp ,
May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.
Thank you.