Forum Discussion
Copy Data activity to load a SQL Server view into my Fabric Lakehouse.
I am using a Copy Data activity to load a SQL Server view into my Fabric Lakehouse. Until last week the pipeline always finished in about 5 minutes. Since yesterday it suddenly takes around 10 minutes, almost double the time.
Has anyone seen this kind of slowdown before? What could be the possible reasons, and how can I troubleshoot or fix it?
It is really hard to identify it from here remotely, but there is a way for you to do some checks:
Run the view directly in SQL ServerIf the query time has increased, the source is the problem. If the query runs normally, investigate the gateway or destination.
Compare the Copy Data run diagnostics
Check:
Data read throughput
Data write throughput
Duration of internal execution stages
This identifies whether the slowdown is coming from the source, transfer, or sink.
Check the gateway or SHIR machine
Look at CPU, RAM, and network rate during execution. Any resource pressure can slow down the transfer.
Run a small test copy
Try copying:
A small subset of rows
A direct table instead of a view
If the small test is slow, the destination is likely the issue. If the small test is fast, the view or data volume has changed.
Adjust Copy Behavior
Experiment with:
Parallelism settings
Data Integration Units
Enabling or disabling staging
So in general, to troubleshoot performance problems, focus on the parts of data transfer activity. Is it due to view performance?, Due to table/index performance which that view uses? Is the SQL Server where this view sits, having overloads? Is there something on the network? Is that related with the integration runtime machine that you are using? Is that the configuration that is being used in copy activity or is that something related with the destination (in this case fabric) => Like you have to run Optimize and vacuum on delta files etc. etc. Check each of these points and do tests and try to encapsulate the problem as much as possible.
Hope that helps
Onur
😊 If this post helped you, feel free to give it some Kudos! 👍
✅ And if it answered your question, please mark it as the accepted solution.
4 Replies
Hi Shubham_rai955 ,
Yes, this kind of slowdown does happen from time to time. We’ve seen similar cases where a pipeline that ran fast for weeks suddenly started taking much longer without any change on our side.
Most of the time this turns out to be related to either the source system or the Fabric side load, rather than the pipeline configuration itself. If the SQL Server view is now returning more rows, has changed internally, or is hitting a table that grew in size, the copy will naturally take longer even though the pipeline looks the same. It’s also worth checking if anything changed in the view definition recently.
On the Fabric side, capacity load can also impact runtime. If your workspace is sharing capacity with other jobs or reports, a busy period can slow down copy activities without throwing any errors. We’ve had cases where running the same pipeline at a different time of day completed much faster.
Easiest things to check first are whether the row count from the SQL Server view has increased, whether the view was modified, and whether anything else heavy was running on the Fabric capacity at the time. Rerunning the pipeline at a quieter time can also tell you quickly if it’s a capacity issue.
In our experience, these slowdowns are usually temporary or data-growth related rather than something broken in the pipeline itself.
– Gopi Krishna
It is really hard to identify it from here remotely, but there is a way for you to do some checks:
Run the view directly in SQL ServerIf the query time has increased, the source is the problem. If the query runs normally, investigate the gateway or destination.
Compare the Copy Data run diagnostics
Check:
Data read throughput
Data write throughput
Duration of internal execution stages
This identifies whether the slowdown is coming from the source, transfer, or sink.
Check the gateway or SHIR machine
Look at CPU, RAM, and network rate during execution. Any resource pressure can slow down the transfer.
Run a small test copy
Try copying:
A small subset of rows
A direct table instead of a view
If the small test is slow, the destination is likely the issue. If the small test is fast, the view or data volume has changed.
Adjust Copy Behavior
Experiment with:
Parallelism settings
Data Integration Units
Enabling or disabling staging
So in general, to troubleshoot performance problems, focus on the parts of data transfer activity. Is it due to view performance?, Due to table/index performance which that view uses? Is the SQL Server where this view sits, having overloads? Is there something on the network? Is that related with the integration runtime machine that you are using? Is that the configuration that is being used in copy activity or is that something related with the destination (in this case fabric) => Like you have to run Optimize and vacuum on delta files etc. etc. Check each of these points and do tests and try to encapsulate the problem as much as possible.
Hope that helps
Onur
😊 If this post helped you, feel free to give it some Kudos! 👍
✅ And if it answered your question, please mark it as the accepted solution.
Hi Shubham_rai955 ,
Thank you Ugk161610 , OnurOz for your inputs.I just wanted to check if the issue has been resolved on your end, or if you require any further assistance. Please feel free to let us know, we’re happy to help!
Thank you
Chaithra E.Hi Shubham_rai955 ,
May I ask if you have resolved this issue? Please let us know if you have any further issues, we are happy to help.
Thank you.
