Forum Discussion
copy task succeeds before actually completing
Hi dzav ,
From what I’ve found, this issue might be related to a delay in data availability via the SQL endpoint rather than a bug in the copy task itself. You can look at this document for a further study: How to copy data using copy activity - Microsoft Fabric | Microsoft Learn
Using a wait task, as you’ve done, is a practical workaround. Another approach could be to implement a polling mechanism in your pipeline to check for data availability before proceeding to the next step. This can help ensure that the data is fully populated and ready for the subsequent tasks.
Best Regards
Yilong Zhou
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
I'm seeing the same thing in an integration scenario.
Pipeline Steps:
1) Invoke Pipeline: Multiple Copy Data activities, where each load from Amazon RDS tables to lakehouse tables
2) Copy Data: query and transform lakehouse table data via data warehouse, destination to same warehouse
3) Notebook: API POST to non-Fabric cloud platform
Step 2 is starting before all the data from Step 1 is available in the lakehouse tables. Step 1 has the "Wait on completion" setting checked on the invoke Pipeline activity, and I have confirmed that Step 2's Start time is indeed after Step 1's completion time per the Monitoring Hub.
It appears that Copy tasks writing to Lakehouse tables get marked as Completed prior to the data being available. When Copy Destination = Lakehouse Table, there should be an activity setting that does the polling built-in, rather than requiring users to add complexity to pipelines with Until activities and custom polling.