Forum Discussion

dzav's avatar
dzav
Advocate III
2 years ago
Solved

copy task succeeds before actually completing

I've got a Fabric pipeline with the following 3 steps: 1) Copy task: load data from bronze lakehouse files to bronze delta table 2) Stored procedure task: load data from bronze delta table (via sql...
Need Help
pipelines
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi dzav ,

    From what I’ve found, this issue might be related to a delay in data availability via the SQL endpoint rather than a bug in the copy task itself. You can look at this document for a further study: How to copy data using copy activity - Microsoft Fabric | Microsoft Learn

     

    Using a wait task, as you’ve done, is a practical workaround. Another approach could be to implement a polling mechanism in your pipeline to check for data availability before proceeding to the next step. This can help ensure that the data is fully populated and ready for the subsequent tasks.

     

     

     

    Best Regards

    Yilong Zhou

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi dzav ,

From what I’ve found, this issue might be related to a delay in data availability via the SQL endpoint rather than a bug in the copy task itself. You can look at this document for a further study: How to copy data using copy activity - Microsoft Fabric | Microsoft Learn

 

Using a wait task, as you’ve done, is a practical workaround. Another approach could be to implement a polling mechanism in your pipeline to check for data availability before proceeding to the next step. This can help ensure that the data is fully populated and ready for the subsequent tasks.

 

 

 

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.