Copy Job from one Lakehouse to Another runs forever

[JOAOLBMARTINS]

Hey,

I have created a copy job that gets the data from one warehouse and drops into another.

However, it's running indefinetely. 

Any ideas on how to debug this?

Thanks!

[v-ssriganesh · Accepted solution]

Hello @JOAOLBMARTINS,
Please consider raising a Microsoft support ticket for further investigation. You can explain all the troubleshooting steps you have taken to help them better understand the issue.

You can create a Microsoft support ticket with the help of the link below:
https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket

[JOAOLBMARTINS]

Even though this is not the ideal solution I'll have to take it as an approach.

The issue doesn't seems to be anything done on the user side and more on the backend of how microsoft fabric works.

[v-ssriganesh]

Hello @JOAOLBMARTINS,

We are following up once again regarding your query. Could you please confirm if the issue has been resolved through the support ticket with Microsoft?

If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. 

Thank you.

[v-ssriganesh]

Hi @JOAOLBMARTINS,

Thank you for posting your query in the Microsoft Fabric Community Forum, and thanks to @Vinodh247 for sharing valuable insights.

Could you please confirm if your query has been resolved by the provided solutions? This would be helpful for other members who may encounter similar issues.

Thank you for being part of the Microsoft Fabric Community.

[Vinodh247]

If your copy job from one Lakehouse to another in Microsoft Fabric is running indefinitely, it is likely due to issues related to data volume, schema mismatch, or compute resource limitations. Start by checking the pipeline monitor in the Data Factory to see whether the copy activity is actually progressing or stuck in a queued or long-running state. Review the activity run logs to identify if any data is being read or written. Ensure the source table is accessible and that the target Lakehouse has the correct schema and write permissions. Large datasets or unpartitioned tables can also cause prolonged runtimes, especially on smaller Fabric SKUs like F2 or F4.

Better, to isolate the issue, try copying a small sample dataset or test the read/write operations through a notebook using Spark. This can help determine whether the problem lies with the pipeline configuration or the data itself. Finally, check for capacity bottlenecks and consider scaling up or freeing compute resources. If none of these steps help, export the activity logs and raise a support ticket with Microsoft for deeper investigation.

Please 'Kudos' and 'Accept as Solution' if this answered your query.

[JOAOLBMARTINS]

yeah, the problem is that at beginning, the first few shots it runs fine. Then the other day it simply stops to run and get stuck for hours.

This should be a simple truncate/insert statement data between two different lakehouses.

Monitor doesn't give much details about what is going on behind the scenes.