Forum Discussion
Copy Job Actitivity Error
Hi Anonymous ,
You're correct that the new Copy Job feature in Microsoft Fabric currently requires using temporary storage when copying data from a local SQL Server to a warehouse. This is because the feature relies on staging the data before it can be ingested into the warehouse.
The ability to copy data directly to the warehouse without using temporary storage is on the roadmap. While there's no specific ETA yet, it's something that the internal team is working on.
In the meantime, a common workaround is to first load the data into a cloud Lakehouse and then perform another copy activity to move it to the warehouse. This ensures that the data is staged properly before final ingestion.
You can also look at this topic: Solved: On Prem SQL Server to Fabric Warehouse via Copy Da... - Microsoft Fabric Community
Best Regards
Yilong Zhou
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Hi,
I can confirm Yilongs recommendation. Lakehouse provides you the ability to directly write into destination without staging. Currently, a lot of nice features are not available in data warehouse anyway. I saw that quite a lot of features are coming to data warehouse in Q1 2025, therefore suggesting to work with lake house and even benefit from direct lake, etc.
Regards,
Oktay
Did I answer your question? Then please mark my post as the solution.
If I helped you, click on the Thumbs Up to give Kudos.
Thanks for the reply.
According to what you've explained, if I use the new Copy Job feature to copy data from a local SQL Server (using a gateway) and having a warehouse in Fabric as the destination, this feature not allow me to copy directly to the warehouse. I understand that the feature is in preview.
Do you know if we will have this possibility of copying without using temporary storage?
As an alternative, you suggest that I use Copy Data in Pipeline. I had tried this method before, but I don't think I had incremental copying and it always append data, which didn't work for me.
Thanks you
Best regards
Orlando
Hi Anonymous ,
You're correct that the new Copy Job feature in Microsoft Fabric currently requires using temporary storage when copying data from a local SQL Server to a warehouse. This is because the feature relies on staging the data before it can be ingested into the warehouse.
The ability to copy data directly to the warehouse without using temporary storage is on the roadmap. While there's no specific ETA yet, it's something that the internal team is working on.
In the meantime, a common workaround is to first load the data into a cloud Lakehouse and then perform another copy activity to move it to the warehouse. This ensures that the data is staged properly before final ingestion.
You can also look at this topic: Solved: On Prem SQL Server to Fabric Warehouse via Copy Da... - Microsoft Fabric Community
Best Regards
Yilong Zhou
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.