Forum Discussion
Copy activity: Disable staging?
Hi,
i try to copy data from warehouse (view) to warehouse (table). Is there a way to disable staging?
I get the validation error "Direct copying data to Data Warehouse using Copy command is not supported for the specified source. Please enable staging." when i disable the option.
Staging seems to slow down copying a lot...
Thanks!
7 Replies
Probably my answer arrived a bit late. I have faced the same problem and the copy activity takes ages to complete.
My solution was to write a stored procedure that performs first a truncate and then the insert into table statement. I have inserted 220M of rows in 3 minutes using this method.
hey there, having the same issue, how did you accomplish the stored procedure method?
Direct copy without staging only applies certain scenarios, you can refer to How to configure Data Warehouse in copy activity in Data Factory in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
Is there any chance that this will also be available for wh -> wh in the future?
Or is there another option with copy activity for my szenario? My source is a lakehouse view (bronze layer in lh tables), destination is a warehouse table (silver layer). Using copy activity has the advantage of logging, fault tolerance etc...
Please submit an idea if it doesnt already exist, to support Copying from Lakehouse to Warehouse, without Staging.
It's very difficult to swallow the sales nonsense like "Lakehouse and Warehouse are made for each other". But you can't properly use a pipeline to ingest from Lakehouse to Warehouse. But you can from external ADLS gen2????
T-SQL COPY INTO actually works fine to import Lakehouse into Warehouse
I tried that first with very few issues.
Then I tried a pipeline and I'm having all kinds of hassles. Like needing to use staging. And getting parquet data type mapping issues which I'm pretty sure are due to needing this staing step.
Crazy thing is this is far more difficult that it ever used to be. You're going backwards people
