topic Re: Copy activity: Disable staging? in Pipelines

Copy activity: Disable staging?

cw88 — Mon, 24 Jul 2023 12:49:03 GMT

Hi,

i try to copy data from warehouse (view) to warehouse (table). Is there a way to disable staging?

I get the validation error "Direct copying data to Data Warehouse using Copy command is not supported for the specified source. Please enable staging." when i disable the option.

Staging seems to slow down copying a lot...

Thanks!

Re: Copy activity: Disable staging?

GraceGu — Tue, 25 Jul 2023 05:33:36 GMT

Direct copy without staging only applies certain scenarios, you can refer to How to configure Data Warehouse in copy activity in Data Factory in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

Re: Copy activity: Disable staging?

cw88 — Tue, 25 Jul 2023 11:36:14 GMT

Is there any chance that this will also be available for wh -> wh in the future?

Or is there another option with copy activity for my szenario? My source is a lakehouse view (bronze layer in lh tables), destination is a warehouse table (silver layer). Using copy activity has the advantage of logging, fault tolerance etc...

Re: Copy activity: Disable staging?

ajarora — Tue, 25 Jul 2023 19:32:32 GMT

Please submit an idea if it doesnt already exist, to support Copying from Lakehouse to Warehouse, without Staging.

Re: Copy activity: Disable staging?

cw88 — Wed, 26 Jul 2023 08:49:56 GMT

Idea is submitted: https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=0b0b7044-912b-ee11-a81c-000d3a0ec312

Re: Copy activity: Disable staging?

ManuMas — Wed, 18 Oct 2023 20:04:30 GMT

Probably my answer arrived a bit late. I have faced the same problem and the copy activity takes ages to complete.

My solution was to write a stored procedure that performs first a truncate and then the insert into table statement. I have inserted 220M of rows in 3 minutes using this method.

Re: Copy activity: Disable staging?

NickMcDermaid — Tue, 12 Dec 2023 07:22:21 GMT

It's very difficult to swallow the sales nonsense like "Lakehouse and Warehouse are made for each other". But you can't properly use a pipeline to ingest from Lakehouse to Warehouse. But you can from external ADLS gen2????

T-SQL COPY INTO actually works fine to import Lakehouse into Warehouse

I tried that first with very few issues.

Then I tried a pipeline and I'm having all kinds of hassles. Like needing to use staging. And getting parquet data type mapping issues which I'm pretty sure are due to needing this staing step.

Crazy thing is this is far more difficult that it ever used to be. You're going backwards people

Re: Copy activity: Disable staging?

Midway — Mon, 07 Oct 2024 21:28:44 GMT

hey there, having the same issue, how did you accomplish the stored procedure method?