How pipeline copy activity works in Fabric?

[ravikiransrini]

Hi everyone,

I am copying data from different sources to the lakehouse and warehouse in Fabric.

When I am using pipeline copy activity, it asks me to enable staging, if the source is not ADLS Gen2 or Azure Storage account.

Why is this the case even if the copy is from the lakehouse to the warehouse?

I want to know how the pipeline copy activity works here. Does it use the COPY INTO T-SQL command? Because the COPY INTO command too supports only ADLS Gen2.

Then how does dataflow gen2  seamlessly copy data to the warehouse without any staging layer?

Finally, why is this restriction only for warehouses and not lakehouses?

Thanks for reading my question.

I am referring to these sources:

https://learn.microsoft.com/en-us/sql/t-sql/statements/copy-into-transact-sql?view=fabric

https://learn.microsoft.com/en-us/fabric/data-warehouse/ingest-data#supported-data-formats-and-sourc...

https://learn.microsoft.com/en-us/fabric/data-warehouse/ingest-data-pipelines

[v-kathullac]

Hi @ravikiransrini  ,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for the issue worked? or Let us know if you need any further assistance?
If our response addressed, please mark it as Accept as solution and click Yes if you found it helpful.

Regards,

Chaithanya.

[v-kathullac]

Hi @ravikiransrini  ,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for the issue worked? or Let us know if you need any further assistance?
If our response addressed, please mark it as Accept as solution and click Yes if you found it helpful.

Regards,

Chaithanya.

[v-kathullac]

Hi @ravikiransrini  ,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for the issue worked? or Let us know if you need any further assistance?
If our response addressed, please mark it as Accept as solution and click Yes if you found it helpful.

Regards,

Chaithanya.

[ravikiransrini]

I was looking for a more in-depth product answer. You have only pasted what is already available on the internet.

[suparnababu8]

Hi @ravikiransrini 

Thanks for your response. Yes, the official microsoft documentation which is presented on the internet telling the same.  Please read below threads for more clarification.

Move and transform data with dataflow and data pipelines - Microsoft Fabric | Microsoft Learn

Dataflow Gen2 data destinations and managed settings - Microsoft Fabric | Microsoft Learn

Solved: How Does "Enable Staging" Work in Microsoft Fabric... - Microsoft Fabric Community

Configure Data Warehouse in copy activity in Data Factory in Microsoft Fabric - Microsoft Fabric | M...

Still if you need more information, Pls raise asupport ticket How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn . 

How to raise support ticket pls read this tag - Power BI and Fabric support overview - Power BI | Microsoft Learn

Feel free to reachout any quries

Thank you once again!!

Did I answer your question? Mark my post as a solution!

Proud to be a Super User!

[suparnababu8 · Accepted solution]

Hi @ravikiransrini 

1) Why staging required for other storages (with out ADLS GEN2 & Azurfe Storage)?

Ans: The COPY statemant is the primary way to ingest data into warehouse tables. Data warehouse COPY command direectly supports Azure Blob Storage and ADLS Gen2 as source data stores. 
        If your source data store and format isn't originally supported by a COPY command, use the Staged copy by using the COPY command feature instaed. It automatically converts the data into a COPY command compatible format, then calls a COPY command to load data into Data Warehouse.

Staging activity: When your source data is not natively compatiible with COPY command, enable data copying via an interim stagign storage. In this case, the service automatically converts the data to meet the data format requirements of COPY command. Then it invokes COPY command to load data into Data Warehouse. Finally, it cleans up your temporary data from the storage

Refer this thread for more clarification; Configure Data Warehouse in copy activity in Data Factory in Microsoft Fabric - Microsoft Fabric | M...

2) Does it use the COPY INTO T-SQL command?

Ans: Yes you are correct. This command allows fast data loading but only works with ADLS Gen2 and Azure Blob Storage.  Refer this thread for more clarification; How to use COPY Into | Microsoft Community Hub

3) why is this restriction only for warehouses and not lakehouses?

Ans: Yes, warehouses have restricttions because they use the COPY INTO command, which requiires specific data formats. Lakehouses use different methods for data ingestion and do not have these same limitations.

Refer this thread for more clarification: Dataflow Gen2 data destinations and managed settings - Microsoft Fabric | Microsoft Learn

Thank you!!!

Did I answer your question? Mark my post as a solution!

Proud to be a Super User!