topic Re: Staging for copying data from external source to fabric in Pipelines

Staging for copying data from external source to fabric

GuillermoBCN — Mon, 22 Apr 2024 12:03:38 GMT

Hello experts,

I need your assistance on documentation on how to build a staging for a data ingestion from a On Premise database (SAP HANA type) to Fabric. With HANA views it is possible to create dataflows without the need for Staging, but we want to access database tables without using views, and for that, in copy activity, the enable staging flag is mandatory as HANA is external.

Since on the options of staging i see i can choose between lakehouse and blob storage, i decided to use the latter since we have an Azure resource group with storage account and i have created a container for this activity.

Is there some documentation we could use for this purpose? i.e., what configurations will our Basis team to perform on the Azure side. As i was unable to use my Service principal credentials i would assume that service principals needs some permissions or roles on working with Azure blob storage.

Thanks in advance!

Guillermo

Re: Staging for copying data from external source to fabric

amitchandak — Tue, 23 Apr 2024 03:46:32 GMT

GuillermoBCN , We have been using SAP HANA for some time and recently, I was able to uncheck the staging option in dataflow gen2. Also, the pipeline has a connector with has HANA connector and I have not used staging for that

Re: Staging for copying data from external source to fabric

Anonymous — Tue, 23 Apr 2024 10:28:05 GMT

Hi GuillermoBCN ,

Thanks for using Fabric Community.
"Supporting OneLake as staging for on-premise to DW is on the roadmap. In the mean time, it's better to simply load to a Lakehouse in the same workspace as the Warehouse. Those lakehouse tables will be accessable from the SQL endpoint, and you can load from there to the Warehouse if you want to."

Hope this is helpful. Please let me know incase of further queries.

Re: Staging for copying data from external source to fabric

Anonymous — Thu, 25 Apr 2024 07:50:49 GMT

Hi GuillermoBCN

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .

Thanks

Re: Staging for copying data from external source to fabric

Anonymous — Fri, 26 Apr 2024 10:32:58 GMT

Hi GuillermoBCN ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .

Thanks