topic Re: Best way to access external Azure Storage Account using SPN and reachable through a private endp in Data Engineering

Best way to access external Azure Storage Account using SPN and reachable through a private endpoint

MrCalm — Mon, 09 Sep 2024 19:42:02 GMT

I'm helping a customer on a Fabric POC, and we are currently struggling to find the right approach to access an external Azure Storage Account (external meaning that it is owned by another organisation). The Storage account has configured networking security, where we can be provided access either through IP whitelisting or Managed Private Networks/Endpoints. The authentication is to be handled via SPNs.

We have successfully configured a Azure Data Factory Pipeline (NOT Fabric Data Pipeline), that uses a Managed Virtual Network Integration Runtime and a Managed Private Endpoint to access the storage account securely. As far as I can tell the same option is not available in Fabric Pipelines.

A Pure Fabric solution is desired.

Please advise on the best way forward.

Thanks in advance, René

Re: Best way to access external Azure Storage Account using SPN and reachable through a private endp

Anonymous — Tue, 10 Sep 2024 02:01:41 GMT

HI @MrCalm,

In fact, fabric data pipeline also support azure storage account. You can choose data source 'azure blob' and choose ‘SPN authentication’ to use your azure account and SPN.

BTW, are you worked with the fabric trial or fabric capacity? (free trial is hosted in a virtual environment)
They may cause some issues when you try to interaction between free trial and payment environment and products.

Regards,

Xiaoxin Sheng

Re: Best way to access external Azure Storage Account using SPN and reachable through a private endp

MrCalm — Tue, 10 Sep 2024 07:24:38 GMT

Hi Xiaoxin Sheng,

We are currently on a Trial capacity, and hoped that we could implement an end-to-end example before deciding to buy an actual capacity.

My primary concern is the networking security features of Fabric and the limitations and considerations that these feature pose on the available types of workloads (i.e. Notebooks [Spark clusters] or Data Factory/Pipeline).

To me Trusted Workspace Access seems to be the least restrictive of the networking security features, as we would be able to create a short-cut in a lakehouse to the desired folder i ADLSg2. Trusted Network Access seems to be limited to real Fabric Capacities and not trail.

Does Trusted Workspace Access work between tenants, or is it limited to same tenant?

I have spend quite some time getting the Managed Private Endpoint feature to work against the ADSLg2. I have tried using Notebooks with ContainerClient and BlobClient (python lib azure.staorage.blob), but also generating Short-cuts, but both were unsuccessful.

The struggles with managed private endpoints lead me to think that there may be other options for this.

Hope this clarifies my question a bit more,

Regards, René

Re: Best way to access external Azure Storage Account using SPN and reachable through a private endp

Anonymous — Wed, 11 Sep 2024 06:17:36 GMT

HI @MrCalm,

AFAIK, the trusted workspace means the same tenant workspace. For work with the external tenant contents, they will have more limits on security and usage. Also across tenant may hard to manage and will affect the performance due to remote network connection.

You can also take a look at the following document about external sharing in fabric to know more about these:

External data sharing in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
Regards,

Xiaoxin Sheng