topic Best practices to ensure network security for ADLSg2 connection from Fabric in Data Engineering

Best practices to ensure network security for ADLSg2 connection from Fabric

amaaiia — Mon, 22 Apr 2024 14:51:14 GMT

I've created an Azure Data Lake Storage gen2 connection from Fabric. Now, I want to configure ADLSg2 firewall to enable access only from Fabric. Which is the best practice to configure this access rules?

I'm using the connection to create a shortcut in Fabric. When all networks are enabled, it works. So now I want to secure the connection.

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

AndyDDC — Tue, 23 Apr 2024 07:04:01 GMT

Hi amaaiia are you using an F SKU for the capacity? If so then if you are running an F64 then you'll be able to create a private endpoint to the storage account when the storage account is disabled from public access.

https://www.serverlesssql.com/managed-private-endpoints-in-microsoft-fabric/

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

Anonymous — Tue, 23 Apr 2024 07:06:00 GMT

Hi amaaiia
Thanks for using Fabric Community.
You can use Workspace identity in this case.
Please refer to this link:
Workspace identity - Microsoft Fabric | Microsoft Learn

Hope this helps. Please let me know if you have any further questions.

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

amaaiia — Tue, 23 Apr 2024 07:14:46 GMT

Hi AndyDDC Anonymous ,

I'm using P1, so I have these features disabled. I showed Trusted Workspace Access new feature, but I can't implement it...

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

Anonymous — Tue, 23 Apr 2024 07:19:55 GMT

Apologies amaaiia
In order to implement Trusted workspace access we need to satisfy some prerequisites as below:

Trusted workspace access in Microsoft Fabric (preview) - Microsoft Fabric | Microsoft Learn

For creating a Workspace Identity you need F64 capacity.

Hence you will need a F64 capacity to enable firewall for ADLS Gen2 account.

Hope this helps. Please let me know if you have any further questions.

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

AndyDDC — Tue, 23 Apr 2024 07:26:23 GMT

Unfortunately Premium P SKUs are not supported when using networking security features, only F64 and above.

Would a viable alternative be to push data into OneLake?

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

amaaiia — Tue, 23 Apr 2024 08:16:13 GMT

I'm a bit confused with the difference between Fabric and Premium Capacities. I understand the "power" of them is the same, but what more differences are there betweee (for example) P1 and F64? Are there any more differences related to features, functionalities, needed user licenses?

Until now, I thought there were similar at all.

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

amaaiia — Tue, 23 Apr 2024 08:17:29 GMT

Precisely, I want to use shortcuts to write data into ADLSg2 from Fabric. I want to have copies of data in ADLSg2.

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

Anonymous — Tue, 23 Apr 2024 08:23:36 GMT

Hi amaaiia

Microsoft Fabric capacity is billed based on the amount of resources utilized. It is significantly cheaper than Power BI Premium capacity.

Differences between Fabric capacities and Power BI Premium capacities

Both Fabric capacities and Power BI Premium capacities provide Fabric features, but there are some key differences between the two:

Pricing: Fabric capacities are significantly cheaper than Power BI Premium capacities.
Features: Power BI Premium capacities include all of the features of Fabric capacities, plus some additional Power BI Premium features, such as advanced security and governance features.
Licensing: Fabric capacities can be licensed independently, while Power BI Premium capacities require a Power BI Premium license.

For using Microsoft Fabric, you’ll have to purchase F SKUs. Since you are paying for the capacity, this means that you need Fabric capacity to be able to leverage Fabric services.

Power BI P1 SKU is equivalent to F64 SKU (8 v-cores). The great thing about Fabric SKUs is the much lower entry point. Unlike Power BI P SKUs, where the entry point is P1 SKU, the entry point into the Fabric world (F2 SKU with 1/4 v-cores) will cost you very less. This is a huge difference and one of the best advantage of Fabric.

The best capacity for an organization will depend on the specific needs and requirements. If you are looking for a low-cost solution with all of the essential Fabric features, then a Fabric capacity is a good option. If you need access to the additional Power BI Premium features, or if you are already using Power BI Premium, then a Power BI Premium capacity is a better option.

You can refer to these links for more information:
Fabric Pricing

Power BI vs Fabric

Solved: Fabric F64 capacity vs. Power BI Premium P1 capaci... - Microsoft Fabric Community

Hope this helps. Please let me know if you have any further questions.

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

R1k91 — Tue, 23 Apr 2024 08:54:56 GMT

amaaiia maybe you can consider "to switch" to F capacity.

it'll happen in anycase next year...

Important update coming to Power BI Premium licensing | Microsoft Power BI Blog | Microsoft Power BI

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

amaaiia — Tue, 23 Apr 2024 10:52:17 GMT

About the table you've posted, it's a little bit confusing for me because the last column says Power BI Premium per capacity (includes Microsoft Fabric), and the capacity I have is Power BI Premium, so this table is not differentiating the P1 SKU I already have with my doubts with F64 SKU, it says there are the same.

On the other hand, you say:

Features: Power BI Premium capacities include all of the features of Fabric capacities, plus some additional Power BI Premium features, such as advanced security and governance features.

But what I have seen is that there are some features (such as Private Endpoints and Workspace indentities) that I'm not being able to create with P1 SKU capacity, they can only be used with F64 SKU or above. So, there are features not available for Premium Capacity. It would be interesting to have the list of features available for each kind of capacity, what can you perform with each one, etc. It's like there's not a place where all the features and functionalities are collected for each Capacity comparison.

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

Anonymous — Tue, 23 Apr 2024 17:40:32 GMT

Thanks for the feedback amaaiia

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

AndyDDC — Tue, 23 Apr 2024 21:41:07 GMT

Unfortunately Microsoft haven't published a list of Fabric features that are/are not supported in Premium SKUs. You'll need to look at the Considerations and Limitations area of any MS documentation when reading about features, especially when new features are announced.

Ultimately MS are deprecating Premium SKUs https://www.microsoft.com/en-us/licensing/news/power-bi-premium-sku-retirement#:~:text=For%20new%20customers%2C%20the%20Power,capacity%20until%20January%201st%2C%202025.

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

AndyDDC — Tue, 23 Apr 2024 21:43:36 GMT

Perhaps you could use the Lakehouse connector in Data Factory to move data from OneLake to ADLSgen2. Mindful of the fact that the data factory instance needs to be created in a managed vnet, which is more expensive.

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

amaaiia — Wed, 24 Apr 2024 06:15:27 GMT

Oh, so Premium per Capacity will dissappear. That's good to know.

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

amaaiia — Fri, 26 Apr 2024 09:28:45 GMT

Hi AndyDDC , Anonymous

I understand I can't use Workspace Identities for secure the access to ADLSg2 account through Fabric. So, isn't there another way to restrict access to my ADLSg2 account so that it only accepts traffic from my Fabric account? Does this mean that with Capacities below F64 (or premium Capacities) this process cannot be secured?

I really need to configure my storage account networking for that...

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

amaaiia — Fri, 26 Apr 2024 10:43:56 GMT

If I switch to F32 or lower I'll keep the same issue..

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

R1k91 — Fri, 26 Apr 2024 10:47:34 GMT

you said you're on a P1 and the corresponding Fabric SKU is the F64 that has the feature you need.

why are you thinking to switch to a lower capacity?

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

amaaiia — Fri, 26 Apr 2024 11:14:53 GMT

I am using a P1 Capacity because it's a Capacity that Microsoft has given to me to evaluate Fabric's functionalities and to do a POC. But when my licence expires, I have to evaluate which one suits me best, and looking at the Capacity metrics, my data flows and pipelines don't need the power an F64 provices. So I will probably consider downgrading from F64 to F16 or F32. That's why I need to know how to take my scenario to a lower license, and not need F64 or higher enabled functionalities.

Re: Best practices to ensure network security for ADLSg2 connection from Fabric

AndyDDC — Fri, 26 Apr 2024 17:22:46 GMT

You could configure a virtual network, assign the storage account to the vnet, and the create a vnet data gateway in power bi/fabric. Although I believe only dataflow Gen2 are supported.

i may test and blog about this very soon