Is it possible to configre vnet data gateway access for Airflow DAGs hosted in Fabric?

[Strebor]

We are trying to migrate our Airflow DAGs from a Kubernetes deployment that has vnet access for its DAGs, to Fabric's Airflow implementation.

We have the vnet added to the Fabric setup already and are able to directly access our VM-hosted SQL Servers in standard copy and pipeline tasks. When setting up connections to the SQL server, we have to make sure to select the vnet as data data gateway dropdown for this to work.

We want to do the same in Airflow - but this doesn't seem to be possible - i.e. the DAGs inside the Airflowjob do not seem to be routed via the Data Gateway and we cannot see any configuration option where we can select the vnet in the same way as we can for other Fabric actions.

Is this a known limitation for the Apache Airflow hosting in Fabric?

[n0elleli]

Hi @Strebor, 

Would love to learn more about your Airflow requirements. Please shoot me an email at noelleli@microsft.com. 

[Strebor]

Hi - I thought I did explain the requirement? it's quite simple.

If we want to run managed Airflow in Azure (i.e. not via a Kubernetes deployment as described here)

We cannot use managed Airflow in Azure Data Factory any more as it now tells us to use Fabric as you can see here:

In Fabric, we can add the vnet to access our VM-hosted SQL server instances as you can see here:

And we can use this vnet for normal Copy Jobs or Pipelines as you can see here:

But there is no option to use these in Airflow Jobs - I would expect there to be some way to specify the vnet gateway in the Airflow environment setup here:

Therefore we cannot use managed Airflow in either ADF or Fabric to access our SQL Server instances that run in VMs in Azure.

My requirement is to be able to use Airflow Jobs in Fabric to access our SQL Server instances via Azure vnet integration. Is this clear enough? 

[n0elleli]

Understood! We have this feature on our roadmap. 

[Strebor]

Wonderful - thank you... is there an ETA for this?

(also did have a look here but could not find it?)

[n0elleli]

No ETA yet, we are working on building out our roadmap for the next semester. We should have a better idea at the end of Sept and will make sure to update the Fabric roadmap!

[v-prasare]

We are following up once again regarding your query. Could you please confirm ifyou have raised ideas with Microsoft?
If not please raise idea and upvote it. If we don’t hear back, we’ll go ahead and close this thread.
Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

Thank you for your understanding and participation.

[Strebor]

Hi - not raised with Microsoft - where do I raise this please?

Seems to be quite core piece of functionality missing from the Fabric managed Airflow implementation - surprised noone else has mentioned it. Seems like it is a regression as it was supported on ADF but now since ADF no longer supports Airflow, we have not option in Fabric.

[v-prasare]

you can raise it here  Microsoft Fabric Ideas.

[v-prasare]

Hi @Strebor ,

Thanks for reaching MS Fabric community support

please refer to below and let me know if this helps?

https://learn.microsoft.com/en-gb/answers/questions/2238062/azure-fabric-airflow-feature-in-vnet-int...

I’d encourage you to submit your detailed feedback and ideas via Microsoft's official feedback channels, such as the Microsoft Fabric Ideas.

Feedback submitted here is often reviewed by the product teams and can lead to meaningful improvement.

Thanks,

prashanth

MS Fabric community support

[Strebor]

Thanks: so the reply to this question shows me that this is NOT possible sadly.

As of now, Airflow jobs in Microsoft Fabric do not support VNet integration similar to Azure Data Factory’s Managed Airflow. There is currently no public documentation or feature availability confirming support for Private Endpoints, VNet peering, or Managed VNet integration within Microsoft Fabric's Airflow offering.

If secure communication with Azure VMs or other resources inside your private VNet is a requirement, you may consider:

• Using ADF Managed Airflow, which supports Managed VNet and private connectivity.
• Deploying Airflow on AKS, which gives full control over networking and access to resources in your VNet.
  
  

Reasons neither of these options work for us:

1. It is not possible for us to set up Managed airflow in ADF any more - it produces an error telling us to use Fabric. See here for announcement 
   
   
2. As stated in original post - we already use AKS but are trying to migrate away from it as it's very devops heavy (requiring management/setup skills after following this tutorial) - we just want a managed version that "just works" (aka Managed) - so were hoping Fabric would give us this. Seems not.