Copy Multiple tables from Azure SQL Via Vnet gateway to Lakehouse

[Vishnu1995]

Hi All,

Fabric newbie here. I do have experience in Azure data services (ADF, Databricks, SQL, ADLS etc)

I have a requirement where I have to copy multiple tables from a Azure SQL Database via a Vnet data gateway to fabric lakehouse.

1. Is there a way to copy it directly using Fabric notebooks? Is it possible to connect notebook to Azure SQL DB via Vnet data gateway?

2. If not, I was thinking about storing the list of tables in a control table and then looping it in Fabric data factory. but as of now Vnet data gateway is not supported in DF.

I know it might be possible using dataflow, but that is a complete new tech to me and I would need some guidance.

Please advise on how to proceed with with scenario. Thank you.

[Vishnu1995 · Accepted solution]

Hi everyone,

I found a solution by creating a managed private endpoint for the Azure SQL database in fabric, and then reading from the database using notebook.

Refer here for the documentation: https://learn.microsoft.com/en-us/fabric/security/security-managed-private-endpoints-create

[DataBard]

1. As far as I've tried, I haven't found a way to use a Fabric notebook with a data gateway in general. Most libraries I've seen assume that your notebook code is already running in the network that your database resides in.
2. Using Data Pipelines with a VNet Gateway is more likely to work for you, as this feature is currently in preview. According to the roadmap, this feature should go live in Q1 of 2025 (roadmap here: https://learn.microsoft.com/en-us/fabric/release-plan/data-factory#data-pipeline-support-vnet-gatewa...).

While dataflows are able to make use of vnet gateways, I would advise against them if this solution is for enterprise use. Dataflow ownership and their performance compared to other options like data pipelines leaves a lot to be desired.

You might also want to keep an eye on Mirroring of On-Prem SQL, as I'm hoping that once you can mirror from On-Prem SQL, you can then Mirror from Azure SQL DB via a gateway as well.

Hopefully this helps! If it does, be sure to accept your answer so others in the community can learn from your quesion.

[v-sgandrathi]

Hi @Vishnu1995,

Thank you for being a part of the Microsoft Fabric Community.

I wanted to check if you had the opportunity to review the information provided by @DataBard  Please feel free to contact us if you have any further questions.

If my response has addressed your query, please Accept it as a solution and give a Kudos so other members can easily find it.

Regards,
Sahasra.

[v-sgandrathi]

Hi @Vishnu1995,

May I ask if you have gotten this issue resolved?

If it is solved, please mark the helpful reply or share your solution and accept it as solution, it will be helpful for other members of the community who have similar problems as yours to solve it faster.

Thanks for using Microsoft Fabric Community Forum.