Forum Discussion

nullabee's avatar
nullabee
New Member
1 day ago

Connect to an Azure SQL Server with no public access

We have an Azure SQL Server that doesn't permit public access. When accessing it from a Fabric workspace, it looks like we have, essentially, 2 options:

  • either setup private endpoints and configure these. This has the (huge) disadvantage that startup time for spark environments goes from 5 seconds to 3 minutes. That means every time I use a spark notebook (even when it doesn't actually use the endpoints), it warms up for 3 minutes, which is a lot of overhead (we historically have a lot of short running notebooks, each now facing this slowdown).
  • or use a data gateway (virtual or on premise). However, when we connect to this same SQL Server from a notebook (using jdbc), it appears that this connection does not pick up on the data gateway (I don't see how it could, anyway). So this means my notebooks cannot communicate to this SQL Server?

(Note: when accessing an ADLS Gen 2 storage (also with public access disabled), there is the option to allow access to it with the workspace identity. A similar option seems not to be present for Azure SQL Server (there is only on option to "Allow Azure Services to access this", but that grants way too much access and probably doesn't affect Fabric connections)?)

Can anybody enlighten me on what is the right approach for this scenario?

So I hope you can tell me:

  • either you inform me that private endpoints are the way to go, and then have a solution to avoid the long warmup for notebooks
  • either you say a gateway is the solution and then I hope you can confirm that connections from a notebook straight to this SQL Server will indeed utilize this gateway
  • or who knows: you may have another option I'm not seeing right now

2 Replies

  • jubinsoni's avatar
    jubinsoni
    Icon for Skilled Sharer rankSkilled Sharer

    Hi nullabee​, I posted a detailed answer on your other thread with the same question, so it stays in one place: Proper way to connect to Azure SQL wrt Firewall. In short: keep the private endpoint in a small ingest workspace (or use a Copy activity via a VNet data gateway) and run the many short notebooks in a workspace without a managed VNet, so they keep the fast starter pools.

  • Hi nullabee​ 

    If you're using notebooks, the only answer is to set up the private endpoints. Gateway connections are not supported in notebooks. 

    How much data are you working with? Would it be feasible to use a gateway and a pipeline with a copy activity to copy the data into a lakehouse, and then interact with it in spark from there?