Forum Discussion

vedant_9's avatar
vedant_9
New Member
9 days ago

Reading on-prem SQL Server / Oracle from a Fabric notebook: is the gateway really not an option?

Hi all, We’re moving our ingestion framework from Databricks to Fabric and have run into a connectivity problem. The setup is metadata-driven. A single PySpark notebook reads a control table in a l...
data engineering
data pipeline
environment
ShafiUllah-bc's avatar
ShafiUllah-bc
Icon for Resolver I rankResolver I
8 days ago

Hi, yes that's right, notebooks can't use the on-prem gateway, so the JDBC call times out.

What I'd suggest:
1. Use Copy job to land the tables in the Lakehouse. It supports on-prem SQL Server and Oracle through the gateway and does watermark based incremental loads for you.
2. Then run your notebook on top of that for the bronze logic.

Just keep in mind watermark loads won't catch deletes, CDC would be needed for that.

Managed private endpoint to on-prem is also possible now (Private Link Service + VPN/ExpressRoute), but it's a proper networking setup. I'd only go that way if you really need direct access from the notebook.