topic Re: Using a VNet gateway with Notebooks in Pipelines

Using a VNet gateway with Notebooks

eb38335 — Fri, 05 Apr 2024 11:11:21 GMT

We just set up our VNet on prem server to connect to an AWS PostGreSQL database with the latest release of Fabric.

It works great with the Pipeline, but after some discussions with MS consultants, our use case for data delivery will be best met by using Notebooks and pspark queries.

However, I don't see a method to trigger the query through the gateway to get to the data, so we get errors that we can't connect to the source.

Does anyone know a work around?

We want to avoid using pipelines in the solution unless the pipeline is trigged somehow in the Notebook.

Re: Using a VNet gateway with Notebooks

Anonymous — Fri, 05 Apr 2024 11:18:22 GMT

Hi eb38335 ,

Thanks for using Fabric Community.
I would like to understand what is the present approach you are following?
What approach you are interested in? Any informaion about your Source and Sink?
Can you please provide these details inorder to guide you better.

Re: Using a VNet gateway with Notebooks

eb38335 — Fri, 05 Apr 2024 11:26:40 GMT

I'm unsure what you are asking beyond the information i provided.

We have enabled the On-prem gateways so that Pipelines and Gen2 dataflows can connect to an AWS PostGreSQL RDS database. That works fine with the Pipeline. But we want to use Notebooks instead to query and pull that data in. The Notebook query template given to us by Microsoft consultants Was:

# Import necessary libraries from pyspark.sql import SparkSession # Create a Spark session spark = SparkSession.builder \ .appName("Read from PostgreSQL") \ .getOrCreate() # Define the PostgreSQL connection properties jdbc_url = "jdbc:postgresql://your-postgres-hostname:5432/your-database" connection_properties = { "user": "your-username", "password": "your-password", "driver": "org.postgresql.Driver" } # Read data from a table in the PostgreSQL database table_name = "your_table_name" df = spark.read.jdbc(url=jdbc_url, table=table_name, properties=connection_properties) # Show the data df.show() # Stop the Spark session spark.stop()

However, we should be triggering the gateway and not including the username and password in the code.

Re: Using a VNet gateway with Notebooks

Anonymous — Fri, 05 Apr 2024 11:41:59 GMT

Hi eb38335 ,

I suggest you to use Azure Key Vault Secrets inorder to store your username and password.
Accessing Azure Key Vault Secrets from Fabric Notebooks (robkerr.ai)

Additional Docs to refer -
Solved: Microsoft Fabric connect to Azure Key Vault - Microsoft Fabric Community

Hope this is helpful. Please let me know incase of further queries.

Re: Using a VNet gateway with Notebooks

eb38335 — Fri, 05 Apr 2024 11:43:08 GMT

Actually, that's what the gateway does. It manages the connection from Fabric to the database for the query when the database path matches what's in the gateway path.

However, I'm unsure how we invoke the gateway from the Notebook like it is invoked from the Pipeline.

Re: Using a VNet gateway with Notebooks

Anonymous — Fri, 05 Apr 2024 12:00:19 GMT

Hi eb38335 ,

Currently we can only invoke the gateway using Pipeline, we cannot invoke it using Fabric Notebooks.

Docs to refer -
VNET Data Gateway for Fabric and Power BI is now Generally Available | Microsoft Fabric Blog | Microsoft Fabric

What is a virtual network (VNet) data gateway | Microsoft Learn

Hope this is helpful. Please let me know incase of further queries.

Re: Using a VNet gateway with Notebooks

eb38335 — Fri, 05 Apr 2024 12:17:43 GMT

Do you know if this feature is available for "On-Premises Data Gateways"?

Re: Using a VNet gateway with Notebooks

Anonymous — Fri, 05 Apr 2024 14:08:16 GMT

Hi eb38335 ,

On-Prem Gateway is supported only for Dataflow Gen 2 and Data Pipeline.

Docs to refer -
How to access on-premises data sources in Data Factory - Microsoft Fabric | Microsoft Learn

Hope this is helpful

Re: Using a VNet gateway with Notebooks

franklloydteh — Thu, 03 Oct 2024 06:10:17 GMT

Is there an existing proposal to make Fabric VNET Data Gateway also support Fabric Notebook? Can we vote on it?