Forum Discussion

eb38335's avatar
eb38335
Regular Visitor
2 years ago
Solved

Using a VNet gateway with Notebooks

We just set up our VNet on prem server to connect to an AWS PostGreSQL database with the latest release of Fabric.

It works great with the Pipeline, but after some discussions with MS consultants, our use case for data delivery will be best met by using Notebooks and pspark queries.

However, I don't see a method to trigger the query through the gateway to get to the data, so we get errors that we can't connect to the source.

Does anyone know a work around? 

We want to avoid using pipelines in the solution unless the pipeline is trigged somehow in the Notebook. 

  • eb38335's avatar
    eb38335
    2 years ago

    Do you know if this feature is available for "On-Premises Data Gateways"?

8 Replies

  • eb38335's avatar
    eb38335
    Regular Visitor

    I'm unsure what you are asking beyond the information i provided.

    We have enabled the On-prem gateways so that Pipelines and Gen2 dataflows can connect to an AWS PostGreSQL RDS database. That works fine with the Pipeline. But we want to use Notebooks instead to query and pull that data in. The Notebook query template given to us by Microsoft consultants Was:

     

    # Import necessary libraries
    from pyspark.sql import SparkSession
     
    # Create a Spark session
    spark = SparkSession.builder \
        .appName("Read from PostgreSQL") \
        .getOrCreate()
     
    # Define the PostgreSQL connection properties
    jdbc_url = "jdbc:postgresql://your-postgres-hostname:5432/your-database"
    connection_properties = {
        "user": "your-username",
        "password": "your-password",
        "driver": "org.postgresql.Driver"
    }
     
    # Read data from a table in the PostgreSQL database
    table_name = "your_table_name"
    df = spark.read.jdbc(url=jdbc_url, table=table_name, properties=connection_properties)
     
    # Show the data
    df.show()
     
    # Stop the Spark session
    spark.stop()

     However, we should be triggering the gateway and not including the username and password in the code. 

     

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi eb38335 ,

    Thanks for using Fabric Community.
    I would like to understand what is the present approach you are following? 
    What approach you are interested in? Any informaion about your Source and Sink?
    Can you please provide these details inorder to guide you better.

  • Is there an existing proposal to make Fabric VNET Data Gateway also support Fabric Notebook? Can we vote on it?