Forum Discussion

P_work's avatar
P_work
Helper I
1 year ago
Solved

Spark reading writing to SQL Database

The following blog post illustrates reading/writing to Fabric Warehouse from Spark notebook. @https://community.fabric.microsoft.com/t5/Data-Engineering-Community-Blog/Spark-Connector-for-Fabric-War...
data engineering
Notebook
  • v-lgarikapat's avatar
    v-lgarikapat
    1 year ago

    Hi P_work ,

    Thanks for reaching out to the Microsoft fabric community forum.

     

    nilendraFabric  

    Thanks for your prompt response

    P_work , I have attached the official document and relevant solved threads related to your use case. I hope you find them helpful

     

    Spark connector for Microsoft Fabric Data Warehouse - Microsoft Fabric | Microsoft Learn

    Solved: Re: Fabric Notebook Outputs to Azure SQL Database ... - Microsoft Fabric Community

    Solved: Re: Write df to DataWarehouse tables - Microsoft Fabric Community

     

    If this post helped resolve your issue, please consider giving it Kudos and marking it as the Accepted Solution. This not only acknowledges the support provided but also helps other community members find relevant solutions more easily.

    We appreciate your engagement and thank you for being an active part of the community.

    Best regards,
    LakshmiNarayana    .

     

     

     

  • g3kuser's avatar
    g3kuser
    1 year ago

    Here is how you connect to SQL DB from Fabric Notebook using SPN credentials

    host = ''
    database = ''
    sql_query = ''
    jdbc_url = f'jdbc:sqlserver://{host};database={database};authentication=ActiveDirectoryServicePrincipal;encrypt=true;trustServerCertificate=true;'
    db_properties = {
    "user": f"{client_id}",
    "password": f"{client_secret}",
    "driver": "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    }
    #For Reading
    spark.read.jdbc(url=jdbc_url, table=f"({sql_query}) AS query_alias", properties=db_properties)
    #For Writing data
    df.write.jdbc(url=jdbc_url, table=table_name, mode=mode, properties=db_properties)


g3kuser's avatar
g3kuser
Helper II
1 year ago

Here is how you connect to SQL DB from Fabric Notebook using SPN credentials

host = ''
database = ''
sql_query = ''
jdbc_url = f'jdbc:sqlserver://{host};database={database};authentication=ActiveDirectoryServicePrincipal;encrypt=true;trustServerCertificate=true;'
db_properties = {
"user": f"{client_id}",
"password": f"{client_secret}",
"driver": "com.microsoft.sqlserver.jdbc.SQLServerDriver"
}
#For Reading
spark.read.jdbc(url=jdbc_url, table=f"({sql_query}) AS query_alias", properties=db_properties)
#For Writing data
df.write.jdbc(url=jdbc_url, table=table_name, mode=mode, properties=db_properties)