Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
1 year ago
Solved

Write df to DataWarehouse tables

can someone show me how to write a dataframe df to a table in a Warehouse in Fabric?

I try below code, but having issue... I suppose there is no username or password involved here:

 

from pyspark.sql import SparkSession

# Initialize Spark Session
spark = SparkSession.builder.appName("FabricDW").getOrCreate()

# Sample DataFrame
data = [("Alice", 30, "Engineer"),
        ("Bob", 35, "Analyst"),
        ("Charlie", 40, "Manager")]

columns = ["Name", "Age", "JobTitle"]
df = spark.createDataFrame(data, columns)

# JDBC Connection using Interactive Authentication
dw_url = "jdbc:sqlserver://testconnectionstring.datawarehouse.fabric.microsoft.com:1433;" \
         "database=WideWorldImportersDW;encrypt=true"

dw_properties = {
    "driver": "com.microsoft.sqlserver.jdbc.SQLServerDriver"
}

# Write DataFrame to Data Warehouse Table
df.write.jdbc(url=dw_url, table="dbo.Employee", mode="overwrite", properties=dw_properties)

print("✅ Data written successfully to Fabric Data Warehouse!")
 
Py4JJavaError: An error occurred while calling o6216.jdbc. : com.microsoft.sqlserver.jdbc.SQLServerException: Could not login because the authentication failed. ClientConnectionId:6797891d-d522-4841-9792-5c16025f39b1 at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:259)
data engineering
  • nilendraFabric's avatar
    nilendraFabric
    1 year ago

     

    hello Anonymous 

     

     

    Prepare the JDBC URL with the correct endpoint, database name, encryption, and authentication

    dw_url = (“jdbc:sqlserver://<your_server_identifier>.datawarehouse.fabric.microsoft.com:1433;”
“DatabaseName=WideWorldImportersDW;”
“encrypt=true;”
“Authentication=ActiveDirectoryPassword”)

     


    Provide authentication details in the connection properties
    dw_properties = {
“driver”: “com.microsoft.sqlserver.jdbc.SQLServerDriver”,
“user”: “[email protected]”,“password”: “your_password” 
}

     


    Write the DataFrame to the target table in the Data Warehouse
    df.write.jdbc(url=dw_url, table=“dbo.Employee”, mode=“overwrite”, properties=dw_properties)

4 Replies

  • nilendraFabric's avatar
    nilendraFabric
    Super User
    1 year ago

     

    hello Anonymous 

     

     

    Prepare the JDBC URL with the correct endpoint, database name, encryption, and authentication

    dw_url = (“jdbc:sqlserver://<your_server_identifier>.datawarehouse.fabric.microsoft.com:1433;”
“DatabaseName=WideWorldImportersDW;”
“encrypt=true;”
“Authentication=ActiveDirectoryPassword”)

     


    Provide authentication details in the connection properties
    dw_properties = {
“driver”: “com.microsoft.sqlserver.jdbc.SQLServerDriver”,
“user”: “[email protected]”,“password”: “your_password” 
}

     


    Write the DataFrame to the target table in the Data Warehouse
    df.write.jdbc(url=dw_url, table=“dbo.Employee”, mode=“overwrite”, properties=dw_properties)

  • v-hashadapu's avatar
    v-hashadapu
    Community Support
    1 year ago

    Hi Anonymous , Thank you for reaching out to the Microsoft Community Forum.

    Please let us know if your issue is solved. If it is, consider marking the answer that helped 'Accept as Solution', so others with similar queries can find it easily. If not, please share the details.
    Thank you.

    specail thanks to nilendraFabric for your prompt reply

     

  • v-hashadapu's avatar
    v-hashadapu
    Community Support
    1 year ago

    Hi Anonymous , Thank you for reaching out to the Microsoft Community Forum.

    Please let us know if your issue is solved. If it is, consider marking the answer that helped 'Accept as Solution', so others with similar queries can find it easily. If not, please share the details.
    Thank you.

  • v-hashadapu's avatar
    v-hashadapu
    Community Support
    1 year ago

    Hi @tanthiamhuat88 , Thank you for reaching out to the Microsoft Community Forum.

    Please let us know if your issue is solved. If it is, consider marking the answer that helped 'Accept as Solution', so others with similar queries can find it easily. If not, please share the details.
    Thank you.