Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
1 year ago
Solved

Error : Notebook writing table into a Warehouse

Hello,   I try to write a spark dataframe into a Warehouse from a notebook with this tutorial : Spark connector for Microsoft Fabric Data Warehouse - Microsoft Fabric | Microsoft Learn   I can re...
data engineering
Lakehouse
Notebook
  • Anonymous's avatar
    Anonymous
    1 year ago

    Hello CMJJ 

    Thanks for also testing the case.
    I found a solution even if it is not the best because I really think the "writing" function of msspark doesn't work.
    So I used this :

    warehouse_url = "my_url"
jdbc_url = f"jdbc:sqlserver://{warehouse_url}:1433;database={database_name}"
final_table = f"[schema].[table]"

token = mssparkutils.credentials.getToken("pbi")

df_joined.write \
    .format("jdbc") \
    .option("url", jdbc_url) \
    .option("dbtable", final_table) \
    .option("accessToken", token) \
    .option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver") \
    .mode("overwrite") \
    .save()
Anonymous's avatar
Anonymous
Not applicable
1 year ago

Hello,

Thanks for your help.

I don't like the lakehouse staging because I don't need it in my architecture. So I manage the writing with this solution :

warehouse_url = "my_url"
jdbc_url = f"jdbc:sqlserver://{warehouse_url}:1433;database={database_name}"
final_table = f"[schema].[table]"

token = mssparkutils.credentials.getToken("pbi")

df_joined.write \
    .format("jdbc") \
    .option("url", jdbc_url) \
    .option("dbtable", final_table) \
    .option("accessToken", token) \
    .option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver") \
    .mode("overwrite") \
    .save()
sholy29's avatar
sholy29
Frequent Visitor
1 year ago

Hi Ard,

 

I have similar problem just like you. In my case it was overwrite not append (Py4JJavaError: An error occurred while calling o6923.synapsesql. : com.microsoft.spark.fabric.tds.write.error.FabricSparkTDSWriteError: Write orchestration failed.) I want to try this your solution.

 

A quick question. Where can I find the warehouse_url in the Warehouse Item?

 

Thanks