Forum Discussion
Error : Notebook writing table into a Warehouse
Hello, I try to write a spark dataframe into a Warehouse from a notebook with this tutorial : Spark connector for Microsoft Fabric Data Warehouse - Microsoft Fabric | Microsoft Learn I can re...
Hello CMJJ
Thanks for also testing the case.
I found a solution even if it is not the best because I really think the "writing" function of msspark doesn't work.
So I used this :warehouse_url = "my_url" jdbc_url = f"jdbc:sqlserver://{warehouse_url}:1433;database={database_name}" final_table = f"[schema].[table]" token = mssparkutils.credentials.getToken("pbi") df_joined.write \ .format("jdbc") \ .option("url", jdbc_url) \ .option("dbtable", final_table) \ .option("accessToken", token) \ .option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver") \ .mode("overwrite") \ .save()
Hello CMJJ
Thanks for also testing the case.
I found a solution even if it is not the best because I really think the "writing" function of msspark doesn't work.
So I used this :
warehouse_url = "my_url"
jdbc_url = f"jdbc:sqlserver://{warehouse_url}:1433;database={database_name}"
final_table = f"[schema].[table]"
token = mssparkutils.credentials.getToken("pbi")
df_joined.write \
.format("jdbc") \
.option("url", jdbc_url) \
.option("dbtable", final_table) \
.option("accessToken", token) \
.option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver") \
.mode("overwrite") \
.save()
Thanks Ard for the code sample. I had to make one change to your code as it was failing when the notebook runs under the identity of a service principal with your code.
Change:
token = mssparkutils.credentials.getToken("pbi")
To:
token = notebookutils.credentials.getToken('https://database.windows.net')
My guess is that under the covers .synapsesql is using a pbi token and that token for service principals doesn't have the right access (whereas it does for regular user accounts) as mentioned here. Seems like a .synapsesql bug.