Forum Discussion
Error : Notebook writing table into a Warehouse
Hello CMJJ
Thanks for also testing the case.
I found a solution even if it is not the best because I really think the "writing" function of msspark doesn't work.
So I used this :warehouse_url = "my_url" jdbc_url = f"jdbc:sqlserver://{warehouse_url}:1433;database={database_name}" final_table = f"[schema].[table]" token = mssparkutils.credentials.getToken("pbi") df_joined.write \ .format("jdbc") \ .option("url", jdbc_url) \ .option("dbtable", final_table) \ .option("accessToken", token) \ .option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver") \ .mode("overwrite") \ .save()
Hi Anonymous ,
Thank you for using Microsoft Community Forum.
Akash_Varuna explanation is correct Schema mismatches often cause this error. Differences in column names, data types, or column count between the DataFrame and the target table can lead to a write failure. Using overwrite can fix this by recreating the table, but it deletes existing data, so use it cautiously.
In addition to schema mismatches, the error mentioning "URL suffix not allowed" points to a possible issue with how Spark writes temporary files to OneLake storage especially if wildcards like *.parquet appear in the path. This can happen if the Fabric connector generates an invalid path for staging data. To work around this, consider writing the DataFrame to a Lakehouse first, then loading it into the Warehouse from there.
Lastly, permissions on both the Warehouse and OneLake storage are critical. Ensure the notebook identity (user, service principal, or managed identity) has the necessary write permissions for the Warehouse table and read/write access to OneLake.
If the problem continues, double-check that you’re using the latest version of the Microsoft Fabric Spark connector, and enable detailed logging to get clearer insights into the failure.
I hope this will reslove your issue, if you need any further assistance, feel free to reach out.
If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.
Thankyou.
Hello,
Thanks for your help.
I don't like the lakehouse staging because I don't need it in my architecture. So I manage the writing with this solution :
warehouse_url = "my_url"
jdbc_url = f"jdbc:sqlserver://{warehouse_url}:1433;database={database_name}"
final_table = f"[schema].[table]"
token = mssparkutils.credentials.getToken("pbi")
df_joined.write \
.format("jdbc") \
.option("url", jdbc_url) \
.option("dbtable", final_table) \
.option("accessToken", token) \
.option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver") \
.mode("overwrite") \
.save()Hi Ard,
I have similar problem just like you. In my case it was overwrite not append (Py4JJavaError: An error occurred while calling o6923.synapsesql. : com.microsoft.spark.fabric.tds.write.error.FabricSparkTDSWriteError: Write orchestration failed.) I want to try this your solution.
A quick question. Where can I find the warehouse_url in the Warehouse Item?
Thanks
