Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
1 year ago
Solved

Error : Notebook writing table into a Warehouse

Hello,   I try to write a spark dataframe into a Warehouse from a notebook with this tutorial : Spark connector for Microsoft Fabric Data Warehouse - Microsoft Fabric | Microsoft Learn   I can re...
data engineering
lakehouse
notebook
Anonymous's avatar
Anonymous
Not applicable
1 year ago

Hi Anonymous ,

Thank you for using Microsoft Community Forum.

 

Akash_Varuna  explanation is correct Schema mismatches often cause this error. Differences in column names, data types, or column count between the DataFrame and the target table can lead to a write failure. Using overwrite can fix this by recreating the table, but it deletes existing data, so use it cautiously.

 

In addition to schema mismatches, the error mentioning "URL suffix not allowed" points to a possible issue with how Spark writes temporary files to OneLake storage especially if wildcards like *.parquet appear in the path. This can happen if the Fabric connector generates an invalid path for staging data. To work around this, consider writing the DataFrame to a Lakehouse first, then loading it into the Warehouse from there.

 

Lastly, permissions on both the Warehouse and OneLake storage are critical. Ensure the notebook identity (user, service principal, or managed identity) has the necessary write permissions for the Warehouse table and read/write access to OneLake.

 

If the problem continues, double-check that you’re using the latest version of the Microsoft Fabric Spark connector, and enable detailed logging to get clearer insights into the failure.

 

I hope this will reslove your issue, if you need any further assistance, feel free to reach out.

 

If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

 

Thankyou.

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Hello,

    Thanks for your help.

    I don't like the lakehouse staging because I don't need it in my architecture. So I manage the writing with this solution :

    warehouse_url = "my_url"
jdbc_url = f"jdbc:sqlserver://{warehouse_url}:1433;database={database_name}"
final_table = f"[schema].[table]"

token = mssparkutils.credentials.getToken("pbi")

df_joined.write \
    .format("jdbc") \
    .option("url", jdbc_url) \
    .option("dbtable", final_table) \
    .option("accessToken", token) \
    .option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver") \
    .mode("overwrite") \
    .save()
    • sholy29's avatar
      sholy29
      Frequent Visitor
      1 year ago

      Hi Ard,

       

      I have similar problem just like you. In my case it was overwrite not append (Py4JJavaError: An error occurred while calling o6923.synapsesql. : com.microsoft.spark.fabric.tds.write.error.FabricSparkTDSWriteError: Write orchestration failed.) I want to try this your solution.

       

      A quick question. Where can I find the warehouse_url in the Warehouse Item?

       

      Thanks