Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
1 year ago
Solved

Error : Notebook writing table into a Warehouse

Hello,   I try to write a spark dataframe into a Warehouse from a notebook with this tutorial : Spark connector for Microsoft Fabric Data Warehouse - Microsoft Fabric | Microsoft Learn   I can re...
data engineering
Lakehouse
Notebook
  • Anonymous's avatar
    Anonymous
    1 year ago

    Hello CMJJ 

    Thanks for also testing the case.
    I found a solution even if it is not the best because I really think the "writing" function of msspark doesn't work.
    So I used this :

    warehouse_url = "my_url"
jdbc_url = f"jdbc:sqlserver://{warehouse_url}:1433;database={database_name}"
final_table = f"[schema].[table]"

token = mssparkutils.credentials.getToken("pbi")

df_joined.write \
    .format("jdbc") \
    .option("url", jdbc_url) \
    .option("dbtable", final_table) \
    .option("accessToken", token) \
    .option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver") \
    .mode("overwrite") \
    .save()
Akash_Varuna's avatar
Akash_Varuna
Super User
1 year ago

Hi Anonymous  , It might indeed be due to schema mismatches. When using append, if there are schema differences between the source and the target table (like column names, types, or counts), Spark will not be able to write and will throw an error. In such cases, you may need to use overwrite instead. With overwrite, it will drop and recreate the table with the new schema, solving the mismatch issue, but this deletes all existing data, so use it cautiously. If it's not schema mismatches, the error might be related to the URL path or storage permissions.