Forum Discussion
Solved
Error when writing dataframe to delta table
Hello all,
second time trying to post this, as the first try was marked as spam (???).
I try to write a dataframe back to a delta table and get an error. This is the code I use to create the dataframe:
dfCustomer = spark.read.table("LakehouseOperations.factCustomerBase")
dfScope = spark.read.table("LakehouseOperations.tecDataScopeSnapshots")
dfCustomerJoined = dfCustomer.join(dfScope, dfCustomer.snapshot_date == dfScope.scopeDate, "inner").drop("scopeDate", "scopeDateBuckets")
dfCustomerJoined.write.mode("overwrite").format("delta").option("overwriteSchema", "true").save("Tables/factCustomerBase")
Then I get the error, see the screenshots here:
How can I solve this? Thanks!
The solution was to change the spark session configuration via the workspace settings:
What it actually does / means... I don't know.
4 Replies
Hi JayJay11 ,
Thanks for using Fabric Community.
As part of testing, can you try to create a new table.dfCustomerJoined.write.mode("overwrite").format("delta").option("overwriteSchema", "true").save("Tables/factCustomerBaseJoined")
Let me know the output of it.The solution was to change the spark session configuration via the workspace settings:
What it actually does / means... I don't know.
