topic Re: Error when writing dataframe to delta table in Data Engineering

Error when writing dataframe to delta table

JayJay11 — Wed, 10 Apr 2024 08:09:26 GMT

Hello all,

second time trying to post this, as the first try was marked as spam (???).

I try to write a dataframe back to a delta table and get an error. This is the code I use to create the dataframe:

dfCustomer = spark.read.table("LakehouseOperations.factCustomerBase") dfScope = spark.read.table("LakehouseOperations.tecDataScopeSnapshots") dfCustomerJoined = dfCustomer.join(dfScope, dfCustomer.snapshot_date == dfScope.scopeDate, "inner").drop("scopeDate", "scopeDateBuckets") dfCustomerJoined.write.mode("overwrite").format("delta").option("overwriteSchema", "true").save("Tables/factCustomerBase")

Then I get the error, see the screenshots here:

How can I solve this? Thanks!

Re: Error when writing dataframe to delta table

Anonymous — Thu, 11 Apr 2024 05:22:56 GMT

Hi JayJay11 ,

Thanks for using Fabric Community.
As part of testing, can you try to create a new table.

dfCustomerJoined.write.mode("overwrite").format("delta").option("overwriteSchema", "true").save("Tables/factCustomerBaseJoined")

Let me know the output of it.

Re: Error when writing dataframe to delta table

JayJay11 — Thu, 11 Apr 2024 05:54:56 GMT

The solution was to change the spark session configuration via the workspace settings:

What it actually does / means... I don't know.

Re: Error when writing dataframe to delta table

Anonymous — Thu, 11 Apr 2024 05:57:38 GMT

Glad to know that you were able to resolve your issue. Please continue using Fabric Community on your further queries.

Re: Error when writing dataframe to delta table

Naveen-004 — Fri, 31 May 2024 04:26:39 GMT

I got the same error even after using the mentioned spark properties.