Forum Discussion

Naveen-004's avatar
Naveen-004
Frequent Visitor
2 years ago
Solved

Notebook fails to write data to a Delta table

Code: df.write.format("delta").mode("overwrite").option("mergeSchema", "true").save(delta_table_path) Error: Py4JJavaError: An error occurred while calling o29386.save. : org.apache.spark.SparkE...
data engineering
lakehouse
notebook
  • richbenmintz's avatar
    richbenmintz
    2 years ago

    Hi Naveen-004,

    Are you able to share all the Code in your Notebook, Spark is a lazily evaluated there could be issues in earlier commands that are only getting executed when you try to write the dataframe.

Naveen-004's avatar
Naveen-004
Frequent Visitor
2 years ago

I checked the Spark history and found that one of the stages was skipped, leading to a failure. Can you assist with troubleshooting the error?

 

I don't believe resource constraints are the issue. The computer we are using has 8 Spark driver cores, 56GB Spark driver memory, 8 Spark executor cores, and 56GB Spark executor memory.

Dynamically allocate executors: Enabled
Spark executor instances: 9

Spark Properties:
I have set the first two properties based on below community forum
https://community.fabric.microsoft.com/t5/Data-Engineering/Writing-dataframe-to-delta-table-fails-with-error-Spark/m-p/3833372/highlight/true

  • spark.sql.parquet.datetimeRebaseModeInRead: CORRECTED
  • spark.sql.parquet.datetimeRebaseModeInWrite: CORRECTED
  • spark.sql.parquet.vorder.enabled: true
  • spark.ms.autotune.enabled: false
  • spark.microsoft.delta.optimizeWrite.enabled: true
  • spark.microsoft.delta.merge.lowShuffle.enabled: true

I retried the write operation, but it failed with the same error.

 

v-cboorla-msft's avatar
v-cboorla-msft
Microsoft Employee
2 years ago

Hi Naveen-004 

 

Apologies for the inconvenience.

Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

 

Thank you.