Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Calling all Data Engineers! Fabric Data Engineer (Exam DP-700) live sessions are back! Starting October 16th. Sign up.

Reply
QuinnDatatyk
Frequent Visitor

Spark Write to OneLake cannot overwrite data schema

I cannot overwrite table schema using spark.write.mode("overwrite")

For example, create a table with 2 column: col1 as int and col2 as string

QuinnDatatyk_0-1713088635792.png

Then change col1 to data type int and use spark.write.mode("overwrite")

QuinnDatatyk_2-1713088720743.png

It return error 

QuinnDatatyk_3-1713088773822.png

 

 

 

1 ACCEPTED SOLUTION
frithjof_v
Super User
Super User

I think if you want to overwrite the schema while overwriting the data, you could write like this:

 

df\
   .write\
   .mode("overwrite")\
   .option("overwriteSchema", "true")\
   .saveAsTable("tableName")

 

So I think you only need to add .option("overwriteSchema", "true") in your code

 

 

Ref. https://community.fabric.microsoft.com/t5/General-Discussion/Dropping-and-recreating-lakehouse-table...

 

View solution in original post

2 REPLIES 2
frithjof_v
Super User
Super User

I think if you want to overwrite the schema while overwriting the data, you could write like this:

 

df\
   .write\
   .mode("overwrite")\
   .option("overwriteSchema", "true")\
   .saveAsTable("tableName")

 

So I think you only need to add .option("overwriteSchema", "true") in your code

 

 

Ref. https://community.fabric.microsoft.com/t5/General-Discussion/Dropping-and-recreating-lakehouse-table...

 

I tried and successed. Thank you very much.

Helpful resources

Announcements
FabCon Global Hackathon Carousel

FabCon Global Hackathon

Join the Fabric FabCon Global Hackathon—running virtually through Nov 3. Open to all skill levels. $10,000 in prizes!

September Fabric Update Carousel

Fabric Monthly Update - September 2025

Check out the September 2025 Fabric update to learn about new features.

FabCon Atlanta 2026 carousel

FabCon Atlanta 2026

Join us at FabCon Atlanta, March 16-20, for the ultimate Fabric, Power BI, AI and SQL community-led event. Save $200 with code FABCOMM.