Forum Discussion
Error: Writing data into Warehouse using a pyspark notebook
Thanks for the recomendation.
Schema mismatch
I have checked both the source dataframe (bus_mgt_data_reporting_status) and the destination table in the warehouse. Both of them have the same structure in terms column name, column order and data type.
Without stopping the current session, I tried writing the dataframe (bus_mgt_data_reporting_status) into a new table (with an new table name) in the warehouse but I get the same error.
However, when I stop the current session and use a dummy data to write into a new table in the warehouse It went pretty well.
My guess is that it has to do with the session as at when writing the bus_mgt_data_reporting_status table to the warehouse. What do you recommend.
Community SupportHi sholy29 , Thank you for reaching out to the Microsoft Community Forum.
Based on your description, the issue you're facing is almost certainly tied to a corrupted or unstable Spark session. The fact that writing fails even to a new table within the same session but works fine after restarting rules out schema mismatch and points directly to stale execution plans or internal memory/cache inconsistencies that affect write orchestration in Fabric's Spark runtime.
To resolve this without restarting your session, force Spark to fully evaluate and materialize your DataFrame before the write. Do this by adding .cache() followed by .count() before the write call:
df_pivoted = df_pivoted.cache() df_pivoted.count() df_pivoted.write.mode("overwrite").synapsesql("WH.dbo.bus_mgt_data_reporting_status")
This ensures that all transformations are resolved ahead of the write and breaks any lingering state that might interfere with orchestration. If you're working with a large dataset, you can optionally add .repartition(n) before the write to help avoid shuffle-related failures.
If this helped solve the issue, please consider marking it “Accept as Solution” so others with similar queries may find it more easily. If not, please share the details, always happy to help.
Thank you.