Forum Discussion
Error: Writing data into Warehouse using a pyspark notebook
Thanks for the recomendation.
Schema mismatch
I have checked both the source dataframe (bus_mgt_data_reporting_status) and the destination table in the warehouse. Both of them have the same structure in terms column name, column order and data type.
Without stopping the current session, I tried writing the dataframe (bus_mgt_data_reporting_status) into a new table (with an new table name) in the warehouse but I get the same error.
However, when I stop the current session and use a dummy data to write into a new table in the warehouse It went pretty well.
My guess is that it has to do with the session as at when writing the bus_mgt_data_reporting_status table to the warehouse. What do you recommend.
Hi
Thanks, I tried your recommendation but I still got the same error. This is really futrating because two days ago, it worked.
Hi sholy29 , Thank you for reaching out to the Microsoft Community Forum.
Your issue is almost certainly caused by instability in the Spark-to-Synapse SQL write orchestration layer. Microsoft recently rolled out Runtime 1.3 and introduced the writeHeavy Spark resource profile as the default for ingestion-heavy workloads like yours. If your workspace hasn’t been updated to use this or if your session has cached stale planning state, you’ll encounter errors like FabricTDSWriteError.
Make sure your Spark pool is running Fabric Runtime 1.3 or later. Disable the Fast Data Path layer, which is known to cause intermittent failures during warehouse writes, especially after session reuse or failed writes. Set the Spark config as follows before your write:
spark.conf.set("spark.fabric.tds.write.enableFastDataPath", "false")
Optionally, if you're not already using the writeHeavy resource profile, apply it:
spark.conf.set("spark.fabric.resourceProfile", "writeHeavy")
Also, force the DataFrame to be evaluated and cached before writing to avoid Spark triggering both compute and orchestration in the same stage:
df_pivoted = df_pivoted.cache() df_pivoted.count() df_pivoted.write.mode("overwrite").synapsesql("WH.dbo.bus_mgt_data_reporting_status")
If this doesn’t solve the issue, then the best course of action is raising a Microsoft Fabric support ticket to fix the issue. Provide them with all the necessary details including Error screenshots and all the troubleshooting steps you have tried so far. This will help them better understand the issue and provide a solution.
Below is the link to help create Microsoft Support ticket:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft LearnIf this helped solve the issue, please consider marking it “Accept as Solution” so others with similar queries may find it more easily. If not, please share the details, always happy to help.
Thank you.