Forum Discussion
Error: Writing data into Warehouse using a pyspark notebook
Hi everyone I currently have a pyspark notebook that writes data into Warehouse using the synapsesql (df_pivoted.write.mode("overwrite").synapsesql("Data Reporting Status WH.dbo.bus_mgt_data_repo...
Thanks for the recomendation.
Schema mismatch
I have checked both the source dataframe (bus_mgt_data_reporting_status) and the destination table in the warehouse. Both of them have the same structure in terms column name, column order and data type.
Without stopping the current session, I tried writing the dataframe (bus_mgt_data_reporting_status) into a new table (with an new table name) in the warehouse but I get the same error.
However, when I stop the current session and use a dummy data to write into a new table in the warehouse It went pretty well.
My guess is that it has to do with the session as at when writing the bus_mgt_data_reporting_status table to the warehouse. What do you recommend.