Forum Discussion
Solved
Notebook. New Errors when loding a parquet file into a df
I create a parquet file (transformed data) which I then use to create dims and facts. Its been working fine. However I changed the code from "overwrite" to append the other day and now this code ...
Hi DebbieE ,
I found this hint from the error message you provided:
snappy.parquet. Column: [ID], Expected: int, Found: BINARY.You can try to check the data format of the ID column. If it does not help, you can provide me with your code so that I can find out if you are having problems.
Best Regards,
Ada Wang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
In my case also when the write mode is overwrite then this error [Caused by: org.apache.spark.sql.execution.datasources.SchemaColumnConvertNotSupportedException: column: [amount], physicalType: INT64, logicalType: int] not there. Once I make it append, I am getting the error. And I have to keep the incoming data appending. So any help would be appriciated on this.