Forum Discussion
MangoMagic
1 year agoRegular Visitor
notebook pyspark constant errors writing more than 2 files
I am trying to output many lakehouse tables to parquet files under the Files section within the lakehouse, but in pyspark notebook it can only seem to output 2 tables without failing, anything more t...
- 1 year ago
The data exists in the delta table in the lakehouse and it seems to display the data without issues - so I'm assuming the structure of the delta files are fine.
But it seems like a issue with data quality specifically column names with suprious characters like $ . " etc. But if it is already in the lakehouse as delta table don't understand why it can't output the data without failing. Anyway I had to clean up all the column names then it worked
ObungiNiels
Resolver III
1 year agoHi MangoMagic ,
what's the error message? It seems to be cropped from your second screen shot. Would help for analysis. 🙂