Forum Discussion
notebook pyspark constant errors writing more than 2 files
The data exists in the delta table in the lakehouse and it seems to display the data without issues - so I'm assuming the structure of the delta files are fine.
But it seems like a issue with data quality specifically column names with suprious characters like $ . " etc. But if it is already in the lakehouse as delta table don't understand why it can't output the data without failing. Anyway I had to clean up all the column names then it worked
It looks like there is a problem with 3rd delta file. can you check whether its actually a delta file and it contains _delta_log/ folder inside it and also verify access permissions for that file. Since spark executes lazily, it failes at the write step only and it wont fail at read step (but high chance that there is problem in read step).
Try to read & write it separately without a loop in another cell.
If this is the reason and it addressed your query, please accept it as a solution or reason for issue and give a 'Kudos' so other members can easily find it.
The data exists in the delta table in the lakehouse and it seems to display the data without issues - so I'm assuming the structure of the delta files are fine.
But it seems like a issue with data quality specifically column names with suprious characters like $ . " etc. But if it is already in the lakehouse as delta table don't understand why it can't output the data without failing. Anyway I had to clean up all the column names then it worked