Forum Discussion

DebbieE's avatar
DebbieE
Icon for Community Champion rankCommunity Champion
2 years ago

Pyspark Create a ProcessedFiles. Update the flag but I then can't resave OR then view the original P

I have a CurrentProcessed.parquet file containing filename: string (nullable = true)  processedTime: timestamp (nullable = true)  fullyProcessedFlag: integer (nullable = true)   And throughout t...
lakehouse
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi DebbieE 

 

I reproduced the same error when trying to update a parquet file with your method. However I haven't figured out a solution yet. Still need some time to work on this, as I'm new to Spark and not very experienced by now.

 

Best Regards,
Jing