Pyspark Create a ProcessedFiles. Update the flag but I then can't resave OR then view the original P

Hi DebbieE 

I was interested in your question and reproduced your error. I am not entirely sure but I assume that the behaviour has something to do with the way spark processes data and keeps track of the schema of each dataframe and where it comes from. In your case I would assume that there is some kind of circular dependency of the schema you are overwritting to the original schema of the file and that is why spark runs into that error - but as I said, pretty unsure about that and actually just an assumption.

Nevertheless I was able to at least come up with a solution/ workaround: 

For the exact reason of different versions (in terms of schema-versions) of the parquet-file it seems obvious to use delta instead of parquet. So spark can keep track of the originally read schema while creating a newer version of the file since the "older" version is still kept in the lakehouse.

dflog_delta = spark.read.format('delta').load("Tables/" + delta_table_name)

from pyspark.sql.functions import lit
df_Processed_delta = dflog_delta.select("filename","processedTime").withColumn("fullyProcessedFlag",lit(1))

df_Processed_delta.write.mode("overwrite").option("overwriteSchema", "true")\
.format('delta').saveAsTable(delta_table_name)

print("Delta file overwritten successfully.")

Hope this solution works for you. 🙂

BR

Martin