Pyspark Create a ProcessedFiles. Update the flag but I then can't resave OR then view the original P

I have a CurrentProcessed.parquet file containing

filename: string (nullable = true) 

processedTime: timestamp (nullable = true) 

fullyProcessedFlag: integer (nullable = true)

And throughout the process the flag is 0. At the end I create a dataframe of the three rows in the parquet file and then set the flag to 1. Processed. 

from pyspark.sql.functions import lit
# Update the fullyProcessedFlag column to 1 for all rows
df_Processed = dflog.select("filename","processedTime").withColumn("fullyProcessedFlag",lit(1))

display(df_Processed)

there is a slight change to the schema

fullyProcessedFlag: integer (nullable = false)

I then overwrite the parquet file with the new data. All processed

workspace_id = "########-####-####-####-############"
lakehouse_id = "########-####-####-####-############"

df_Processed.write.mode("overwrite").option("overwriteSchema", "true")\
.parquet(f"abfss://{workspace_id}@onelake.dfs.fabric.microsoft.com/{lakehouse_id}/Files/Data/Silver/Log/ProcessedFiles.parquet")

# Print a success message
print("Parquet file overwritten successfully.")

And it errors. It removes the files from the parquet folder and leaves the folder empty. 

With the error

Caused by: org.apache.spark.SparkFileNotFoundException: Operation failed: "Not Found", 404, HEAD, It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.

I am at a complete loss with this and its now a blocker. I can't get past it so cant close my run off by resetting the flag in the parquet file.

Can anyone see what I am doing wrong in the code? could the slight schema change to nullable = false be causing the issue (I can't sort this out either and change it to true)