Forum Discussion

DebbieE's avatar
DebbieE
Icon for Community Champion rankCommunity Champion
2 years ago

Pyspark Create a ProcessedFiles. Update the flag but I then can't resave OR then view the original P

I have a CurrentProcessed.parquet file containing filename: string (nullable = true)  processedTime: timestamp (nullable = true)  fullyProcessedFlag: integer (nullable = true)   And throughout t...
lakehouse
DebbieE's avatar
DebbieE
Icon for Community Champion rankCommunity Champion
2 years ago

I dont want to do that. I want to have everything in the one file. Because otherwise it will get really confusing. 

 

I still want to see the old data in the latest file with 0 as processed in the latest. 

frithjof_v's avatar
frithjof_v
Icon for Community Champion rankCommunity Champion
2 years ago

I think you could still use a delta table.

 

If you don't want to overwrite the information in the table, perhaps you could update or upsert instead?

 

https://docs.delta.io/latest/delta-update.html#update-a-table

 

I'm not sure I fully understand the functionality you are wanting to achieve. Could you show some pictures of what you're wanting to achieve?

 

Please remember don't show any sensitive or internal information.