topic Re: Delta Lake VACUUM, how does it work? in Data Engineering

Delta Lake VACUUM error

amaaiia — Tue, 16 Apr 2024 13:40:31 GMT

I have a table I daily overwrite from a DFg2. I have been doing this overwrite for several days, so it generates a new parquet file every day, and this overwrite is added in the metadata. I understand that with the VACUUM command I can delete those parquet files that are not used since X days ago. I guess my old parquet files are not used because what I do is to overwrite the whole table. I clicked on Run VACUUM command using retention threshold 7 days, and it didn't work. In fact, I got an error:

Executing vacuum with retention hours: 168.0 An error occurred while calling o313.vacuum. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 24 in stage 28.0 failed 4 times, most recent failure: Lost task 24.3 in stage 28.0 (TID 980) (vm-bb589657 executor 1): java.lang.AssertionError: assertion failed

How does VACUUM command work? Why isn't it working for my very simple use case?

Re: Delta Lake VACUUM, how does it work?

Anonymous — Wed, 27 Mar 2024 10:31:35 GMT

Hi @amaaiia ,

Thanks for using Fabric Community.
Did you got a chance to look into this doc?
Use table maintenance feature to manage delta tables in Fabric - Microsoft Fabric | Microsoft Learn

Can you please share the screenshot of the issue you are facing?

Re: Delta Lake VACUUM, how does it work?

Anonymous — Thu, 28 Mar 2024 08:27:28 GMT

Hi @amaaiia ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

Re: Delta Lake VACUUM, how does it work?

Anonymous — Fri, 29 Mar 2024 10:17:15 GMT

Re: Delta Lake VACUUM error

amaaiia — Tue, 16 Apr 2024 13:41:18 GMT

I've deleted the table, probably internal issues with the tables, because with other tables I can run VACUUM command.

Re: Delta Lake VACUUM error

Anonymous — Tue, 16 Apr 2024 14:42:46 GMT

Hi @amaaiia ,

It was great to know that you were able to get to a resolution . We expect you to keep using this forum and also motivate others to do that same . You can always help other community members by answering to their queries