topic Re: VACUUM with less than 1 hour retaintion in Data Engineering

VACUUM with less than 1 hour retaintion

DennesTorres — Sat, 02 Sep 2023 01:57:31 GMT

Hi,

For testing purposes, is it possible to use VACUUM with less than 1 hour retaintion ?

In order to do it with a 1 hour retaintion I had to set a spark configuration:

spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")

But what if, for tests purpose, I would like to VACUUM with less than 1 hour ?

Kind Regards,

Dennes

Re: VACUUM with less than 1 hour retaintion

puneetvijwani — Sat, 02 Sep 2023 18:22:06 GMT

HI DennesTorres lets try like this ..i have seen some users doing that

spark.sql("VACUUM bronze_nyc_tlc_green.nycgreentaxi RETAIN 0 HOURS")

This can be the way to vacuum everything that is not in use. PS: I havent tested this , to be honest too tired today ( Just kidding )
Note : This is also can be risky and should only be done in a controlled environment where you are certain that you do not need older versions of the data.

Let me also know if that works for u !

Re: VACUUM with less than 1 hour retaintion

HimanshuS-msft — Wed, 06 Sep 2023 17:25:22 GMT

Hi DennesTorres ,
When i tested it , it did not threw any error , but at the same time it did not threw any error .
Just curiuos as to what is the bigger problem which you are tring to solve ?

Thanks
Himanshu

Re: VACUUM with less than 1 hour retaintion

DennesTorres — Wed, 06 Sep 2023 21:48:09 GMT

Hi,

It worked for me in lakehouses.

I'm concerned how these files would affect the performance.

Kind Regards,

Dennes