Forum Discussion

Peter_23's avatar
Peter_23
Advocate V
10 months ago
Solved

vacuum mode dont work

Hi community, I try to understand vaccum in delta table, I have a table with this history.

 

 

I ran the maintenance  with 7 days from window commands. 🙂

 

I run the code in notebook.

df_version_1 = spark.read.format("delta").option("versionAsOf", 1).load("path/Tables/Publi")
df_version_1.show()

 

and I get the version 1 from the table.  ðŸ˜•  I think it's wrong because after the vacuum mode the version won't be find, will be?

 

Thanks in advance.

data engineering
Lakehouse
Notebook
Spark Job Definition
  • v-pnaroju-msft's avatar
    v-pnaroju-msft
    10 months ago

    Hi Peter_23,

    We wanted to see if the information we gave helped fix your problem. If you need more help, please feel free to contact the Microsoft Fabric community.

    Thank you.

9 Replies

  • AntoineW's avatar
    AntoineW
    Super User
    10 months ago

    Hello Peter_23, 

     

    Hope you are fine, this is a summary of VACUUM : 

    • VACUUM only deletes unused data files older than the retention period.
    • It does not delete the Delta transaction log (the history/versions).
    • You can still read versionAsOf=1 as long as the data files used by version 1 still exist—either because:
      1. they are newer than the 7‑day retention, or
      2. they’re still referenced by newer versions (e.g., you only appended new data and older files remain part of the latest snapshot).

    So what you’re seeing—being able to spark.read versionAsOf=1 after a VACUUM RETAIN 7 DAYS—can be perfectly correct.

     

    Some docs that can help you : 

    - https://learn.microsoft.com/en-us/azure/databricks/delta/vacuum

    - https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-table-maintenance

     

    Hope it can help you ! 

    Best regards,

    Antoine

  • v-pnaroju-msft's avatar
    v-pnaroju-msft
    Community Support
    10 months ago

    Thankyou p-mndl, AntoineW, and BhaveshPatel, for your responses.

    Hi Peter_23,

    We appreciate your inquiry through the Microsoft Fabric Community Forum.

    We would like to inquire whether have you got the chance to check the solutions provided by p-mndl, AntoineW, and BhaveshPatel to resolve the issue. We hope the information provided helps to clear the query. Should you have any further queries, kindly feel free to contact the Microsoft Fabric community.

    Thank you.

    • Peter_23's avatar
      Peter_23
      Advocate V
      10 months ago

      Thanks BhaveshPatel

       

      The describe command show me:

       

       

      and finally ...

      spark.read.format("delta").option("versionAsOf", 1).load...

       

  • p-mndl's avatar
    p-mndl
    New Member
    10 months ago

    You are correct as in it should remove older entries. How exactly did you run your vacuum?

  • BhaveshPatel's avatar
    BhaveshPatel
    Super User
    10 months ago

    Hi Peter_23 

     

    You can use below vaccum commands to retain the period as you would like. For Microsoft, they can keep vaccum for 7 days but you can change programmatically using python.. 

     

    spark.sql("set spark.databricks.delta.retentionDurationCheck.enabled = false")
    spark.sql("VACUUM DimDate RETAIN 8 HOURS")
     
  • v-pnaroju-msft's avatar
    v-pnaroju-msft
    Community Support
    10 months ago

    Hi Peter_23,

    We are following up to see if what we shared solved your issue. If you need more support, please reach out to the Microsoft Fabric community.

    Thank you.

  • v-pnaroju-msft's avatar
    v-pnaroju-msft
    Community Support
    10 months ago

    Hi Peter_23,

    We wanted to see if the information we gave helped fix your problem. If you need more help, please feel free to contact the Microsoft Fabric community.

    Thank you.