topic Re: Question About Table History Discrepancy in Lakehouse in Data Engineering

Question About Table History Discrepancy in Lakehouse

stellahe102 — Wed, 14 May 2025 16:19:19 GMT

Hi,

In our lakehouse environment, we have about 20 tables. I’ve noticed that only 2 of these tables retain just the past 30 days of parquet files, while the rest maintain a full history (beyond 30 days). I’m still getting familiar with using the vacuum option and other setting in delta tables, so I’m not sure what dependencies might be affecting this.

This difference is a recent change—before this month (2025 May), all tables stored full history. Now, only these 2 tables are limited to 30 days of history, which caught my attention. Do you have any idea why this might be happening?

I checked with the workspace owner, and there haven’t been any recent changes on their end. Could this behavior be related to the fact that these 2 tables are shortcut targets from another workspace owned by a different user? or anything else that have affected this, please share, thanks very much

Re: Question About Table History Discrepancy in Lakehouse

v-ssriganesh — Thu, 15 May 2025 04:00:49 GMT

Hi @stellahe102,
Thank you for reaching out to the Microsoft Fabric Forum Community.

The discrepancy where two shortcut tables retain only 30 days of parquet files is likely due to VACUUM operations or predictive optimization on the source tables in the other workspace, limiting their history. Since shortcuts reference source data, maintenance like VACUUM with a 30-day retention (e.g., delta.deletedFileRetentionDuration = interval 30 days) affects their history.

Steps to Investigate:

Check source table properties: SHOW TBLPROPERTIES source_table_name for delta.deletedFileRetentionDuration.
Review source workspace for VACUUM jobs.
Coordinate with the source workspace owner to align retention policies.

Please refer below official documentation:

Delta Lake VACUUM Command

If this information is helpful, please “Accept as solution” and give a "kudos" to assist other community members in resolving similar issues more efficiently.
Thank you.

Re: Question About Table History Discrepancy in Lakehouse

v-ssriganesh — Mon, 19 May 2025 14:34:49 GMT

Hi @stellahe102,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

Re: Question About Table History Discrepancy in Lakehouse

v-ssriganesh — Thu, 22 May 2025 13:59:38 GMT

Hi @stellahe102,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.

Re: Question About Table History Discrepancy in Lakehouse

v-ssriganesh — Sun, 25 May 2025 16:17:01 GMT

Hi @stellahe102,
I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please accept it as a solution and give it a 'Kudos' so other community members with similar problems can find a solution faster.
Thank you.