topic Re: Running Vacuum on Lakehouse table throws error in Data Engineering

Running Vacuum on Lakehouse table throws error

govindarajan_d — Sun, 17 Dec 2023 19:04:56 GMT

Hi All,

I have been trying to run vacuum on multiple tables using both Adhoc maintenance option as well as Spark and I am getting the following error:

These tables are updated by Dataflow Gen 2 on a daily basis. The dataflow was running fine without issues for past 1 month.

Re: Running Vacuum on Lakehouse table throws error

AndyDDC — Sun, 17 Dec 2023 14:59:06 GMT

Hi @govindarajan_d has the VACUUM command ever worked on these tables?

Re: Running Vacuum on Lakehouse table throws error

govindarajan_d — Sun, 17 Dec 2023 19:06:16 GMT

Hi @AndyDDC ,

This is the first time I am running VACUUM on these tables.

For few other tables that have been updated by Spark, VACUUM works fine. Two tables that are updated by Dataflow Gen 2 throws the error.

Re: Running Vacuum on Lakehouse table throws error

AndyDDC — Mon, 18 Dec 2023 12:47:09 GMT

@govindarajan_d I've tested using a Dataflow Gen2 to load into a Delta table and then ran the VACUUM command and it worked successfully. What version of Spark are you using in the workspace that the lakehouse is in?

Re: Running Vacuum on Lakehouse table throws error

Anonymous — Tue, 19 Dec 2023 09:35:12 GMT

Hi @govindarajan_d ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet . Did you got a chance to look into @AndyDDC reply?
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

Re: Running Vacuum on Lakehouse table throws error

Anonymous — Thu, 21 Dec 2023 06:37:22 GMT

Hi @govindarajan_d ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

Re: Running Vacuum on Lakehouse table throws error

govindarajan_d — Fri, 22 Dec 2023 17:48:00 GMT

Hi @AndyDDC. 3.4 version of Spark. And yes, it works for many other tables, but not these 2 tables

Re: Running Vacuum on Lakehouse table throws error

Anonymous — Fri, 22 Dec 2023 06:54:40 GMT

Hi @govindarajan_d ,

Apologies for the issue you have been facing.

This might require a deeper investigation from our engineering team and they can guide you better for the issue with those two specified tables.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support
Please provide the ticket number here as we can keep an eye on it.

Re: Running Vacuum on Lakehouse table throws error

govindarajan_d — Fri, 22 Dec 2023 17:48:49 GMT

Support request number: 2312220040004539

Re: Running Vacuum on Lakehouse table throws error

Anonymous — Tue, 26 Dec 2023 07:40:05 GMT

Thanks for sharing the request number.

Hope you got some insights over your query. Please continue using Fabric Community for your further queries.

Re: Running Vacuum on Lakehouse table throws error

VincentMeyers — Wed, 24 Jan 2024 12:54:07 GMT

I got exactly the same problem for all my tables in my lakehouse, coming from dataflows

Re: Running Vacuum on Lakehouse table throws error

Anonymous — Thu, 25 Jan 2024 13:33:02 GMT

I got the same problem