topic Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table in Data Engineering

Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

Anonymous — Fri, 09 Feb 2024 02:34:27 GMT

I have no idea how it got into this state. Any help with fixing this would be appreciated.

I'm getting this error when i try and view a table in a datalake or manually run a vacuum

Delta table 'zsd_ship' has atleast '100' transaction logs, since last checkpoint. For performance reasons, it is recommended to regularly checkpoint the delta table more frequently than every '100' transactions. As a workaround, please use SQL or Spark to retrieve table schema.

The "_last_checkpoint" file contents are:

{"version":10,"size":4}

However the last checkpoint is file is file "00000000000000000340.checkpoint.parquet"

Can i just change the version in the last checkpoint file?

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

Anonymous — Fri, 09 Feb 2024 05:44:00 GMT

Hi @Anonymous ,

Thanks for using Fabric Community.
Can you please help me understand the issue?
What was the last operation you performed and from when you are facing this issue?
What was the error you encountered while using VACCUM command?

Try to share the output of this command -

It would be helpful if you can share few screenshots.

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

audrianna21 — Fri, 09 Feb 2024 22:22:21 GMT

@Anonymous,

We've been experiencing the same thing for the past 5 days. The Lakehouse we are seeing this issue in has close to 40 delta tables. This error is only occuring on the older delta tables that have had Dataflow Gen2 flows loading data to them for the last 30 days. I've created some newer delta tables in the last couple days. The new tables aren't having the same issue.

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

Anonymous — Mon, 12 Feb 2024 02:45:48 GMT

Hi @Anonymous

I nuked the original table and recreated it (problem gone), however, another Datalake we are running in the same way is having this issue as well.

The Datalake is updated via a Gen2 Dataflow that runs hourly and is replacing tables.

Each one of the tables in the Datalake has this error message when you hover over the red exclimation mark on the table.

Results of the Spark SQL for the last ~115 rows:

Results of the failed manual table vacuum

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

Anonymous — Mon, 12 Feb 2024 09:24:04 GMT

Hi @Anonymous , @audrianna21

Thanks for using Fabric community.

Apologies for the issue you have been facing. I would like to check are you still facing this issue?

It's difficult to tell what could be the reason for this performance.

If the issue still persists, please reach out to our support team so they can do a more thorough investigation on why this it is happening. If its a bug, we will definitely would like to know and properly address it.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thank you.

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

Anonymous — Tue, 13 Feb 2024 08:09:01 GMT

Hello @Anonymous ,

We haven’t heard from you on the last response and was just checking back to see if you got a chance to create a support ticket.
In case if support ticket is created please provide the ticket number as it would help us to track for more information.
Thanks

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

jmalone — Wed, 14 Feb 2024 00:14:40 GMT

@Anonymous I am seeing the exact same issue. It is appearing on older tables that were loaded via Dataflows Gen2. The message just started appearing in the past few days.

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

audrianna21 — Wed, 14 Feb 2024 00:20:13 GMT

I opened at support ticket yesterday. Someone contacted me to say they would look into it and get back to me. I haven't heard back.

i can confirm if you delete the table and recreate it the error goes away but that's not a feasible option for me.

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

Maderthaner — Wed, 14 Feb 2024 12:37:46 GMT

Same issue here.

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

Anonymous — Wed, 14 Feb 2024 13:07:01 GMT

Hi @audrianna21 ,

Can you please share the support ticket number, I will try to get any updates from the team internally?

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

audrianna21 — Wed, 14 Feb 2024 13:11:06 GMT

The support request ID is: 2402120040003273.

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

Anonymous — Fri, 16 Feb 2024 14:40:21 GMT

Hi @audrianna21 ,

I haven't got any updates internally. Incase if there is any update I will share it over here.

Thank you

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

Anonymous — Sun, 18 Feb 2024 21:38:11 GMT

Hi @Anonymous

Appologies, I was on leave. I've raised a support ticket: Case 2402180030001106

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

kblackburn — Tue, 20 Feb 2024 02:10:14 GMT

I am having the same issue. I noticed that my Gen2 flow had not appended any data to my lakehouse table in several days. Then today is the first day that I see the error message. I can also confim that if you create a new table, the flow works and the lakehouse table is rendered without error.

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

KA78 — Wed, 06 Mar 2024 07:21:59 GMT

Perhaps any news on this topic? I'm having the same issue.

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

kblackburn — Wed, 06 Mar 2024 15:12:57 GMT

Hello. I submitted a ticket yesterday. I'll update here when I have some additional insight. As a follow up, after a week or so, I was able to run optimizer and vacuum successfully but the sql endpoit and semantic model are not updating / not in sync with lakehouse

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

kblackburn — Fri, 08 Mar 2024 22:43:51 GMT

I received a support call yesterday and they had me run this code in a notebook, then refresh the lakehouse table that had the error, and it worked. Maybe it will work for you. Good luck!

%%spark

import org.apache.spark.sql.delta.DeltaLog

DeltaLog.forTable(spark,"Tables/yourtablenamehere").checkpoint()

@KA78

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

JFTxJ — Mon, 11 Mar 2024 15:28:59 GMT

This worked for me.

For anyone getting an error looking for the org.apache.spark.sql.delta.DeltaLog import, make sure you include the first line "%%spark" indicating that this is Scala code (not pyspark).

@kblackburn I have hundreds of tables with this issue, is-there an easy to to have this code run for every table in my lakehouse? I am not very familiar with Scala, so your help would be appreciated.

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

KA78 — Wed, 13 Mar 2024 20:49:18 GMT

Thanks kblackburn ! This is a good workaround. Thanks for sharing! I hope they'll fix this soon. Perhaps any news on a fix?

Re: Error DeltaTableIsInfrequentlyCheckpointed when accessing lakehouse table

Anonymous — Wed, 13 Mar 2024 19:45:19 GMT

This will iterate through all tables for all lakehouses (except the dataflows staging one) in the same workspace as the default lakehouse for the notebook.

%%spark import org.apache.spark.sql.delta.DeltaLog val lakehouses = spark.catalog.listDatabases() lakehouses.collect().sortWith(_.name < _.name).foreach { lakehouse => if (lakehouse.name != "DataflowsStagingLakehouse") { val tables = spark.catalog.listTables(lakehouse.name) tables.collect().sortWith(_.name < _.name).foreach { table => DeltaLog.forTable(spark, s"${lakehouse.locationUri}/${table.name}").checkpoint() println(s"Completed code run for lakehouse: ${lakehouse.name}, table: ${table.name}") } } }