cannot retreive lakehouse table schema(columns)

[cfccai]

hello,

I am using Fabric lakehouse to store MS SQL server cdc tables(integrate from on-prem SQL server)

it works good for a long time. Today it suddently shows me this: 

Click on details:

While the error does not impact the table reading. I can successfuly query the tables or read the tables thru notebook.

Any thoughts on this?

Thanks a lot.

[Anonymous]

Hi @cfccai ,

Is my follow-up just to ask if the problem has been solved?

If so, can you accept the correct answer as a solution or share your solution to help other members find it faster?

Thank you very much for your cooperation!

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[cfccai]

not fixed. still got problem.

[Anonymous]

Hi @cfccai ,

Thanks for the reply from AndyDDC .

This error occurs when the Delta table has accumulated at least 100 transaction logs since the last checkpoint.

Run this code in your notebook and then refresh the Lakehouse table where the error occurs to check if it is working correctly:

%%spark

import org.apache.spark.sql.delta.DeltaLog

DeltaLog.forTable(spark, “Tables/yourtablenamehere”).checkpoint()

Here's a user with the same problem as you, check his reply to see if it helps you.

Error DeltaTableIsInfrequentlyCheckpointed when ac... - Page 2 - Microsoft Fabric Community

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[cfccai]

Thanks Yang.

I run above code and got below error msg:

[Anonymous]

Hi @cfccai ,

This error usually occurs due to a permissions issue.

Double-check that the name is entered correctly; any misspellings or incorrect references can cause permissions issues.

Ensure that the user or service principal running the Spark job has the Storage Blob Data Contributor role for the ADLS Gen2 storage account.

For more details, please see:

Handle Livy errors on Apache Spark in Synapse - Azure Synapse Analytics | Microsoft Learn

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[cfccai]

hello, Yang

Thanks for you reply.

1. You can see clearly the table name is exactly the same as the name on the left side of the screen. No mistake.

2. I can run other all spark jobs successfully in the same notebook same Fabric worksapce. Why do I need "the Storage Blob Data Contributor role for the ADLS Gen2 storage account" for running this code block you mentioned? 

[Anonymous]

Hi @cfccai ,

Yes, I can see that the table name is exactly the same as the name on the left side of the screen.

I can't reproduce your error, the solution mentioned is because the solution given for the same error message mentioned in the official documentation is that one, which didn't help you unfortunately.

Due to the fact that “you can successfully run all other Spark jobs from the same laptop” and that “it has worked for a long time before and this error is recent”, it is not possible to determine what the problem is. I think you can create a support ticket for free and a dedicated Microsoft engineer will resolve the issue for you.

It would be great if you continue to share in this issue to help others with similar problems after you know the root cause or solution.

The link of Power BI Support:  https://powerbi.microsoft.com/en-us/support/

For how to create a support ticket, please refer to How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

Thank you for your understanding.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[AndyDDC]

Hi @cfccai have you run any maintenance on the Delta tables? 

Delta table maintenance in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

Delta Lake table optimization and V-Order - Microsoft Fabric | Microsoft Learn

[cfccai]

hello @AndyDDC 

Thanks for you suggestion. 

I tried to run maintenance for my delta table:

And it always failed like below shown.

Click Details :

[AndyDDC]

How did the original tables get created?  Are they native lakehouse delta tables?

[cfccai]

They are originally pulled from SQL server CDC tables.  They are created by Fabric pipeline thru a copy activity.