topic Re: cannot retreive lakehouse table schema(columns) in Data Warehouse

cannot retreive lakehouse table schema(columns)

Anonymous — Tue, 03 Dec 2024 12:41:21 GMT

hello,

I am using Fabric lakehouse to store MS SQL server cdc tables(integrate from on-prem SQL server)

it works good for a long time. Today it suddently shows me this:

Click on details:

While the error does not impact the table reading. I can successfuly query the tables or read the tables thru notebook.

Any thoughts on this?

Thanks a lot.

Re: cannot retreive lakehouse table schema(columns)

AndyDDC — Tue, 03 Dec 2024 22:35:11 GMT

Hi Anonymous have you run any maintenance on the Delta tables?

Delta table maintenance in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

Delta Lake table optimization and V-Order - Microsoft Fabric | Microsoft Learn

Re: cannot retreive lakehouse table schema(columns)

Anonymous — Wed, 04 Dec 2024 02:46:57 GMT

hello AndyDDC

Thanks for you suggestion.

I tried to run maintenance for my delta table:

And it always failed like below shown.

Click Details :

Re: cannot retreive lakehouse table schema(columns)

Anonymous — Wed, 04 Dec 2024 06:44:53 GMT

Hi Anonymous ,

Thanks for the reply from AndyDDC .

This error occurs when the Delta table has accumulated at least 100 transaction logs since the last checkpoint.

Run this code in your notebook and then refresh the Lakehouse table where the error occurs to check if it is working correctly:

%%spark import org.apache.spark.sql.delta.DeltaLog DeltaLog.forTable(spark, “Tables/yourtablenamehere”).checkpoint()

Here's a user with the same problem as you, check his reply to see if it helps you.

Error DeltaTableIsInfrequentlyCheckpointed when ac... - Page 2 - Microsoft Fabric Community

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: cannot retreive lakehouse table schema(columns)

Anonymous — Wed, 04 Dec 2024 06:56:12 GMT

Thanks Yang.

I run above code and got below error msg:

Re: cannot retreive lakehouse table schema(columns)

AndyDDC — Wed, 04 Dec 2024 09:35:12 GMT

How did the original tables get created? Are they native lakehouse delta tables?

Re: cannot retreive lakehouse table schema(columns)

Anonymous — Wed, 04 Dec 2024 12:30:44 GMT

They are originally pulled from SQL server CDC tables. They are created by Fabric pipeline thru a copy activity.

Re: cannot retreive lakehouse table schema(columns)

Anonymous — Thu, 05 Dec 2024 08:16:30 GMT

Hi Anonymous ,

This error usually occurs due to a permissions issue.

Double-check that the name is entered correctly; any misspellings or incorrect references can cause permissions issues.

Ensure that the user or service principal running the Spark job has the Storage Blob Data Contributor role for the ADLS Gen2 storage account.

For more details, please see:

Handle Livy errors on Apache Spark in Synapse - Azure Synapse Analytics | Microsoft Learn

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

Re: cannot retreive lakehouse table schema(columns)

Anonymous — Thu, 05 Dec 2024 08:30:18 GMT

hello, Yang

Thanks for you reply.

1. You can see clearly the table name is exactly the same as the name on the left side of the screen. No mistake.

2. I can run other all spark jobs successfully in the same notebook same Fabric worksapce. Why do I need "the Storage Blob Data Contributor role for the ADLS Gen2 storage account" for running this code block you mentioned?

Re: cannot retreive lakehouse table schema(columns)

Anonymous — Tue, 17 Dec 2024 06:44:43 GMT

Hi Anonymous ,

Yes, I can see that the table name is exactly the same as the name on the left side of the screen.

I can't reproduce your error, the solution mentioned is because the solution given for the same error message mentioned in the official documentation is that one, which didn't help you unfortunately.

Due to the fact that “you can successfully run all other Spark jobs from the same laptop” and that “it has worked for a long time before and this error is recent”, it is not possible to determine what the problem is. I think you can create a support ticket for free and a dedicated Microsoft engineer will resolve the issue for you.

It would be great if you continue to share in this issue to help others with similar problems after you know the root cause or solution.

The link of Power BI Support: https://powerbi.microsoft.com/en-us/support/

For how to create a support ticket, please refer to How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

Thank you for your understanding.

Best Regards,
Yang
Community Support Team

Re: cannot retreive lakehouse table schema(columns)

Anonymous — Wed, 25 Dec 2024 07:38:46 GMT

Hi Anonymous ,

Is my follow-up just to ask if the problem has been solved?

If so, can you accept the correct answer as a solution or share your solution to help other members find it faster?

Thank you very much for your cooperation!

Best Regards,
Yang
Community Support Team

Re: cannot retreive lakehouse table schema(columns)

Anonymous — Wed, 25 Dec 2024 07:46:28 GMT

not fixed. still got problem.