topic Re: Dataflow Gen2 :Unable to create a table on the Lakehouse SQL catalog due to metadata refresh failure in Pipelines

Dataflow Gen2 :Unable to create a table on the Lakehouse SQL catalog due to metadata refresh failure

mandania — Tue, 03 Dec 2024 14:43:30 GMT

My DF Gen2 pipeline had been running smoothly for few months now and suddently after 15th november it stopped working (MS Fabric had released an update??). I have tried everything to fix the problem but its in vain. This is an internal error that only microsoft can help. this is the error message if it help.

"kontitrak: Error Code: TridentLakeClientUserException, Error Details: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataSource.Error: Failed to insert a table., InnerException: Unable to create a table on the Lakehouse SQL catalog due to metadata refresh failure, for Lakehouse Id: #{0} and Batch Id: #{1}. underlying error code: '#{2}', error: #{3}, Underlying error: Unable to create a table on the Lakehouse SQL catalog due to metadata refresh failure, for Lakehouse Id: b7314f1d-abb8-40b7-b722-6f6d45wc7144 and Batch Id: b7314f1d-abb8-40b7-b722-45w23qwc714@22c30cb9-4b0b-4a66-8175-14c40f76a01a$2024-12-03T11:40:39.4535432Z@f296f27b-ce8a-461a-b97e-45w23qwc714. underlying error code: 'TridentLakeClientUserException', error: [error=[code=TridentLakeClientUserException,pbi.error=[code=TridentLakeClientUserException,parameters=[ErrorMessage=...,HttpStatusCode=...],details={},exceptionCulprit=1]]] Details: Reason = DataSource.Error;ErrorCode = Lakehouse036;Message = Unable to create a table on the Lakehouse SQL catalog due to metadata refresh failure, for Lakehouse Id: b7314f1d-abb8-40b7-b722-45w23qwc714 and Batch Id: b7314f1d-abb8-40b7-b722-45w23qwc714@22c30cb9-4b0b-4a66-8175-14c40f76a01a$2024-12-03T11:40:39.4535432Z@f296f27b-ce8a-461a-b97e-45w23qwc714. underlying error code: 'TridentLakeClientUserException', error: [error=[code=TridentLakeClientUserException,pbi.error=[code=TridentLakeClientUserException,parameters=[ErrorMessage=...,HttpStatusCode=...],details={},exceptionCulprit=1]]];Detail = [error = [...]];Message.Format = Unable to create a table on the Lakehouse SQL catalog due to metadata refresh failure, for Lakehouse Id: #{0} and Batch Id: #{1}. underlying error code: '#{2}', error: #{3};Message.Parameters = {"b7314f1d-abb8-40b7-b722-45w23qwc714", "b7314f1d-abb8-40b7-bgh6-45w23qwc714@22c30cb9-4b0b-4a66-8175-14c40f76a01a$2024-12-03T11:40:39.4535432Z@f296f27b-ce8a-461a-b97e-45w23qwc714", "TridentLakeClientUserException", "[error=[code=TridentLakeClientUserException,pbi.error=[code=TridentLakeClientUserException,parameters=[ErrorMessage=...,HttpStatusCode=...],details={},exceptionCulprit=1]]]"};ErrorCode = Lakehouse045;Microsoft.Data.Mashup.Error.Context = System (Request ID: 7f39f621-9d5a-4d00-93c9-d2cc81aa8e14)."

Re: Dataflow Gen2 :Unable to create a table on the Lakehouse SQL catalog due to metadata refresh failure

lbendlin — Tue, 03 Dec 2024 20:59:15 GMT

If you have a Pro license you can open a Pro ticket at https://admin.powerplatform.microsoft.com/newsupportticket/powerbi
Otherwise you can raise an issue at https://community.fabric.microsoft.com/t5/Issues/idb-p/Issues .

Re: Dataflow Gen2 :Unable to create a table on the Lakehouse SQL catalog due to metadata refresh failure

Anonymous — Wed, 04 Dec 2024 06:50:41 GMT

Hi mandania ,

Based on the error message and the information provided, the issue appears to be related to a failed metadata refresh when attempting to create a table in the Lakehouse SQL Catalog.

The error message instructs TridentLakeClientUserException that if you already have the table in Lakehouse, then simply drag and drop it and recreate it. You must have the exact same schema, or if the schema of the data source has changed, then you may need to update the mapping of the dataflow to reflect the new structure.

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Dataflow Gen2 :Unable to create a table on the Lakehouse SQL catalog due to metadata refresh failure

mandania — Thu, 05 Dec 2024 07:35:41 GMT

Unfortunately i had to create a new workspace and recreate everything from the beginning. Dropping and recreating the tables was not an option since i was not able to do it. I sincerely hope that Fabric team fixes this issue because its not a very good experience for us.

Re: Dataflow Gen2 :Unable to create a table on the Lakehouse SQL catalog due to metadata refresh failure

Anonymous — Thu, 05 Dec 2024 07:42:32 GMT

Hi mandania ,

Thank you very much for your feedback, we will take this under advisement and get this back to the product team!

Best regards,

Adamk Kong

Re: Dataflow Gen2 :Unable to create a table on the Lakehouse SQL catalog due to metadata refresh failure

mandania — Thu, 05 Dec 2024 07:42:37 GMT

Thank you lbendlin

Re: Dataflow Gen2 :Unable to create a table on the Lakehouse SQL catalog due to metadata refresh failure

mandania — Fri, 06 Dec 2024 10:55:04 GMT

I was able to get some help from the fabric support and they managed to solve the problem. This is what they replied "In the previous cases, Product team has identified this bug is caused due to ownership of AD token failure and they have resolved it and took actions so that it will not occur again" My guess is this, the lakehouse that was throwing the error was owned by one of my other colleague who no longer works for us. We had removed his account (that was used to create the LH) and i think this caused a chain of error. unfortunately LH ownership is not something that you can change on fly, so you have to keep in mind that if the account holder is no longer with the compnay , you should probably not remove this person account until fabric comes with a solution.

Re: Dataflow Gen2 :Unable to create a table on the Lakehouse SQL catalog due to metadata refresh failure

Anonymous — Mon, 09 Dec 2024 05:36:23 GMT

Hi mandania ,

Your comments and feedback are valuable to us and very much appreciated.

Best regards,

Adamk Kong