Couldn't refresh the entity because of an issue with the mashup document

[je777999]

Hi , 

I am new with MS Fabric, contiuously practicing and able to create a dataflowgen2 with data destination warehouse/lakehouse and have a PowerBI connect to it. 

Now I have a dataflowgen2 that will create a new table based on the data it will get from datasource, but i am getting this error "Error Code: Mashup Exception Data Source Error, Error Details: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataSource.Error: Failed to insert a table., InnerException: Unable to create a table on the Lakehouse SQL catalog due to metadata refresh failure," 

What could be the reason of getting the issue?

[je777999 · Accepted solution]

Hi guys,

Thanks for your message 😃

I am not sure why, but i think i know what to do on my case, the issue issue only appear on my end if i click the publish button as soon as i update the DataFLow Gen 2. 

What i do now is after updating Dataflow Gen2,

- i don't click click the Publish button right away,

- I click other tables first,

- close the Dataflow Gen2 to see the workspace again

- find the updated Dataflow Gen2

- wait for few minutes(1-5) because MS Fabric is slow in everything, i dont know if this is because im using Free Trial

- after waiting for few minutes, near the name of my dataflow gen 2 iclick the "Refresh now" button, the one that looks like circle

it is crazy because it works on my end. it seems there is a glitch or delay issue with MS Fabric.

The only issue now is when the dataflowgen2 is triggered by Pipeline, sometimes 'failed' sometimes 'Succeeded'

[prathijp]

I also faced the sample error today. Again even if the dataflow was giving error, the data was getting loaded. For me staging was disabled already. However I got the error corrected and the dataflow succeeded when I renamed all the queries on left side and removed the space between words. Like Transform File  to TransformFile, Sample File to SampleFile. Not sure if this is a similar case being discussed here. But this worked for me.

[aeelissa]

Hello @Anonymous , I faced the same somehow, the destination tables are being updated but I still get an error code and that the flow failed:

Any idea of how to solve it?
Thanks in advance!

[TimoRiikonen]

Here is one good way to test if this is a defect in Fabric or in your own code after you have disabled staging:
- Filter errors out from your results.
I would expect all of the code will result with something to store to the Lakehouse, so probably it should valid it through.

[dsajda]

Hi guys, I am doing the Tutorial: Microsoft Fabric for Power BI users - Power BI | Microsoft Learn but I have the same issue when I clic on publish it fails, I already tried next solutions;

1. 

What i do now is after updating Dataflow Gen2,

- i don't click click the Publish button right away,

- I click other tables first,

- close the Dataflow Gen2 to see the workspace again

- find the updated Dataflow Gen2

- wait for few minutes(1-5) because MS Fabric is slow in everything, i dont know if this is because im using Free Trial

- after waiting for few minutes, near the name of my dataflow gen 2 iclick the "Refresh now" button, the one that looks like circle

2. To Add, the problem seems to get solved after disabling staging of power query. Just by disabling staging, our loading speed was down to average less than 5 minutes and rarely get the error.

But any of them works, could please someone help me ?

When I clic on pulish appear this error.

[je777999]

Hi @dsajda,

Good day,

Upon checking the screenshot you've shared, It appears to be not the same with the error mentioned in this page. 

You got:

Expression: We cannot convert a value of type Table to type Function.

Details

   Reason = Expression.Error

   Value = #table ({} {})

Please review your power query and see which part was set trying to convert a value of type Table into type Function. 

[AmirZ]

Hi there,

When I try to disable DataflowGen2's Staging, it removes the destination Warehouse!
How do you solve this issue?

[jasiluka12]

in my case i was having the same error when using Warehouse as data destination.

as soon as i changed from Warehouse to Lakehouse , the issue was gone..

Seems like a totally random error as my tables had very minimal transformations with no concatenation between multiple columns. In fact it had no new columns created at all... Almost feels like this is a bug of some sort.

[je777999 · Accepted solution]

Hi guys,

Thanks for your message 😃

I am not sure why, but i think i know what to do on my case, the issue issue only appear on my end if i click the publish button as soon as i update the DataFLow Gen 2. 

What i do now is after updating Dataflow Gen2,

- i don't click click the Publish button right away,

- I click other tables first,

- close the Dataflow Gen2 to see the workspace again

- find the updated Dataflow Gen2

- wait for few minutes(1-5) because MS Fabric is slow in everything, i dont know if this is because im using Free Trial

- after waiting for few minutes, near the name of my dataflow gen 2 iclick the "Refresh now" button, the one that looks like circle

it is crazy because it works on my end. it seems there is a glitch or delay issue with MS Fabric.

The only issue now is when the dataflowgen2 is triggered by Pipeline, sometimes 'failed' sometimes 'Succeeded'

[samarsouiss]

Hi ,  

I have a similar error 

Un problème s’est produit lors de l’actualisation du flux de données : « Something went wrong, please try again later. If the error persists, please contact support. ». Code d’erreur : ActionUserFailure. (ID de requête : d9dc4255-48c8-41ff-b82b-4e5adf914ebb).

Exchange_2024_WriteToDataDestination : Un problème s’est produit lors de l’actualisation du flux de données : « Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: Erreur lors du remplacement du contenu de la table par de nouvelles données dans une version : #{0}., Erreur sous-jacente : Parquet : class parquet::ParquetStatusException (message: 'Unknown error: Microsoft.Mashup.Engine1.Runtime.ValueException: [DataSource.Error] AzureDataLakeStorage n'a pas r�ussi � obtenir de r�ponse�: ��La demande a �t� abandonn�e�: La demande a �t� annul�e.��. � Microsoft.Internal.ParallelAppender`1.Dispose() � Microsoft.Mashup.Common.StreamExtensions.<>c__DisplayClass11_0.<AfterDispose>b__0() � Microsoft.Mashup.Common.StreamExtensions.NotifyingStream.HandleCallback() � Microsoft.Mashup.Common.StreamExtensions.NotifyingStream.Close() � ParquetSharp.IO.ManagedOutputStream.Close(String& exception) Record') Details: Reason = DataFormat.Error;ErrorCode = Lakehouse036;Message = Parquet : class parquet::ParquetStatusException (message: 'Unknown error: Microsoft.Mashup.Engine1.Runtime.ValueException: [DataSource.Error] AzureDataLakeStorage n'a pas r�ussi � obtenir de r�ponse�: ��La demande a �t� abandonn�e�: La demande a �t� annul�e.��. � Microsoft.Internal.ParallelAppender`1.Dispose() � Microsoft.Mashup.Common.StreamExtensions.<>c__DisplayClass11_0.<AfterDispose>b__0() � Microsoft.Mashup.Common.StreamExtensions.NotifyingStream.HandleCallback() � Microsoft.Mashup.Common.StreamExtensions.NotifyingStream.Close() � ParquetSharp.IO.ManagedOutputStream.Close(String& exception) Record');Message.Format = Parquet : class parquet::ParquetStatusException (message: 'Unknown error: Microsoft.Mashup.Engine1.Runtime.ValueException: [DataSource.Error] AzureDataLakeStorage n'a pas r�ussi � obtenir de r�ponse�: ��La demande a �t� abandonn�e�: La demande a �t� annul�e.��. � Microsoft.Internal.ParallelAppender`1.Dispose() � Microsoft.Mashup.Common.StreamExtensions.<>c__DisplayClass11_0.<AfterDispose>b__0() � Microsoft.Mashup.Common.StreamExtensions.NotifyingStream.HandleCallback() � Microsoft.Mashup.Common.StreamExtensions.NotifyingStream.Close() � ParquetSharp.IO.ManagedOutputStream.Close(String& exception) Record');Microsoft.Data.Mashup.Error.Context = User GatewayObjectId: 77e723d8-6485-4042-847a-90f74dfb3b18 ». Code d’erreur : Mashup Exception Data Format Error. (ID de requête : d9dc4255-48c8-41ff-b82b-4e5adf914ebb).

 

In my case i want to extract data from SQL Server via connetion Gateway to Fabric lakehouse

I don't know what is the problem . i tried everything : recreate dataflow ,lakehouse ,update gateway recreate connection nothing works ! please help me ! Thank u in advance

[je777999 · Accepted solution]

To Add, the problem seems to get solved after disabling staging of power query. Just by disabling staging, our loading speed was down to average less than 5 minutes and rarely get the error.

[bkgulshan11]

Thank you that Help 😀

[Anonymous]

Hi @je777999  ,
Thanks for using Fabric Community.

There are a few possible reasons why you might be getting this error:

1. You are trying to create a table with a non-nullable column that contains null values: Dataflow Gen 2 does not support creating tables with non-nullable columns that contain null values. To fix this, you can either remove the non-nullable constraint from the column, or you can remove the null values from the data before running the dataflow.
2. You are trying to create a table with a column type that is not supported by the Lakehouse SQL catalog:  Dataflow Gen 2 supports a limited set of data types for Lakehouse SQL tables. If you are trying to create a table with a column type that is not supported, you will need to change the column type to a supported type. You can refer to this link for understanding Dataflow Gen2 limitations: Link1

Please refer to these links for more help:
DataFlow Gen2
Similar Issue
On-Prem Gateway Errors

Hope this helps. Please let us know if you have any further queries.

[NeilJustice]

Did anyone get an answer to resolution number 1 in your response : 

1. You are trying to create a table with a non-nullable column that contains null values: Dataflow Gen 2 does not support creating tables with non-nullable columns that contain null values. To fix this, you can either remove the non-nullable constraint from the column, or you can remove the null values from the data before running the dataflow.

How do you remove the non nullable values? Thx in advance

[KNP]

Hi @Anonymous, 

Any advice on how to achieve number 1? How do you remove the constraint?

Thanks.

[20231215150134]

Hi Nikhilan,

Thanks for your comment.

In #1, how does one "remove the non-nullable constraint from the column" that is causing the dataflow Gen 2 to fail? 

For now, I will replace null with 'Unknown', but I would be very much interested in the alternative that saves time and resources.

Thanks!

[Anonymous]

Hi @je777999 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Please let us know if you have any further queries.

[DavidSerna]

I have the same error,

jornada: Código de error: 999999, Detalles del error: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: No se puede cargar el archivo o ensamblado 'ParquetSharp, Version=0.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' ni una de sus dependencias. El sistema no puede encontrar el archivo especificado. Details: GatewayObjectId: 29a32c17-fdc7-4294-9961-d270354f1e9b (Id. de solicitud: 48d87264-360e-4d04-b47f-aac70f3fdc86).

[je777999]

We use Lakehouse and ensure that the DataflowGen2's Staging is set to disable. By default Staging is enabled. 

Current setup

DataflowGen2 - use as ETL

Lakehouse - our data destination

never encountered the issue again, after disabling staging 😃 hope works on your end as well 😃