Couldn't refresh the entity because of an issue with the mashup document

[je777999]

Hi , 

I am new with MS Fabric, contiuously practicing and able to create a dataflowgen2 with data destination warehouse/lakehouse and have a PowerBI connect to it. 

Now I have a dataflowgen2 that will create a new table based on the data it will get from datasource, but i am getting this error "Error Code: Mashup Exception Data Source Error, Error Details: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataSource.Error: Failed to insert a table., InnerException: Unable to create a table on the Lakehouse SQL catalog due to metadata refresh failure," 

What could be the reason of getting the issue?

[je777999 · Accepted solution]

Hi guys,

Thanks for your message 😃

I am not sure why, but i think i know what to do on my case, the issue issue only appear on my end if i click the publish button as soon as i update the DataFLow Gen 2. 

What i do now is after updating Dataflow Gen2,

- i don't click click the Publish button right away,

- I click other tables first,

- close the Dataflow Gen2 to see the workspace again

- find the updated Dataflow Gen2

- wait for few minutes(1-5) because MS Fabric is slow in everything, i dont know if this is because im using Free Trial

- after waiting for few minutes, near the name of my dataflow gen 2 iclick the "Refresh now" button, the one that looks like circle

it is crazy because it works on my end. it seems there is a glitch or delay issue with MS Fabric.

The only issue now is when the dataflowgen2 is triggered by Pipeline, sometimes 'failed' sometimes 'Succeeded'

[jasiluka12]

in my case i was having the same error when using Warehouse as data destination.

as soon as i changed from Warehouse to Lakehouse , the issue was gone..

Seems like a totally random error as my tables had very minimal transformations with no concatenation between multiple columns. In fact it had no new columns created at all... Almost feels like this is a bug of some sort.

[je777999 · Accepted solution]

Hi guys,

Thanks for your message 😃

I am not sure why, but i think i know what to do on my case, the issue issue only appear on my end if i click the publish button as soon as i update the DataFLow Gen 2. 

What i do now is after updating Dataflow Gen2,

- i don't click click the Publish button right away,

- I click other tables first,

- close the Dataflow Gen2 to see the workspace again

- find the updated Dataflow Gen2

- wait for few minutes(1-5) because MS Fabric is slow in everything, i dont know if this is because im using Free Trial

- after waiting for few minutes, near the name of my dataflow gen 2 iclick the "Refresh now" button, the one that looks like circle

it is crazy because it works on my end. it seems there is a glitch or delay issue with MS Fabric.

The only issue now is when the dataflowgen2 is triggered by Pipeline, sometimes 'failed' sometimes 'Succeeded'

[je777999 · Accepted solution]

To Add, the problem seems to get solved after disabling staging of power query. Just by disabling staging, our loading speed was down to average less than 5 minutes and rarely get the error.

[v-nikhilan-msft]

Hi @je777999  ,
Thanks for using Fabric Community.

There are a few possible reasons why you might be getting this error:

1. You are trying to create a table with a non-nullable column that contains null values: Dataflow Gen 2 does not support creating tables with non-nullable columns that contain null values. To fix this, you can either remove the non-nullable constraint from the column, or you can remove the null values from the data before running the dataflow.
2. You are trying to create a table with a column type that is not supported by the Lakehouse SQL catalog:  Dataflow Gen 2 supports a limited set of data types for Lakehouse SQL tables. If you are trying to create a table with a column type that is not supported, you will need to change the column type to a supported type. You can refer to this link for understanding Dataflow Gen2 limitations: Link1

Please refer to these links for more help:
DataFlow Gen2
Similar Issue
On-Prem Gateway Errors

Hope this helps. Please let us know if you have any further queries.

[KNP]

Hi @v-nikhilan-msft, 

Any advice on how to achieve number 1? How do you remove the constraint?

Thanks.

[20231215150134]

Hi Nikhilan,

Thanks for your comment.

In #1, how does one "remove the non-nullable constraint from the column" that is causing the dataflow Gen 2 to fail? 

For now, I will replace null with 'Unknown', but I would be very much interested in the alternative that saves time and resources.

Thanks!

[v-nikhilan-msft]

Hi @je777999 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Please let us know if you have any further queries.

[DavidSerna]

I have the same error,

jornada: Código de error: 999999, Detalles del error: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: No se puede cargar el archivo o ensamblado 'ParquetSharp, Version=0.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' ni una de sus dependencias. El sistema no puede encontrar el archivo especificado. Details: GatewayObjectId: 29a32c17-fdc7-4294-9961-d270354f1e9b (Id. de solicitud: 48d87264-360e-4d04-b47f-aac70f3fdc86).

[je777999]

We use Lakehouse and ensure that the DataflowGen2's Staging is set to disable. By default Staging is enabled. 

Current setup

DataflowGen2 - use as ETL

Lakehouse - our data destination

never encountered the issue again, after disabling staging 😃 hope works on your end as well 😃