Dataflow gen2 refresh Error

[smpa01]

There was a problem refreshing the dataflow: 'Something went wrong, please try again later. If the error persists, please contact support.'. Error code: EntityUserFailure. (Request ID: 460e8fb5-e1f7-4477-a39e-2e720a3b56e3).

Apply_tbl_Activity_Incremental_Over_Periods: There was a problem refreshing the dataflow: 'Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: The value does not support versioning. Details: Reason = Expression.Error;ErrorCode = 10855;Detail = #table({"ID", "DateTime", "Performed By", "Action", "Comments", "UpdateDate", "DWH Last Updated At", "for_filtering_ir"}, {});Microsoft.Data.Mashup.Error.Context = User GatewayObjectId: 08a260a3-97fa-4307-bb55-6046f036f912'. Error code: 104100. (Request ID: 460e8fb5-e1f7-4477-a39e-2e720a3b56e3).

Dataflow gen2 fails with the above error. 

I am ingesting dat from on prem sql server with incremental refresh to a lakehouse

[v-veshwara-msft]

Hi @smpa01 ,
Thanks for engaging with Microsoft Fabric Community.
Just checking in to see if your issue is resolved and if you’ve had a chance to raise a support ticket. If the issue has been resolved, it would be helpful if you could share the resolution steps here for the benefit of the wider community.

I also followed the repro steps you shared and was able to observe the same behavior on my end. The error still occurred even after creating a new table and using a new Dataflow Gen2.

Hope the support team is able to help resolve the issue.
Thanks @miguel, @LuitwielerMSFT and @wardy912 for addressing this and sharing valuable insights.

Please reach out for further assistance.

Thank you.

[oe_lopez]

I had the same error, but I had a similar dataflow gen 2 working, so I did very basic troubleshooting and removed all non text columns and started adding them and sure enough it was a number column that was giving me the error. Fixed that column and it all worked.

regards,

[smpa01]

I’m currently under significant time pressure to deliver a production-ready workflow for my client. Over the past two weeks, I’ve been evaluating Dataflow Gen2 for my use case, but have encountered several critical issues and unexpected behaviors:

- Incremental refresh does not function as expected

- Refresh operations result in Lakehouse tables with the correct schema and row count, but contain no data

- No REST API support for triggering refreshes using a service principal (SPN)

These challenges are difficult to diagnose or address intuitively, making DF Gen2 unreliable for my current complex production scenario.

While I plan to raise a formal support ticket once immediate client deliverables are met, I’ve documented the steps to reproduce these issues (outlined above). My current focus is to design a robust and scalable data engineering solution—ideally leveraging Notebooks as the primary mechanism, rather than relying on low-code artifacts like Dataflows or Pipelines. This is particularly important in the context of enterprise-level BI that integrates data from diverse sources, including on-prem systems, Databricks, Dremio, SharePoint, and others.

In contrast, Dataflow Gen1—though limited to producing flat tables and lacking SQL or Spark transformation capabilities—proved far more stable in practice. From a functional standpoint, it presented minimal issues. The unfortunate limitation is that it cannot insert data into a destination.

[v-veshwara-msft]

Hi @smpa01 ,

Thanks for the detailed follow-up and for sharing your observations. We completely understand the pressure of delivering a stable, production-ready solution, and we appreciate you taking the time to list out the issues and repro steps despite that.

It’s clear that some of the current limitations in Dataflow Gen2, especially around incremental refresh and service principal support for refresh, are making it difficult to use in your production scenario.

When you're able to, raising a support ticket will allow the engineering team to look into the issue more deeply. In the meantime, using notebooks could be a good alternative, especially since they offer more control and flexibility when working with different data sources like Databricks, SharePoint, and on-prem systems.

Thanks again for your feedback - it helps improve the experience for everyone. Feel free to share any further updates here.

[LuitwielerMSFT]

@smpa01 Hey there! I am PM for Incremental refresh for DFgen2. If you encounter issues, please feel free to open a support ticket to get to the bottom of this.

Looking at your scenario, we think that switching from non-incremental to incremental for the same table is causing the issue. If you check out the docs here: Incremental refresh in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

you see that we mention the following: 

• When you write to a lakehouse, the dataflow maintains it owns bookkeeping of the files that are written to the lakehouse. This is in line with the standard lakehouse pattern. However, this means that if other writers like Spark or other processes write to the same table, it may cause issues with the incremental refresh. We recommend that you don't use other writers to write to the same table while using incremental refresh. If you do, you need to ensure that the other writers don't interfere with the incremental refresh process. Processes like table maintenance and vacuuming are not supported either while using incremental refresh.

What probably happens in your situation is that you try to write to an existing table in which the files are not partitioned to the dates we use for incremental refresh. We advise if you need to have historic data loaded sepretly, to make that a seperate table and use a view that combines both tables into one. 

Hope this helps! and to better understand we need to have a support ticket such as we can dive deeper into your scenario 🙂 

[wardy912]

Hi @smpa01 

 Mashup error code 10855 is defined as 
Mashup Table: Appears to be an empty table with defined columns but no data.

Ensure your query returns at least one row by adding something like

if Table.IsEmpty(Source) then ... else ...

It would also be helpful if you paste the query into power BI so you can go through each transformation to identify the exact point the flow fails.

[smpa01]

Thanks for this. This table has incremental refresh baked in which might result in a an empty table. i am not sure how that needs to be handled for lakehouse ingestion.

This part should be internally baked in to incemental refresh - 

if Table.IsEmpty(Source) then ... else ...

[MG86]

I keep getting the same error too when trying to setup a Dataflow Gen2 with Incremental Refresh and a Lakehouse as destination.

I followed the guides from Microsoft, including making sure that the lakehouse table uses a Fixed schema and that the query fully folds. I've done multiple attempts but I keep getting the error: 

Apply_Activities_Incremental_Over_Periods: There was a problem refreshing the dataflow: 'Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: The value does not support versioning. Details: Reason = Expression.Error;ErrorCode = 10855;

[smpa01]

@miguel can you please provide some insights on to why df gen2 keeps on faikling for some tables from on prem sql db. I keep on getting this, I am trying to ingest a base table without incremental refresh and it is still failing

[miguel]

I'm not sure sure that we can help with the information shared on this thread. Do you have a way for us to repro the behavior or could you go into details as to how we could create something similar in our own environments ? Or perhaps a more detailed definition of your mashup document ?

assuming that you're using the latest version of the gateway, please raise a support ticket so our engineering team can take a closer look at this scenario 

[smpa01]

To reproduce

#1. Ingest a data from on prem sql server to a lakehouse with fixed schema and replace using a df gen2. Do not factor in incremental refresh.

There is no issue in this step.

#2. Go back and factor in incremental refresh in the same df gen2 and run the dataflow. It errors out in this stage with 10855.

#3. If you factor in incremental refresh in #1 before doing a base ingestion, it gets error out with tge same error.

What I desire is to be able to ingest from on prem sql server db to lakehouse with incremental baked in without any error.

[smpa01]

I keep on getting refresh error 104500, do you know what it is for?

There was a problem refreshing the dataflow: 'Something went wrong, please try again later. If the error persists, please contact support.'. Error code: 104500. (Request ID: f356ef9a-2194-46be-8aa2-72483c85d585).

dim_activity_WriteToDataDestination: There was a problem refreshing the dataflow: 'Something went wrong, please try again later. If the error persists, please contact support.'. Error code: 999999. (Request ID: f356ef9a-2194-46be-8aa2-72483c85d585).