Forum Discussion
Dataflow gen2 refresh Error
There was a problem refreshing the dataflow: 'Something went wrong, please try again later. If the error persists, please contact support.'. Error code: EntityUserFailure. (Request ID: 460e8fb5-e1f7-4477-a39e-2e720a3b56e3).
Apply_tbl_Activity_Incremental_Over_Periods: There was a problem refreshing the dataflow: 'Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: The value does not support versioning. Details: Reason = Expression.Error;ErrorCode = 10855;Detail = #table({"ID", "DateTime", "Performed By", "Action", "Comments", "UpdateDate", "DWH Last Updated At", "for_filtering_ir"}, {});Microsoft.Data.Mashup.Error.Context = User GatewayObjectId: 08a260a3-97fa-4307-bb55-6046f036f912'. Error code: 104100. (Request ID: 460e8fb5-e1f7-4477-a39e-2e720a3b56e3).
Dataflow gen2 fails with the above error.
I am ingesting dat from on prem sql server with incremental refresh to a lakehouse
14 Replies
Hi smpa01
Mashup error code 10855 is defined as
Mashup Table: Appears to be an empty table with defined columns but no data.Ensure your query returns at least one row by adding something like
if Table.IsEmpty(Source) then ... else ...It would also be helpful if you paste the query into power BI so you can go through each transformation to identify the exact point the flow fails.
Thanks for this. This table has incremental refresh baked in which might result in a an empty table. i am not sure how that needs to be handled for lakehouse ingestion.
This part should be internally baked in to incemental refresh -
if Table.IsEmpty(Source) then ... else ...
smpa01 Hey there! I am PM for Incremental refresh for DFgen2. If you encounter issues, please feel free to open a support ticket to get to the bottom of this.
Looking at your scenario, we think that switching from non-incremental to incremental for the same table is causing the issue. If you check out the docs here: Incremental refresh in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
you see that we mention the following:
- When you write to a lakehouse, the dataflow maintains it owns bookkeeping of the files that are written to the lakehouse. This is in line with the standard lakehouse pattern. However, this means that if other writers like Spark or other processes write to the same table, it may cause issues with the incremental refresh. We recommend that you don't use other writers to write to the same table while using incremental refresh. If you do, you need to ensure that the other writers don't interfere with the incremental refresh process. Processes like table maintenance and vacuuming are not supported either while using incremental refresh.
What probably happens in your situation is that you try to write to an existing table in which the files are not partitioned to the dates we use for incremental refresh. We advise if you need to have historic data loaded sepretly, to make that a seperate table and use a view that combines both tables into one.
Hope this helps! and to better understand we need to have a support ticket such as we can dive deeper into your scenario 🙂
Hi smpa01 ,
Thanks for engaging with Microsoft Fabric Community.
Just checking in to see if your issue is resolved and if you’ve had a chance to raise a support ticket. If the issue has been resolved, it would be helpful if you could share the resolution steps here for the benefit of the wider community.I also followed the repro steps you shared and was able to observe the same behavior on my end. The error still occurred even after creating a new table and using a new Dataflow Gen2.
Hope the support team is able to help resolve the issue.
Thanks miguel, LuitwielerMSFT and wardy912 for addressing this and sharing valuable insights.Please reach out for further assistance.
Thank you.
I’m currently under significant time pressure to deliver a production-ready workflow for my client. Over the past two weeks, I’ve been evaluating Dataflow Gen2 for my use case, but have encountered several critical issues and unexpected behaviors:
- Incremental refresh does not function as expected
- Refresh operations result in Lakehouse tables with the correct schema and row count, but contain no data
- No REST API support for triggering refreshes using a service principal (SPN)
These challenges are difficult to diagnose or address intuitively, making DF Gen2 unreliable for my current complex production scenario.While I plan to raise a formal support ticket once immediate client deliverables are met, I’ve documented the steps to reproduce these issues (outlined above). My current focus is to design a robust and scalable data engineering solution—ideally leveraging Notebooks as the primary mechanism, rather than relying on low-code artifacts like Dataflows or Pipelines. This is particularly important in the context of enterprise-level BI that integrates data from diverse sources, including on-prem systems, Databricks, Dremio, SharePoint, and others.
In contrast, Dataflow Gen1—though limited to producing flat tables and lacking SQL or Spark transformation capabilities—proved far more stable in practice. From a functional standpoint, it presented minimal issues. The unfortunate limitation is that it cannot insert data into a destination.
Hi smpa01 ,
Thanks for the detailed follow-up and for sharing your observations. We completely understand the pressure of delivering a stable, production-ready solution, and we appreciate you taking the time to list out the issues and repro steps despite that.
It’s clear that some of the current limitations in Dataflow Gen2, especially around incremental refresh and service principal support for refresh, are making it difficult to use in your production scenario.
When you're able to, raising a support ticket will allow the engineering team to look into the issue more deeply. In the meantime, using notebooks could be a good alternative, especially since they offer more control and flexibility when working with different data sources like Databricks, SharePoint, and on-prem systems.
Thanks again for your feedback - it helps improve the experience for everyone. Feel free to share any further updates here.
I had the same error, but I had a similar dataflow gen 2 working, so I did very basic troubleshooting and removed all non text columns and started adding them and sure enough it was a number column that was giving me the error. Fixed that column and it all worked.
regards,
Hey rubymaya 🙂
I understand the frustration here. This error (10855) is actually expected behavior. When you first ingest data without incremental refresh, the dataflow writes to a standard Delta table structure. When you then turn on incremental refresh for the same table, the dataflow needs a different file structure, where it partitions data into time-based buckets and manages its own bookkeeping of the files. Those two structures are incompatible, which is what causes this error.
The fix: point your incremental refresh at a new, empty table. Don't enable incremental refresh on a table that was previously loaded without it. If you need the historical data alongside, you can load that into a separate table and create a view that combines both.
This is documented in the Lakehouse caveats section of the incremental refresh docs.
Also, since you're using an on-prem gateway, make sure it's updated to at least version May 2025 (3000.270+), which is required for Lakehouse incremental refresh to work correctly.
On the query folding requirement: I hear you, and I know that's a change from Gen1. It's by design to ensure performance, but I understand it makes certain scenarios harder. We appreciate you sharing that feedback.
