Forum Discussion
Dataflow gen2 refresh Error
I’m currently under significant time pressure to deliver a production-ready workflow for my client. Over the past two weeks, I’ve been evaluating Dataflow Gen2 for my use case, but have encountered several critical issues and unexpected behaviors:
- Incremental refresh does not function as expected
- Refresh operations result in Lakehouse tables with the correct schema and row count, but contain no data
- No REST API support for triggering refreshes using a service principal (SPN)
These challenges are difficult to diagnose or address intuitively, making DF Gen2 unreliable for my current complex production scenario.
While I plan to raise a formal support ticket once immediate client deliverables are met, I’ve documented the steps to reproduce these issues (outlined above). My current focus is to design a robust and scalable data engineering solution—ideally leveraging Notebooks as the primary mechanism, rather than relying on low-code artifacts like Dataflows or Pipelines. This is particularly important in the context of enterprise-level BI that integrates data from diverse sources, including on-prem systems, Databricks, Dremio, SharePoint, and others.
In contrast, Dataflow Gen1—though limited to producing flat tables and lacking SQL or Spark transformation capabilities—proved far more stable in practice. From a functional standpoint, it presented minimal issues. The unfortunate limitation is that it cannot insert data into a destination.
Hi smpa01 ,
Thanks for the detailed follow-up and for sharing your observations. We completely understand the pressure of delivering a stable, production-ready solution, and we appreciate you taking the time to list out the issues and repro steps despite that.
It’s clear that some of the current limitations in Dataflow Gen2, especially around incremental refresh and service principal support for refresh, are making it difficult to use in your production scenario.
When you're able to, raising a support ticket will allow the engineering team to look into the issue more deeply. In the meantime, using notebooks could be a good alternative, especially since they offer more control and flexibility when working with different data sources like Databricks, SharePoint, and on-prem systems.
Thanks again for your feedback - it helps improve the experience for everyone. Feel free to share any further updates here.