Dataflow gen2 refresh Error

I’m currently under significant time pressure to deliver a production-ready workflow for my client. Over the past two weeks, I’ve been evaluating Dataflow Gen2 for my use case, but have encountered several critical issues and unexpected behaviors:

- Incremental refresh does not function as expected

- Refresh operations result in Lakehouse tables with the correct schema and row count, but contain no data

- No REST API support for triggering refreshes using a service principal (SPN)

These challenges are difficult to diagnose or address intuitively, making DF Gen2 unreliable for my current complex production scenario.

While I plan to raise a formal support ticket once immediate client deliverables are met, I’ve documented the steps to reproduce these issues (outlined above). My current focus is to design a robust and scalable data engineering solution—ideally leveraging Notebooks as the primary mechanism, rather than relying on low-code artifacts like Dataflows or Pipelines. This is particularly important in the context of enterprise-level BI that integrates data from diverse sources, including on-prem systems, Databricks, Dremio, SharePoint, and others.

In contrast, Dataflow Gen1—though limited to producing flat tables and lacking SQL or Spark transformation capabilities—proved far more stable in practice. From a functional standpoint, it presented minimal issues. The unfortunate limitation is that it cannot insert data into a destination.