Dataflow gen2 refresh Error

Hey rubymaya  🙂

I understand the frustration here. This error (10855) is actually expected behavior. When you first ingest data without incremental refresh, the dataflow writes to a standard Delta table structure. When you then turn on incremental refresh for the same table, the dataflow needs a different file structure, where it partitions data into time-based buckets and manages its own bookkeeping of the files. Those two structures are incompatible, which is what causes this error.

The fix: point your incremental refresh at a new, empty table. Don't enable incremental refresh on a table that was previously loaded without it. If you need the historical data alongside, you can load that into a separate table and create a view that combines both.

This is documented in the Lakehouse caveats section of the incremental refresh docs.

Also, since you're using an on-prem gateway, make sure it's updated to at least version May 2025 (3000.270+), which is required for Lakehouse incremental refresh to work correctly.

On the query folding requirement: I hear you, and I know that's a change from Gen1. It's by design to ensure performance, but I understand it makes certain scenarios harder. We appreciate you sharing that feedback.