Dataflow is caching data between linked entities and not updating.

[Nerble]

Situation:

I have a dataflow set up like this in the service:

At step 1, I have incremental refresh enabled - pointing at an insert and LAST_UPDATE date/time column.   Enhanced Compute Engine is set to ON.  This is a table with ~2m rows.

At step 2, Enhanced Compute Engine is set to "Optimized."  For testing purposes I'm only linking the table from step 1 and not performing any transformations at all on that table.

Result:

I am seeing a mismatch in data between linked entities in step 1 and 2:

Here is what a particular value looks like in the Extract step, with the filter applied at the very last step (after the incremental refresh step):

Here is the same table, with a filter applied as the very first step against the linked table:

What is going on here?  I've had this issue crop up periodically now and I can resolve it by dropping the cached table in the extract or sometimes by toggling ECU on and off...but nothing seems terribly clear about what's causing the issue

[rdnguyen]

@Nerble , there is something unstable happening within the Enhanced Computed Engine logic. I am having a very similar issue at it's been months without any solution.

[Nerble]

Yeah I'm still having this issue crop up as well.