I have a dataflow set up like this in the service:
At step 1, I have incremental refresh enabled - pointing at an insert and LAST_UPDATE date/time column. Enhanced Compute Engine is set to ON. This is a table with ~2m rows.
At step 2, Enhanced Compute Engine is set to "Optimized." For testing purposes I'm only linking the table from step 1 and not performing any transformations at all on that table.
I am seeing a mismatch in data between linked entities in step 1 and 2:
Here is what a particular value looks like in the Extract step, with the filter applied at the very last step (after the incremental refresh step):
Here is the same table, with a filter applied as the very first step against the linked table:
What is going on here? I've had this issue crop up periodically now and I can resolve it by dropping the cached table in the extract or sometimes by toggling ECU on and off...but nothing seems terribly clear about what's causing the issue
Just to determine if your situation is similar to other users we've encountered recently, have you updated your gateway to the latest version. If not, does the issue still occur after you update to the latest version?