Forum Discussion

Nerble's avatar
Nerble
Frequent Visitor
3 years ago

Dataflow is caching data between linked entities and not updating.

Situation:

I have a dataflow set up like this in the service:

 

At step 1, I have incremental refresh enabled - pointing at an insert and LAST_UPDATE date/time column.   Enhanced Compute Engine is set to ON.  This is a table with ~2m rows.

 

At step 2, Enhanced Compute Engine is set to "Optimized."  For testing purposes I'm only linking the table from step 1 and not performing any transformations at all on that table.

 

Result:

I am seeing a mismatch in data between linked entities in step 1 and 2:

 

Here is what a particular value looks like in the Extract step, with the filter applied at the very last step (after the incremental refresh step):

 

 

Here is the same table, with a filter applied as the very first step against the linked table:

 

What is going on here?  I've had this issue crop up periodically now and I can resolve it by dropping the cached table in the extract or sometimes by toggling ECU on and off...but nothing seems terribly clear about what's causing the issue

2 Replies

  • Nerble , there is something unstable happening within the Enhanced Computed Engine logic. I am having a very similar issue at it's been months without any solution.

    • Nerble's avatar
      Nerble
      Frequent Visitor

      Yeah I'm still having this issue crop up as well.