Forum Discussion

sarahshh's avatar
sarahshh
Frequent Visitor
1 year ago
Solved

Incremental refresh for dataflow takes as much time as full refresh

Hello,

I've used incremental refresh for my dataflow, but it takes almost as long as the full refresh.

He're below is the configuration and the timings:

 

for the full refresh (~15min):

 

for incremental refresh (~15min):

 

Do you know why ?

Thanks !

Dataflow
  • Deku's avatar
    Deku
    1 year ago

    Can you define a view in the you SQL source that defines the query, including the CTEs, that you call in your data flow?

11 Replies

  • Rigensis's avatar
    Rigensis
    Icon for Resolver I rankResolver I
    1 year ago

    From your screenshots it looks like the Incremental refresh, at least in the way it is set up now, is not actually loading a portion of the data during incremental load.
    As we see in your screenshots, in both full and incremental refreshes the 'Rows Processed' column shows pretty much the same number of rows.
    This could be, as Deku mentions, due to query folding not working. Or it could be that the date field to which you have set the incremental load is affecting a lot more rows that you initially expect.