Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
6 years ago
Solved

Incremental Refresh didn't save time

I have a very large dataset of over 25 million rows and I set up a 90-day incremental refresh for it. But the refresh took nearly as long as the usual refresh. Does anybody know how to reduce the incremental refresh time? 

Need Help
  • GilbertQ's avatar
    GilbertQ
    6 years ago
    Hi there

    It would then appear that both of those data sources do not support query folding.

    Which means that it has to load all the data first, and then determine which data to load incrementally. In doing it in this manner it will not be any quicker using incremental refreshing.

13 Replies

  • GilbertQ's avatar
    GilbertQ
    Super User
    6 years ago
    Hi there

    When you say you set up the incremental refresh what is your data source?

    If it is not a data source where query folding can happen, or where it cannot read individual data it will have to read all the data first, before it can incrementally load the new data.
    • Anonymous's avatar
      Anonymous
      Not applicable
      6 years ago

      Hi,

       

      I tried to use Dataflow and Snowflake. But they were both slow. I saw the warning of query folding when I set it up. Is there any other way I can speed up the refresh without changing the data source? 

       

      Thanks,