Forum Discussion

MightyMicrobe's avatar
MightyMicrobe
Helper II
4 years ago
Solved

Incremental Refresh on Compute Entities

Hi all,  we are building a complex ETL model using dataflows and I'd like to clarify how to use incremental refreshes on a P1 instance.    Our Requirements: - A large dataset from a SQL server (~...
Dataflow
Need Help
  • MightyMicrobe's avatar
    MightyMicrobe
    4 years ago

    This has been a while since I posted the original question. We have implemented the incremental refresh between dataflows (for example, staging and processing dataflows) and it's been working excellent. 

     

    The confusion is between linked and compute entities, as mentioned above. The linked entities cannot be changed from downstream dataflows and so the question does not apply to them. But if you load your original data into a compute entity, then it is possible to implement IR between them. 

     

    In our case, it reduced the downstream processing of a large dataflow table (~500 million rows) from 4 hours to about 20 minutes. 

MightyMicrobe's avatar
MightyMicrobe
Helper II
4 years ago

Thanks for your reply. Does it mean that incremental refresh logic implemented at the source is propagated throughout all linked entities? Can you post a link to where I can read more?

fred31330's avatar
fred31330
Advocate I
4 years ago

Hi

I am wondering how you finally solved this challenge as I am in the same situation. Could you kindly share your feedback?

Thanks!