Forum Discussion
Incremental Refresh on Compute Entities
This has been a while since I posted the original question. We have implemented the incremental refresh between dataflows (for example, staging and processing dataflows) and it's been working excellent.
The confusion is between linked and compute entities, as mentioned above. The linked entities cannot be changed from downstream dataflows and so the question does not apply to them. But if you load your original data into a compute entity, then it is possible to implement IR between them.
In our case, it reduced the downstream processing of a large dataflow table (~500 million rows) from 4 hours to about 20 minutes.
This has been a while since I posted the original question. We have implemented the incremental refresh between dataflows (for example, staging and processing dataflows) and it's been working excellent.
The confusion is between linked and compute entities, as mentioned above. The linked entities cannot be changed from downstream dataflows and so the question does not apply to them. But if you load your original data into a compute entity, then it is possible to implement IR between them.
In our case, it reduced the downstream processing of a large dataflow table (~500 million rows) from 4 hours to about 20 minutes.
MightyMicrobe, one more question if I may ask: when you enabled IR on your computed entity, did you have your Enhanced Configuration Engine for this dataflow set to Optimized or On?
I am asking this because it *seems* to work ONLY if ECE is not switched off. In my case, when ECE is switched off, I have error messages in the refresh log about invalid PBI credentials.
Thanks!
Hey fred31330, we're dealing with pretty large tables (100-600 million rows) in our dataflows, and most of them are cascaded, so by default, we always set the Enhanced Engine to ON. I'm not sure if this is the best practice or, in some cases, enhanced compute should not be turned on, but this seems to work well.
We're on a P1 Premium if that helps.
Thanks for your feedback! Exactly what I needed.