Forum Discussion
Incremental Refresh on Compute Entities
This has been a while since I posted the original question. We have implemented the incremental refresh between dataflows (for example, staging and processing dataflows) and it's been working excellent.
The confusion is between linked and compute entities, as mentioned above. The linked entities cannot be changed from downstream dataflows and so the question does not apply to them. But if you load your original data into a compute entity, then it is possible to implement IR between them.
In our case, it reduced the downstream processing of a large dataflow table (~500 million rows) from 4 hours to about 20 minutes.
This has been a while since I posted the original question. We have implemented the incremental refresh between dataflows (for example, staging and processing dataflows) and it's been working excellent.
The confusion is between linked and compute entities, as mentioned above. The linked entities cannot be changed from downstream dataflows and so the question does not apply to them. But if you load your original data into a compute entity, then it is possible to implement IR between them.
In our case, it reduced the downstream processing of a large dataflow table (~500 million rows) from 4 hours to about 20 minutes.
I'm still confused about this, as I asked what I think is a similar question here: https://community.fabric.microsoft.com/t5/Service/Incremental-Refresh-with-Linked-Referenced-Tables/m-p/3488305#M209333
Are you saying you implemented two incremental refreshes? One on the source staging table and another one on a linked referenced/computed entity?
I put an incremental refresh on the staging table when first bringing it in from an on-prem SQL server and it takes 30 seconds. But my linked referenced/computed entity to that staging table takes 3 minutes (I haven't done any transformations to it yet).
So I'm trying to figure out if I need to create a second incremental refresh on the downstream linked tables.