Hi All,
In my present project, we are using dataflows to perform data transformations centrally and reusing the dataflow entities in different reports. The models are created mainly from the dataflows and the reports are live connected to the datasets (models).
The data size we are dealing with is large and we intend to implement incremental refresh where for the refresh partitions, we would only refresh last 2 months data.
For the dataflows, it’s well documented and we can set up incremental refresh as required and is working fine as per my test. However, I could not find any official documentation regarding the incremental refresh from datasets to dataflows. This is important for us because otherwise we would end up doing full truncate & load every time during our model refreshes (while refreshing datasets from dataflows).
I did some testing on this, and to my surprise it seems working for datasets as well! I have even checked from SSMS connecting the xmla endpoint of the dataset (we are using premium capacity BTW) and checked the individual partitions – and the time stamps also suggest that after initial full load, for all subsequent refreshes, only the months falling under refresh partition as per the incremental refresh policy – got refreshed!
Now all I found on the net suggests the other way round (as query folding does not happen on dataflow – incremental refresh would not work). Can please anyone help me to understand if this is a latest update from Microsoft and if so, is there any official documentation link which we can refer for best practices etc.? (this will give us more confidence in terms of the project design)
Note: I have tried separately with the dataflow ‘enhanced compute engine’ set to ‘optimized’ & ‘On’ – and it seems working for both the cases!
** Sorry for posting this under 'Desktop', only after posting I realized it should have been posted under 'Service' but haven't found any option to switch the product. Hope it's fine?
@GilbertQ @ibarrau @v-janeyg-msft @Burningsuit @lbendlin
