Forum Discussion
Incremental Refresh on a dataset with Dataflows as a data source
Always if you want an enterprise DW then you would need to build an enterprise data warehouse that would be separate to dataflows in my opinion.
Whilst dataflows are great they do not have all the features that you would expect to have in a data warehouse table.
Hi GilbertQ
Thank you very much for your answer, you are right that Dataflows do not offer all features of a "real DWH" - therefore the setup described above is using an AWS Redshift DWH => Incremental Refresh on Dataflows => Incremental Refresh on depending Datasets because we would like to use Power BI (Service) to query/anlayse the data from the DWH.
We were planning to use Dataflows as an abstraction layer and ingest mechanism as proposed by Microsoft so that this ingested data can be used by the depending (Shared-) Datasets and thereby to implement a modern Self-Service BI solution.
Because of the size of the data and the limited time available during nightly data refesh we require an incremental data refresh on all stages of the dataload (Dataflows & Datasets) to provide our users with updated date every day at 07:00 latest.
So the the incremental data refreshs have to work in tandem - currently it appears that this (incremental refreshs for both Dataflows and depending Datasets) is not working (yet) and we still have no official confirmation from Microsoft if this will be working/implemented in the future. If this is not working/or not designed this way we don't see that Power BI can be used in larger Enterprise environments...
Thanks again for your support and best regards
- Hi there
Currently you would have to stager them.
And yes currently unless you are using Power BI Premium you can use Enhanced data flow which should allow for incremental refreshing.