Dataflows and Incremental Refreshes

robarivas · ‎04-06-2020

I'm a Premium user. Having a little trouble making sense of the best practice workflow around using Dataflows and Incremental Refreshes for the purpose of creating a Power BI Report/Dataset. My understanding is that Incremental Refresh only has value if Query Folding is involved. If true then does that mean there is basically no value (from an Incremental Refresh perspective) in using a large entity from a Dataflow for a Dataset (i.e., Power BI Report)? Because won't that dataset take a very long time to refresh (even in the Service) since Dataflows (Azure Data Lake Gen 2) currently are not a foldable source?

Everything about Dataflows, Linked and Computed Entities, and Incremental Refresh sound very cool but ultimately its the Power BI Dataset that matters (to me anyway). If you can't get that report to refresh quickly then all that cool stuff doesn't matter...at least not for Power BI Reports.

Anonymous · ‎04-15-2020

Hi @robarivas ,

Is there anything else need to be provided from my side about this thread?

Best Regards

Rena

robarivas · ‎04-15-2020

Thanks @Anonymous. So it sounds like we are in agreement that dataflows are not a good source to use for a Power BI report if the following conditions exist:

the number of rows in the dataflow is very large
a foldable database exists as an alternative
and your goal is to have your Power BI report refresh quickly and/or incrementally

Agreed?

I assume Microsoft is working on changing this situation (in other words -- trying to get Dataflows to be a foldable source). If so, is there a rough timeline as to when that might occur (2021, 2020, Q3 2020, May 2020, etc.)?

Anonymous · ‎04-07-2020

Hi @robarivas ,

Yes, if the data source don't support query folding, it may retrieve the full dataset from the data source. Then in this case there is no meaningful to configure incremental refresh... Please refer the following documentations for more details:

https://docs.microsoft.com/en-us/power-bi/service-premium-incremental-refresh#configure-incremental-...

https://docs.microsoft.com/en-us/power-bi/service-dataflows-incremental-refresh

https://radacad.com/how-to-use-dataflow-to-make-the-refresh-of-power-bi-solution-faster

Best Regards

Rena