Forum Discussion

Tejinder's avatar
Tejinder
Helper I
2 years ago
Solved

Dataflow vs Dataset incremental refresh times are very different, Help ?

I have scheduled an incremental refresh from an SQL server through PBi dataflows, a full refresh takes about 12 minutes but an incremental refresh takes 50+ minutes, why would that be the case? 

When I try the same native query and schedule the incremental refresh on the dataset instead of dataflow, it's much faster, under 3 minutes for incremental refresh. but I want to use dataflows to get my data as I have multiple reports from the same DB. 

here are the partitions of my dataset:

 



Also, My query is fairly complex with 10+ JOINS. would doing these JOINS  in power query instead, speed up incremental in dataflows.

Thanks 

Need Help
  • lbendlin's avatar
    lbendlin
    2 years ago

    Refresh times for incremental refresh depend heavily on the housekeeping activities (partition consolidation, starting over etc) as well on the current performance of your data source.

     

     I want to use dataflows to get my data as I have multiple reports from the same DB.

    That's not necessarily an argument for dataflows over datasets.  Dataflows are there to shield you ( the developer ) from slow data sources.  If your data source is not slow then use datasets.

     

    Also, My query is fairly complex with 10+ JOINS. would doing these JOINS  in power query instead, speed up incremental in dataflows.

    Not likely.  Spend your energy on making the SQL query faster through correct indexes and statistics.

     

1 Reply

  • lbendlin's avatar
    lbendlin
    Super User
    2 years ago

    Refresh times for incremental refresh depend heavily on the housekeeping activities (partition consolidation, starting over etc) as well on the current performance of your data source.

     

     I want to use dataflows to get my data as I have multiple reports from the same DB.

    That's not necessarily an argument for dataflows over datasets.  Dataflows are there to shield you ( the developer ) from slow data sources.  If your data source is not slow then use datasets.

     

    Also, My query is fairly complex with 10+ JOINS. would doing these JOINS  in power query instead, speed up incremental in dataflows.

    Not likely.  Spend your energy on making the SQL query faster through correct indexes and statistics.