Forum Discussion

Clampazzo's avatar
Clampazzo
Icon for Resolver I rankResolver I
2 years ago
Solved

Why would a full initial incremental refresh be faster than a import of the same model?

I have a new model that I'm testing out.  The model pulls in about 50 million records and is connected to databricks.

 

I created an incremental refresh of the model and kicked it off in the Power BI Service.  It took about half an hour to finish on it's first run.

 

I then created a simple import version of the model with no incremental refresh settings setup.  This version was running at 2 hours when I finally went in and cancelled it.  

 

So is there any reasoning as to why the incremental (even at a full initial refresh) would run so much faster than the same import of just an import model?

 

Need Help
  • pallavi_r's avatar
    pallavi_r
    2 years ago

    Hi Clampazzo ,

     

    First time during full refresh it takes the actual time for refreshing 50 million records. Next refresh onwards, if its is large dataset and incremental is not setup, it takes double memory. Say, your dataset size is 8 gb, during 2nd refresh it takes up memory of 8x2=16gb dataset. It gives memory issues and doesnt succeed in refreshing the dataset from 2nd refresh onwards. That's why it is recommended to go for incremental setup.

    Thanks,

    Pallavi

2 Replies

  • pallavi_r's avatar
    pallavi_r
    Icon for Super User rankSuper User
    2 years ago

    Hi Clampazzo ,

     

    First time during full refresh it takes the actual time for refreshing 50 million records. Next refresh onwards, if its is large dataset and incremental is not setup, it takes double memory. Say, your dataset size is 8 gb, during 2nd refresh it takes up memory of 8x2=16gb dataset. It gives memory issues and doesnt succeed in refreshing the dataset from 2nd refresh onwards. That's why it is recommended to go for incremental setup.

    Thanks,

    Pallavi

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi, Clampazzo 

    The initial full refresh of an incremental refresh may use more efficient algorithms and techniques to process large data sets, and multiple data partitions may be processed simultaneously for faster data loading. Second, Power BI Service may allocate more computational resources to the incremental refresh, thereby increasing processing speed.

     

    Whereas a simple import version of the model may process data sequentially, which is slower, and may not get the same resource allocation as incremental refresh, so the refresh speed will be slower.

     

    If you are using DirectQuery mode when connecting to databricks to get data, then the first full refresh will not take up twice as much memory, whereas a normal import mode will take up twice as much memory, which will also result in slower efficiency when refreshing.

     

     

     

    I hope my suggestions give you good ideas, if you have any more questions, please clarify in a follow-up reply.
    Best Regards,
    Fen Ling,
    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.