Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified in Microsoft Fabric—for free! For a limited time, the Microsoft Fabric Community team will be offering free DP-600 exam vouchers. Prepare now

Reply
Anonymous231
New Member

Solutions for reducing refresh time on PowerBI Desktop

I am working on a dashboard which has a bunch of .csv files placed in the data path folder and the dataset is quite huge. The date range of the data is from 2018 upto the latest month. We receive monthly updates and when we add some new files in the source path and refresh, it takes a lot of time(almost a day or two). I was wondering if there is a way to remove 2018-2019 files by adding some automation steps in PowerQuery? Even if this cannot be achieved, is there any workaround for reducing the refresh time?

 

Also, is it possible to append the new data files without having to refresh the entire tables?

 

 

1 ACCEPTED SOLUTION
v-yiruan-msft
Community Support
Community Support

Hi  @Anonymous231 ,

You can refer the following links to reduce the refresh time:

1. Find the reason for the long refresh time

Why is my Power BI refresh so SLOW?!? 3 Bottlenecks for refresh performance

My dataset is taking too long to refresh!

There are 3 places where the refresh might be slow:
  1. Data source
  2. Power Query engine
  3. Analysis Services engine

2. Connect the file with the latest data only just as suggested by @amitchandak 

Power BI: How to Connect Most Recent File from a Folder

Power BI - Connect to Most Recent File in a Folder

yingyinr_0-1647421145342.png

yingyinr_1-1647421157617.png

3. Use incremental refresh

How you can incrementally refresh any Power BI data source (This example is a CSV File)

Incremental data load from CSV


You can use the same M code and create a Dataflow.  Dataflows DO enable incremental refresh on sources like .csv flat files. 

When the dataflow refreshes, it will only pick up the newest data, perform the transformations, and load that newly shaped data into the Azure Data Lake.

If you do all of your transformations in the dataflow, the Power BI just has to read from the Data Lake to pull data into the model.  That read is usually much faster than reading from .csv files in SharePoint and performing transformations.

I'm pretty sure you can also enable incremental refresh for pulling data from the dataflow.


Best Regards

Community Support Team _ Rena
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

View solution in original post

2 REPLIES 2
v-yiruan-msft
Community Support
Community Support

Hi  @Anonymous231 ,

You can refer the following links to reduce the refresh time:

1. Find the reason for the long refresh time

Why is my Power BI refresh so SLOW?!? 3 Bottlenecks for refresh performance

My dataset is taking too long to refresh!

There are 3 places where the refresh might be slow:
  1. Data source
  2. Power Query engine
  3. Analysis Services engine

2. Connect the file with the latest data only just as suggested by @amitchandak 

Power BI: How to Connect Most Recent File from a Folder

Power BI - Connect to Most Recent File in a Folder

yingyinr_0-1647421145342.png

yingyinr_1-1647421157617.png

3. Use incremental refresh

How you can incrementally refresh any Power BI data source (This example is a CSV File)

Incremental data load from CSV


You can use the same M code and create a Dataflow.  Dataflows DO enable incremental refresh on sources like .csv flat files. 

When the dataflow refreshes, it will only pick up the newest data, perform the transformations, and load that newly shaped data into the Azure Data Lake.

If you do all of your transformations in the dataflow, the Power BI just has to read from the Data Lake to pull data into the model.  That read is usually much faster than reading from .csv files in SharePoint and performing transformations.

I'm pretty sure you can also enable incremental refresh for pulling data from the dataflow.


Best Regards

Community Support Team _ Rena
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
amitchandak
Super User
Super User

@Anonymous231 , Can getting lastest file from folder can help ?

https://wkrzywiec.medium.com/getting-data-from-the-latest-file-in-a-folder-using-power-query-51dfa4b...

Join us as experts from around the world come together to shape the future of data and AI!
At the Microsoft Analytics Community Conference, global leaders and influential voices are stepping up to share their knowledge and help you master the latest in Microsoft Fabric, Copilot, and Purview.
️ November 12th-14th, 2024
 Online Event
Register Here

Helpful resources

Announcements
OCT PBI Update Carousel

Power BI Monthly Update - October 2024

Check out the October 2024 Power BI update to learn about new features.

September Hackathon Carousel

Microsoft Fabric & AI Learning Hackathon

Learn from experts, get hands-on experience, and win awesome prizes.

October NL Carousel

Fabric Community Update - October 2024

Find out what's new and trending in the Fabric Community.