Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified in Microsoft Fabric—for free! For a limited time, the Microsoft Fabric Community team will be offering free DP-600 exam vouchers. Prepare now

Reply
gdimmers
New Member

Excel Database

I have a folder with csv files that I use as a datasource for a powerbi file. The dataset represents about 12M rows of data. A new csv file is added daily and once added the files are static. It works ok, but when i want to make changes or refresh the file it takes 30-60mins presumably because it reloads all the data. I am trying to find if there is a more effcient way to store these data files and connect with BI that makes the file easier to work with.

 

Thanks.

1 ACCEPTED SOLUTION
Ritaf1983
Super User
Super User

Hi @gdimmers 
An incremental refresh is really challenging to deal with CSV because it not availible for query folding.
Please refer to this article, I believe it can be of assistance.

https://pbi-guy.com/2022/07/14/how-to-use-incremental-refresh-on-any-data-source/

Another alternative to avoid dealing with refresh problems in the desktop is to separate the report from the data set.
To publish data sets to the service and then prepare reports from the desktop using the published model as a data source:

Ritaf1983_0-1688434108808.png

If this post helps, then please consider Accepting it as the solution to help the other members find it more quickly

Regards,
Rita Fainshtein | Microsoft MVP
https://www.linkedin.com/in/rita-fainshtein/
Blog : https://www.madeiradata.com/profile/ritaf/profile

View solution in original post

3 REPLIES 3
gdimmers
New Member

Thanks for the response. I have been going down the route of separating the database and the reports like you suggested. It works ok just having to get used to some of the limitations with the live connection...

Glad to help 🙂
If this post helps, then please consider Accepting it as the solution to help the other members find it more quickly

Regards,
Rita Fainshtein | Microsoft MVP
https://www.linkedin.com/in/rita-fainshtein/
Blog : https://www.madeiradata.com/profile/ritaf/profile
Ritaf1983
Super User
Super User

Hi @gdimmers 
An incremental refresh is really challenging to deal with CSV because it not availible for query folding.
Please refer to this article, I believe it can be of assistance.

https://pbi-guy.com/2022/07/14/how-to-use-incremental-refresh-on-any-data-source/

Another alternative to avoid dealing with refresh problems in the desktop is to separate the report from the data set.
To publish data sets to the service and then prepare reports from the desktop using the published model as a data source:

Ritaf1983_0-1688434108808.png

If this post helps, then please consider Accepting it as the solution to help the other members find it more quickly

Regards,
Rita Fainshtein | Microsoft MVP
https://www.linkedin.com/in/rita-fainshtein/
Blog : https://www.madeiradata.com/profile/ritaf/profile

Helpful resources

Announcements
OCT PBI Update Carousel

Power BI Monthly Update - October 2024

Check out the October 2024 Power BI update to learn about new features.

September Hackathon Carousel

Microsoft Fabric & AI Learning Hackathon

Learn from experts, get hands-on experience, and win awesome prizes.

October NL Carousel

Fabric Community Update - October 2024

Find out what's new and trending in the Fabric Community.