PBI Report loading CSV files behaves strange

[tomas12344]

Hi All
I have a very strange behavioral for one of the reports, which shows incorect data, in the report, but after next refresh everything is OK (no changes were made neither in report not in souce)

This report loads several CSV files, up to 4 biger files, whith whole year content, and upt to 12 files, smaller monthly files (there are available only files for past month of the year so the number changes from 1-12 depending on the current calendar month). I dont thnig, that for a CSV file the size is too big (50MB +-, or up to 20MB+-). Files are loaded with gateway and are stored on local server (no onedrive,....). Report is in Service. 

These files are loaded daily without any problems.  But, after the begining of the mont, when new monthly file arives, there is some problem with the loading or processing. But the loading bahaves very strange. There is no error in the loading. After the new file is added, looks the report loads everything correctly. After few days, we can see that in the report some values are doubled for previous period (from actual month). Non of the files was changed or touched. When we identify, that the data is not correct, we just refrsh the report again, and the report is correct (nothing is changed in source, code, report,... just refreshed). Files are loaded as whole folder, not separately one-by-one.
I cannot simulate this behavioral on Desktop, as when refreshing in Desktop everything is ok.

Any idea what could be wrong? Could be there some cache?

[Anonymous]

Hi @tomas12344 ,

According to your description, your data source is a csv file, then your report is imported in connection mode.

Model data is only as current as the latest refresh, and so Import models need to be refreshed, usually on a scheduled basis.

For more details, please refer: Configure scheduled refresh - Power BI | Microsoft Learn

Semantic model modes in the Power BI service - Power BI | Microsoft Learn

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly. 

[tomas12344]

can you explain, how this could help to avoid this behavioral??? 
I'm not an expert, but I guess with no refresh no data is loaded. Or is there any magic???

[Anonymous]

Hi @tomas12344 ,

You can store the csv files on OneDrive or SharePoint if you don't want to manually refresh or set up a scheduled refresh. Then use PBI Desktop to connect to that file to create reports and publish them to PBI Services. If you edit the .csv file in OneDrive or SharePoint, the changes usually show up in Power BI within about an hour after you save it. any visualizations in Power BI based on the semantic model are automatically updated.

For more details, please refer: Use OneDrive for work or school links in Power BI Desktop - Power BI | Microsoft Learn

Refresh a semantic model created from a .csv file on OneDrive - Power BI | Microsoft Learn

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly.

[tomas12344]

still dont see any solution for the problem

[Anonymous]

Hi @tomas12344 ,

In addition to setting up scheduled refreshes, you can also store the csv files in a sharepoint folder, using the sharepoint folder connector. Have you tried this? Where are you stuck?

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly.

[Poojara_D12]

Hi @tomas12344 

• Disable Caching: In Power BI Service, turn off "Allow data preview to download during refresh" in Dataset Settings.

• Use Folder Partitioning: Organize files into subfolders (e.g., by month/year) to avoid duplication during processing.

• Filter Duplicates: Ensure the query excludes duplicate files using logic like Table.Distinct(Folder.Files("FolderPath")).

• Enable Incremental Refresh: Configure incremental refresh to process only new or updated data.

• Update Gateway: Ensure the gateway is updated and correctly handling file updates.

Did I answer your question? Mark my post as a solution, this will help others!

If my response(s) assisted you in any way, don't forget to drop me a "Kudos" 🙂

Kind Regards,
Poojara
Data Analyst | MSBI Developer | Power BI Consultant
Please Subscribe my YouTube for Beginners/Advance Concepts: https://youtube.com/@biconcepts?si=04iw9SYI2HN80HKS 

[tomas12344]

Thank you for answer, but
- filenames have total different name (core of the name) so that cannot be duplicated

- query caching is disabled on this report as it is set as default
- Filter DuplicatesHow could be this posible, there are no duplicate during the month, only on some day, but when run second time, everything is working correctly. So there could be not duplicates, when nothing changes (no data change, source, code, .... nothing)

- incremental refresh is not posible due the nature of data