Forum Discussion
Optimizing data model update time
Hello,
What kind of recomendation coul be to reduce model update time?
I've got folders with > 30Gb of CSV data, and each month I add +2Gb. Last try of update take about 2 hour, but it take more time, there update process was terminated.
At the moment. which I found recomendation:
1) I turned off all auto detect date-time column;
2) Delete column which I not use.
Or is it way to increase time to update model?
Cool. The dataflow serves as intermediate data source. Multiple datasets can now consume from the dataflow instead of recreating the import from files. And yes you need to refresh the dataset, now that your dataflow is updated. But see, even reading from a hyperfast single table Azure source, it takes 20min to refresh, that´s the bare minimum under ideal conditions. If I calculated correctly that equals ~200Mbps! on a shared, free ressource! Therefore, if you find a 60min solution reading from text files that need to be processed row per row, I think that should be acceptable. If that solves your problem, please accept as solution to close the case. Alexander
10 Replies
Hi, does it take 2 hours to refresh the model from PBI desktop or in the service (workspace)? And where do you keep your data? Local harddrive / server or cloud storage? Is it few big files or many hundreds of smaller files?
If your computer / local bandwidth is the limiting factor, then 2 hours for 30+ GB is not too unrealistic.
As a solution, you could
* put your data on cloud storage (e.g. sharepoint) and
* create a dataflow ingesting your data every day.
The dataflow runs on a shared capacity in the cloud, and you can run it during the night so you don´t have to wait for it. With PBI Desktop, you then just read from the dataflow which is really fast.Alexander
Anonymous
All data stored on cloud OneDrive. There are files to each month, at the moment - it 17 files. Refresh time takes through service. version.
Could you give more information about "create a dataflow ingesting your data every day."? I upload new file once per week, and now I trying to refresh all model, in case of first upload.
Hi Denis_Slav ,
Check the reference here.
https://powerbi.microsoft.com/en-us/blog/introducing-power-bi-data-prep-wtih-dataflows/
Best Regards,
KellyDid I answer your question? Mark my post as a solution!
