Forum Discussion
Optimizing data model update time
Cool. The dataflow serves as intermediate data source. Multiple datasets can now consume from the dataflow instead of recreating the import from files. And yes you need to refresh the dataset, now that your dataflow is updated. But see, even reading from a hyperfast single table Azure source, it takes 20min to refresh, that´s the bare minimum under ideal conditions. If I calculated correctly that equals ~200Mbps! on a shared, free ressource! Therefore, if you find a 60min solution reading from text files that need to be processed row per row, I think that should be acceptable. If that solves your problem, please accept as solution to close the case. Alexander
Isn´t it strange that 1 file takes 9min but 5 files take 104 instead of 45min? Something´s not right.
It would be best to reduce number of rows and columns in the CSV before ingesting in dataflow /dataset. This will reduce the amount of time for transfer & transformation upfront. How many rows / columns do your files have today?
As a trial, you could manually reduce the files size by removing rows / columns from the CSV. Let me know the new timing.
Alexander
Anonymous Now, it's ok. Because erlyest file has less rows and size. It's avarage time. I suppose it may be different in 2 times. Files from 0.9Gb to 2.4gb.
Now I'm preparing files for 2 test:
1) COmbine by 3 files to one;
2) Reduce rows in files;
It take time 🙂
Anonymous
I've got very intresting results of testing
# Type Total time Files Total rows Avg time per 1 file Avg time per 1M row 1 ByMonth 1:44:00 12 75 228 279 0:08:40 0:01:23 2 ByMonth + filter on rows 0:22:01 6 31 751 044 0:03:40 0:00:42 3 ByQuater + filter on rows 0:23:32 2 31 751 044 0:11:46 0:00:44 4 By7Month + filter on rows 0:24:31 1 38 268 574 0:24:31 0:00:38 5 ByQuater + delete rows in dataset 0:20:31 2 19 844 404 0:10:15 0:01:02 6 ByMonth + filter on rows 0:58:03 17 101 852 315 0:03:25 0:00:34 And after refreshing dataflow I manualy refreshed data, and it takes 00:20:17.
Very interesting statistic. There is no any significant difference in speed between FILTRED and DELETED file.
But I'm not clear understand. If dataflow was successfully refreshed, why model and report was not updated? Is I need after refresh model after refreshed dataflow?
Cool. The dataflow serves as intermediate data source. Multiple datasets can now consume from the dataflow instead of recreating the import from files. And yes you need to refresh the dataset, now that your dataflow is updated. But see, even reading from a hyperfast single table Azure source, it takes 20min to refresh, that´s the bare minimum under ideal conditions. If I calculated correctly that equals ~200Mbps! on a shared, free ressource! Therefore, if you find a 60min solution reading from text files that need to be processed row per row, I think that should be acceptable. If that solves your problem, please accept as solution to close the case. Alexander