Forum Discussion
Python performance/benchmark - very slow - how to improve?
Are you looking for a way specific to this dataset or more general approach?
If it's the former - how do you plan to structure your data model? Also, how often do you plan to refresh report?
in this article you can find few tips related to compresion, not sure this can impact loading times though
https://powerpivotpro.com/2016/01/restructure-data-table-improved-compression/
I'm looking for general approach on how to speed up load times in ETL area (data load, queries).
I don't have problems with DAX performance (once the data is loaded into columnar data model). It works quite fast, I don't see any difference with Qlik in that area. (Pandas is not a competitor in this area, because it still(?) does not have the columnar processing implemented). http://wesmckinney.com/blog/apache-arrow-pandas-internals/
But I do have datasets which change a lot and I need to reload these into RAM.
My hope was that I could use a combo of Pandas/Parquet and M - but this is very, very slow - even slower than an pure M approach.
I don't have any problems with Qliksense or Pandas performance on the other side, i.e. when I invest in good hardware (or rent performant server in cloud) - i get a considerable boost in reload times, as expected.
So, in summary I'm looking for general approach on how to speed up load times via M (and Python or R combos).
