Forum Discussion
Advocate IData Refresh takes a very long time
When you do joins/appends in Power Query it has to load the entire table into virtual memory. This causes a lot of paging and is very slow.
Does your dataset have to be refreshed on an approximatly hourly basis or is a daily cadence ok?
As this will greatly affect what options are open to you to increase performance.
When you do joins/appends in Power Query it has to load the entire table into virtual memory. This causes a lot of paging and is very slow.
Does your dataset have to be refreshed on an approximatly hourly basis or is a daily cadence ok?
As this will greatly affect what options are open to you to increase performance.
Advocate IHi Anonymous ,
Thank you for this clarification, eventually I kind of figured out this is the case as well.
I changed my data model in order to avoid some of the appends and indeed performance increased substantially.
Thanks again!
GuySevHi
I ran in a similar problem, my power queries running up to 4 hours. I'm using Access to do my joins, the same joins take me 3 seconds now. I found Access to be an excellent tool for extracting from multiple sources, doing your data modeling and specific views, then pushing it out to power Bi. Access also enforces accuracy in your data and it is not difficult to learn. It follows the same data modeling rules as Data modeling in Power BI. Just more strict as it demands parents to child ect aka data referral integrity.
It also acts as a centralization point for your data to maintain and build dimensions etc.
Regards
Wynand
