Forum Discussion
Helper IIncremental and Max Memory usage for a P1
Helper IHi pallavi_r ,
Short answer : The full dataset is probably 22 GB.
Long answer :
Power BI Service -> Workspace->Manage group storage doesn't seem very reliable at first sight.
Yesterday when i uploaded the full dataset it said 7,2 GB, and the half-dataset 3,6 GB.
But on the fabric metric app i had 22 GB for the full one and 9 GB for the half-one.
I left the half dataset the whole night on the workspace to test an increment refresh in "normal" condition, and it worked well like usual. However this time in Power BI Service -> Workspace->Manage group storage it does say "9 GB" like in the Fabric Metrics App. I suppose there is some sort of delay in this page depending on which state of refresh is the dataset. (Or maybe it's just my navigator cache which induce me in error).
Do you think the 22 GB is too much for a P1? Even tho i try to refresh only a small part of it ?
Super UserHi JohnnyDax ,
Few points and my learning from this issue in the past:
Point 1-
Resource governing error occurs if refresh is taking lot more memory and cpu usage, so at that time the refresh fails and stops. I have noticed that this time memory cache is not clear. And on top of this error, if again and again we try to re-refresh, then it shows memory increase continually with each refresh. So better to republish and refresh, rather then re-refreshed memory failed dataset refresh. That's why we will see the 9 to 14 to 25 weird and bloated memory size.
Point 2:
XMLA endpoint partitioned based refresh is th optimum way to deal with this. Normal incremental refresh at times does not work for large dataset format. Probably for a large dataset 7 gb+, incremental is not working with normal setup. Trial and tested by me too. The same issue exactly I had got. XMLA refresh saved me 🙂
Point 3:
For P1 capacity (irrespective of full or incremental), dataset size should be below 14 gb. It has cap of 25 GB, which is like when we refresh the size gets almost doubled. 25/2 = 12.5. Till 12.5-13 it works perfectly fine, the moment it reaches 13-14 gb, the dataset even if we are doing incremental refresh it sometime fails giving memory issue.
Thanks,
Pallavi