Forum Discussion
Incremental and Max Memory usage for a P1
Hi , JohnnyDax
That's a strange result from your test, the full refresh and the first incremental refresh both contain the largest amount of data and they both work. Perhaps you could adjust its refresh frequency to control the amount of data in one incremental refresh to try.
Based on the above information, if you are a Power BI Pro licensee, you can create a support ticket for free and a dedicated Microsoft engineer will come to solve the problem for you.
It would be great if you continue to share in this issue to help others with similar problems after you know the root cause or solution.
The link of Power BI Support: Support | Microsoft Power BI
For how to create a support ticket, please refer to How to create a support ticket in Power BI - Microsoft Power BI Community
Best regards.
Community Support Team_Scott Chang
Hi Anonymous ,
I've experimented with several scenarios, and all seem to lead to the same outcome: the heaviest version fails during incremental refresh.
We're using P1 Capacity, which I believe has a maximum of 25 GB of RAM. According to the Fabric Metrics App, our full dataset is nearing this limit (22GB), which is puzzling since the Capacity Admin Center shows only 7.2 GB. It's possible the system accounts for the maximum dataset size, hence the larger figure during refreshes. I also suspect that the entire dataset might be cached during the incremental refresh, which is the only explanation I can think of for these issues. With half the dataset, it works fine as it stays well below the 25 GB limit.
This behavior seems unusual since I would expect the service to load only the incremental part, not the entire dataset, into RAM. From what I've read in several blog posts, large datasets might not be well-suited for the built-in incremental refresh feature of the Power BI Service. It seems more feasible to use the XMLA endpoint. I tried to manually refresh only the necessary partitions, which has worked for me. However, I'm unsure how to automate this process at specific times.
Could you provide any insights or suggestions on how to better manage or automate this process? We're using Tabular Editor 2 or SSMS.
Thank you for your help.
Hi JohnnyDax
Can you please tell me what is your dataset size after full refresh when large dataset format is enabled. Please check the dataset size in Power BI Service -> Workspace->Manage group storage. Please check the individual dataset size.
Thanks,
Pallavi
Hi pallavi_r ,
Short answer : The full dataset is probably 22 GB.
Long answer :
Power BI Service -> Workspace->Manage group storage doesn't seem very reliable at first sight.
Yesterday when i uploaded the full dataset it said 7,2 GB, and the half-dataset 3,6 GB.
But on the fabric metric app i had 22 GB for the full one and 9 GB for the half-one.
I left the half dataset the whole night on the workspace to test an increment refresh in "normal" condition, and it worked well like usual. However this time in Power BI Service -> Workspace->Manage group storage it does say "9 GB" like in the Fabric Metrics App. I suppose there is some sort of delay in this page depending on which state of refresh is the dataset. (Or maybe it's just my navigator cache which induce me in error).
Do you think the 22 GB is too much for a P1? Even tho i try to refresh only a small part of it ?
