Incremental and Max Memory usage for a P1

Hi Anonymous ,

I've experimented with several scenarios, and all seem to lead to the same outcome: the heaviest version fails during incremental refresh. 

We're using P1 Capacity, which I believe has a maximum of 25 GB of RAM. According to the Fabric Metrics App, our full dataset is nearing this limit (22GB), which is puzzling since the Capacity Admin Center shows only 7.2 GB. It's possible the system accounts for the maximum dataset size, hence the larger figure during refreshes. I also suspect that the entire dataset might be cached during the incremental refresh, which is the only explanation I can think of for these issues. With half the dataset, it works fine as it stays well below the 25 GB limit.

This behavior seems unusual since I would expect the service to load only the incremental part, not the entire dataset, into RAM. From what I've read in several blog posts, large datasets might not be well-suited for the built-in incremental refresh feature of the Power BI Service. It seems more feasible to use the XMLA endpoint. I tried to manually refresh only the necessary partitions, which has worked for me. However, I'm unsure how to automate this process at specific times.

Could you provide any insights or suggestions on how to better manage or automate this process? We're using Tabular Editor 2 or SSMS. 

Thank you for your help.