Exceed Memory on Refresh

Question

Hi I have a P1 Power Bi Premium capacity and the data set I am importing has a fact table coming from Snowflake that is 5GB with about 230m rows and 16 columns (3 years of data). The dim tables of which there are 6 in the semantic model are obviously much smaller with the largest being 120mb with about 9m rows and only 4 columns. I built the model in Power BI Desktop with parameters set to only 1 month of data just for development and testing purposes. Once published to the service I have adjusted the parameters to 3 years and I get the below error. The largest period of data I have had the parameters set to that has successfully refreshed is only 13 months worth. I have developed similar sized models for another organisation also on a P1 and haven't had these issues.

djurecic · Answer

Hi&nbsp;ACSpotlight&nbsp;,
&nbsp;Keep in mind that with import mode, the model size will double during the refresh process. This is because a copy of the exisiting data is stored for use while the refresh is happening while a copy of the data is refeshed.

pankajnamekar25 · Answer

Hello&nbsp;ACSpotlight&nbsp;You're hitting a memory error on refresh in Power BI Service (P1 capacity) after increasing your dataset from 1 month to 3 years. The model exceeds the 25GB per dataset memory limit during processing.SuggestionsOptimize Power Query steps (remove unused columns early)Push aggregations to SnowflakeImplement Incremental Refresh to limit data loadedMonitor memory usage via Premium Metrics appSplit large model into smaller parts or use dataflows&nbsp;&nbsp;&nbsp;Thanks&nbsp;Pankaj Namekar | LinkedInIf this solution helps, please accept it and give a kudos (Like), it would be greatly appreciated.

acspotlight · Answer

Hi Pankaj,&nbsp; Thankyou for your suggestions.&nbsp; I removed a couple of columns from the fact table that were only nice to haves.&nbsp; This reduced the table from 5gb to 4.2gb.&nbsp; I then filtered the date period down to 2 years which reduces it to 2.8gb.&nbsp; Finally I created a copy of this and removed all my dax calculations so only the source tables are in the model file.&nbsp; I then published that and refreshed it and am still getting the same error.&nbsp; I don't understand how I am hitting the 25gb limit when i am only refreshing about 2.8gb of data.

v-sdhruv · Answer

Hi&nbsp;ACSpotlight&nbsp;,
Even if your dataset is trimmed to 2.8GB, additional memory is consumed by:

DAX calculations
Relationships and model metadata
Query execution overhead
Temporary storage during refresh

As pointed out in this thread,
https://community.fabric.microsoft.com/t5/Service/Power-BI-premium-capacity-memory-limitation-during-refresh/m-p/3957281“You need twice as much memory as the size of the semantic model. The correct workaround is to implement incremental refresh and/or selective table and partition refresh"This means you can try to&nbsp;1.Implement Incremental Refresh
2.Simplify DAX calculations
3.Consider Gen2 Autoscale or UpgradeYou can try one of these menthods to avaoid hitting the error.Hope this helps! If the response has addressed your query, please accept it as a solution so that other members can easily find it.Thank you.

v-sdhruv · Answer

Hi&nbsp;ACSpotlight&nbsp;,
&nbsp;I just wanted to check in and ask if the issue you were facing has been resolved. If not, we’d be happy to assist further—please let us know how we can help.
Looking forward to your update!Thank You!

Forum Discussion

Exceed Memory on Refresh

8 Replies

Related content

Trying to understand refresh memory usage

Semantic model Refresh memory limit error

Query Exceeds

Semantic Model Refresh - Memory limit

exceed quotas