help : dataset refresh issue

[askme1217]

i have a dataset around 5 GB size. when i refresh it, it is getting failed with memory usage error. it uses around 50 GB of memory which is maximum in capacity. Why is it using 50 gb of memory even though the dataset size is 5 Gb only? The fact table has around 200 million rows, it does not have any heavy powerquery transformations, just  native sql queries with filters. How can i reduce memory usage and refresh it without error?

[pallavi_r]

Hi @askme1217 ,

Are you using calculated columns or calculated dax tables? Can you try refreshing dataset after removing calculated columns and tables. Calculated column takes memory depending on power query transformation.

Hope you have taken care of unused columns, incremental refresh and inefficient relational modelling (exp. many-many).

Thanks,

Pallavi

[askme1217]

help

[v-sshirivolu · Accepted solution]

Hi @askme1217 ,
Thanks for reaching out to the Microsoft fabric community forum.

With large datasets, Power BI can use much more memory than the dataset size during refresh. Setting up incremental refresh and removing unused columns or large tables can help reduce memory use. If issues persist, consider upgrading your capacity.

Power BI Incremental Refresh Documentation - Incremental refresh and real-time data in Power BI
Optimization Techniques for Import Models (VertiPaq Engine )-
Data reduction techniques for Import modeling - Power BI | Microsoft Learn
Power BI Premium Capacity and Limits - What is Power BI Premium? - Power BI | Microsoft Learn

If the response has addressed your query, please Accept it as a solution and give a 'Kudos' so other members can easily find it

Best Regards,
Sreeteja.
Community Support Team

[v-sshirivolu]

Hi @askme1217 ,

Just wanted to check if you had the opportunity to review the suggestion provided?

If the response has addressed your query, please Accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

[v-sshirivolu]

Hi @askme1217 ,

I hope the information provided above assists you in resolving the issue. If you have any additional questions or concerns, please do not hesitate to contact us. We are here to support you and will be happy to help with any further assistance you may need.

[v-sshirivolu]

Hi @askme1217 ,

I hope the information provided above assists you in resolving the issue. If you have any additional questions or concerns, please do not hesitate to contact us. We are here to support you and will be happy to help with any further assistance you may need.

[Deku]

Are you sure the filters are folding? If not the filtering will occur in the service rather than the database.

Please see these docs

https://learn.microsoft.com/en-us/power-query/native-query-folding

[askme1217]

Yes, when i check the 'View Native Query' it is enabled. so i think query folding happening.