Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Shape the future of the Fabric Community! Your insights matter. That’s why we created a quick survey to learn about your experience finding answers to technical questions. Take survey.

Reply
MAdam90
Regular Visitor

Issue on "Large dataset storage format" setting behavior in Power BI workspaces

I have encountered an error during data refresh when using the "Large dataset storage format" setting in my Power BI workspace, which was set to "Small dataset" by default. I would like to seek clarification regarding this issue.

 

In my workspace, the default dataset storage format is set to "Small dataset," and I do not have the necessary rights or permissions to change this setting. However, I have enabled the "Large dataset storage format" setting at the individual dataset level to accommodate larger datasets. Despite this configuration, I consistently encounter an error during data refresh. The error message received during the refresh process is about out of memory . 

 

below screenshot of  workspace configuration : 

MAdam90_0-1686349816310.png

and the dataset level: 

 

MAdam90_1-1686350106634.png

I am uncertain if the error is a result of the default "Small dataset" of the workspace .

 

Thank you for your assistance

1 ACCEPTED SOLUTION

Hi @MAdam90 ,

 

Please try to consider using other methods:
(1)Using the enhanced refresh REST API , you can perform fine grained data refreshes, so that the memory needed by the refresh can be minimized to fit within your capacity's size.
(2) Optimize the data model by removing unnecessary columns, reducing the number of calculated columns, and using efficient data types. This helps to reduce the memory footprint of the dataset.
(3) Another option is to incrementally refresh the data. You can configure Power BI to refresh only a subset of data, such as the latest data or changed data, rather than refreshing the entire dataset. This helps reduce the memory requirements during the refresh process.

 

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly. 

View solution in original post

4 REPLIES 4
v-tangjie-msft
Community Support
Community Support

Hi @MAdam90 ,

 

The error message you received about insufficient memory indicates that the refresh requires more memory than the capacity can fit.


It's possible that the "Small dataset" setting in your workspace is not the direct cause of the issue, but rather the memory limitations of your Power BI capacity.


In the meantime, you can try using the XMLA endpoint to refresh the new partitions one by one, instead of having multiple partitions being refreshed in parallel when the refresh is done by the Power BI Service. If refreshing one partition still gives the error, that means the capacity needs to be increased in size to fit the refresh.

 

Refer it :Large datasets in Power BI Premium - Power BI | Microsoft Learn

 

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly. 

hello @v-tangjie-msft,  XMLA endpoints for partitioning may not be applicable in my case, as my data is stored as CSV files in Azure, and CSV files don't natively support partitioning. XMLA is more suitable for Power BI datasets directly connected to a database.

Regards,

Hi @MAdam90 ,

 

Please try to consider using other methods:
(1)Using the enhanced refresh REST API , you can perform fine grained data refreshes, so that the memory needed by the refresh can be minimized to fit within your capacity's size.
(2) Optimize the data model by removing unnecessary columns, reducing the number of calculated columns, and using efficient data types. This helps to reduce the memory footprint of the dataset.
(3) Another option is to incrementally refresh the data. You can configure Power BI to refresh only a subset of data, such as the latest data or changed data, rather than refreshing the entire dataset. This helps reduce the memory requirements during the refresh process.

 

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly. 

MAdam90
Regular Visitor

I just want to know if it's the small data that is initialized for the workspace that could be the cause of this issue. I just need a confirmation, thanks

Helpful resources

Announcements
November Carousel

Fabric Community Update - November 2024

Find out what's new and trending in the Fabric Community.

Dec Fabric Community Survey

We want your feedback!

Your insights matter. That’s why we created a quick survey to learn about your experience finding answers to technical questions.

Live Sessions with Fabric DB

Be one of the first to start using Fabric Databases

Starting December 3, join live sessions with database experts and the Fabric product team to learn just how easy it is to get started.