Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

The ultimate Microsoft Fabric, Power BI, Azure AI & SQL learning event! Join us in Las Vegas from March 26-28, 2024. Use code MSCUST for a $100 discount. Register Now

Reply
NewUser777
Resolver I
Resolver I

How does the large dataset refresh in a Power Bi premium P1 capacity

Scenario :

P1 SKU Premium Capacity

Max. Mem limit :  25 GB

CPU Parallel refresh limit : 6 at a time 

 

Total Datasets  refreshed daily in Prod @ 9:00 am EST : 10  

 

All datasets are of 1 gb size except for one big dataset which is of 23 gb . The setting large dataset has been enabled for the 23gb dataset and it is scheduled at @9:00 am EST .

 

 

Questions:

1. How does the  big dataset executes on this capacity when the total max gb allocated is 25gb?

 

2. If all are scheduled at the sametime which takes preference ? Smaller ones if so what order if they have the same data refresh schedule set ( 9:00 am est daily)

 

 

2 ACCEPTED SOLUTIONS
djurecicK2
Super User
Super User

Hi @NewUser777 ,

 The 23 GB dataset will not be able to refresh. Refreshing causes the memory used by the dataset to double:

 

https://learn.microsoft.com/en-us/power-bi/enterprise/service-premium-what-is#dataset-memory-allocat...

 

 

View solution in original post

djurecicK2 Finally got to know the truth , the dataset which was over 20GB was refreshed manually via SSMS . From Power BI service like you have stated it didnt work . 

 

I will close this thread and thanks for your help

 

https://learn.microsoft.com/en-us/power-bi/enterprise/service-premium-connect-tools#dataset-refresh

 

The XMLA endpoint enables a wide range of scenarios for fine-grain refresh capabilities using SSMS, automation with PowerShell, Azure Automation, and Azure Functions using TOM. For example, you can refresh certain incremental refresh historical partitions without having to reload all historical data.

Unlike configuring refresh in the Power BI service, refresh operations through the XMLA endpoint are not limited to 48 refreshes per day, and the scheduled refresh timeout is not imposed.

Date, time, and status for dataset refresh operations that include a write transaction through the XMLA endpoint are recorded and shown in dataset Refresh history.

View solution in original post

3 REPLIES 3
djurecicK2
Super User
Super User

Hi @NewUser777 ,

 The 23 GB dataset will not be able to refresh. Refreshing causes the memory used by the dataset to double:

 

https://learn.microsoft.com/en-us/power-bi/enterprise/service-premium-what-is#dataset-memory-allocat...

 

 

@djurecicK2 I have a model which is over 20gigs and it runs in this capacity and takes almost 3-4 hours to complete . The setting on the model I see is enable large dataset format.Hence I m wondering how would this run in this capacity which imposes 25 gb limit . 

djurecicK2 Finally got to know the truth , the dataset which was over 20GB was refreshed manually via SSMS . From Power BI service like you have stated it didnt work . 

 

I will close this thread and thanks for your help

 

https://learn.microsoft.com/en-us/power-bi/enterprise/service-premium-connect-tools#dataset-refresh

 

The XMLA endpoint enables a wide range of scenarios for fine-grain refresh capabilities using SSMS, automation with PowerShell, Azure Automation, and Azure Functions using TOM. For example, you can refresh certain incremental refresh historical partitions without having to reload all historical data.

Unlike configuring refresh in the Power BI service, refresh operations through the XMLA endpoint are not limited to 48 refreshes per day, and the scheduled refresh timeout is not imposed.

Date, time, and status for dataset refresh operations that include a write transaction through the XMLA endpoint are recorded and shown in dataset Refresh history.

Helpful resources

Announcements
Fabric Community Conference

Microsoft Fabric Community Conference

Join us at our first-ever Microsoft Fabric Community Conference, March 26-28, 2024 in Las Vegas with 100+ sessions by community experts and Microsoft engineering.

February 2024 Update Carousel

Power BI Monthly Update - February 2024

Check out the February 2024 Power BI update to learn about new features.

Fabric Career Hub

Microsoft Fabric Career Hub

Explore career paths and learn resources in Fabric.

Fabric Partner Community

Microsoft Fabric Partner Community

Engage with the Fabric engineering team, hear of product updates, business opportunities, and resources in the Fabric Partner Community.

Top Solution Authors
Top Kudoed Authors