How does the large dataset refresh in a Power Bi premium P1 capacity

[NewUser777]

Scenario :

P1 SKU Premium Capacity

Max. Mem limit :  25 GB

CPU Parallel refresh limit : 6 at a time 

Total Datasets  refreshed daily in Prod @ 9:00 am EST : 10  

All datasets are of 1 gb size except for one big dataset which is of 23 gb . The setting large dataset has been enabled for the 23gb dataset and it is scheduled at @9:00 am EST .

Questions:

1. How does the  big dataset executes on this capacity when the total max gb allocated is 25gb?

2. If all are scheduled at the sametime which takes preference ? Smaller ones if so what order if they have the same data refresh schedule set ( 9:00 am est daily)

[djurecicK2 · Accepted solution]

Hi @NewUser777 ,

 The 23 GB dataset will not be able to refresh. Refreshing causes the memory used by the dataset to double:

https://learn.microsoft.com/en-us/power-bi/enterprise/service-premium-what-is#dataset-memory-allocat...

[djurecicK2 · Accepted solution]

Hi @NewUser777 ,

 The 23 GB dataset will not be able to refresh. Refreshing causes the memory used by the dataset to double:

https://learn.microsoft.com/en-us/power-bi/enterprise/service-premium-what-is#dataset-memory-allocat...

[NewUser777]

@djurecicK2 I have a model which is over 20gigs and it runs in this capacity and takes almost 3-4 hours to complete . The setting on the model I see is enable large dataset format.Hence I m wondering how would this run in this capacity which imposes 25 gb limit . 

[NewUser777 · Accepted solution]

djurecicK2 Finally got to know the truth , the dataset which was over 20GB was refreshed manually via SSMS . From Power BI service like you have stated it didnt work . 

I will close this thread and thanks for your help

https://learn.microsoft.com/en-us/power-bi/enterprise/service-premium-connect-tools#dataset-refresh

The XMLA endpoint enables a wide range of scenarios for fine-grain refresh capabilities using SSMS, automation with PowerShell, Azure Automation, and Azure Functions using TOM. For example, you can refresh certain incremental refresh historical partitions without having to reload all historical data.

Unlike configuring refresh in the Power BI service, refresh operations through the XMLA endpoint are not limited to 48 refreshes per day, and the scheduled refresh timeout is not imposed.

Date, time, and status for dataset refresh operations that include a write transaction through the XMLA endpoint are recorded and shown in dataset Refresh history.