Dataset refresh in PowerBI Embedded workspace fails because it exceeds memory limit (5GB)

[Imme]

I have a dataset and a couple of dataflows in a PowerBI Embedded workspace. The dataflows serve as sources for the dataset. The reports connected to this dataset are embedded on our internal employee portal.

Previously both dataset and dataflows were hosted in a PPU workspace, and all refreshes worked fine. However, the limited number of embed tokens ran out so I've had to switch it to an Embedded workspace.

Now my dataset no longer refreshes because it exceeds the memory limit (we have the A2 SKU with 5GB memory and 2 v-cores) and it takes the dataflows about 5 times as long to refresh. The dataset is only 281MB in storage size, so I'm struggling to understand how it takes more than 5GB to refresh.

To test I did temporarily up the memory limit to 25GB, which did allow refresh of the dataset. It is however not a feasible option to up the SKU permanently as it costs 10x more.

Can someone help me understand why my model is pulling more than 5GB of memory for a refresh? And why my dataflows take 5x longer to refresh in the embedded workspace than in PPU workspace?

Is there a way in which I can keep this data structure (dataflows and dataset running in PowerBI workspaces) and still embed my reports? Or do you have any suggestions for other options how to host my dataset (I've been looking at Azure data factory?)

Thanks in advance!

[hmiyoshi]

I also encountered the same issue while trying to update the Semantic Model for Power BI Embedded SKU A1. Despite the dataset being around 300MB, it consumed 2.7GB of memory and resulted in an error.

Upon investigation, I found a thread where the following explanation was provided:

"When many relationships are defined, it consumes more memory.Additionally, if data refresh and relationship calculations are performed simultaneously, a large amount of data is loaded into memory at the same time, leading to memory exhaustion."

Based on this, I followed these steps, and I was able to successfully perform the update using PowerBI API:

1. Perform an update using the following endpoint 

   POST https://api.powerbi.com/v1.0/myorg/datasets/{datasetId}/refreshes
   {
     "type": "DataOnly"
   }

2. Monitor the  endpoint and wait for the dataset refresh to complete.
   

   GET https://api.powerbi.com/v1.0/myorg/datasets/{datasetId}/refreshes

3. Perform an update using the following endpoint

   POST https://api.powerbi.com/v1.0/myorg/datasets/{datasetId}/refreshes
   {
     "type": "Calculate"
   }

   I hope this method works in your environment as well.
   
   Reference:
   https://learn.microsoft.com/ja-jp/rest/api/power-bi/datasets/refresh-dataset
   https://learn.microsoft.com/ja-jp/rest/api/power-bi/datasets/get-refresh-history