Dedicated capacity workspaces now give Internal Error 500 when I attempt to upload a report

[Anonymous]

We have three dedicated premium capacity resources:

• Test (in northeurope)
• Live for Europe (also in northeurope)
• Live for US (in eastus2)

Some time in the last few days, PowerBI workspaces in the Live Europe capacity have begun to return 500 errors when we try to upload a report. The other two seem to be fine. Also workspaces without dedicated capacity are fine.

Prior to this occurring, all three have worked fine. We are not aware of any configuration changes made recently. Indeed there don't appear to be any configurable settings in the "Power BI Embedded" resource type in Azure apart from the name, which is unchanged, and we are successfully able to create new workspaces and enable them using the same "Power BI Embedded" resource name as before.

When attempting to upload a report manually, here are the error details that the UI told me to include in communication with support:

Activity ID: 1d1603df-1004-429d-a1f4-26458eb23e0c
Request ID: 569201aa-e4bd-bd3f-ac42-ab4ddd0f9d60
Correlation ID: e5a9db11-6ef4-a882-6264-031be46c5306
Status code: 500
Time: Wed Jan 02 2019 12:15:06 GMT+0000 (Greenwich Mean Time)
Version: 13.0.7683.237
Cluster URI: https://wabi-north-europe-redirect.analysis.windows.net

This is a pretty severe blocking issue for us!

[Anonymous]

Further info on this - it's not just uploading reports that is broken, but also actually using existing ones. The reports use Direct Query. They all complain that they cannot load the model for the report.

To fix, I had to create a new dedicated capacity resource and recreate the contents of the workspaces.

I'd like to delete the old (apparently broken) one as soon as possible, as we are paying for it. How can I help you escalate investigation into this issue while the repro still exists?

[dearwicker]

I have "Paused" the problem dedicated capacity resource in the hope that this means I won't be paying anything...

[v-qiuyu-msft]

Hi @Anonymous,

Please check if your capacity hit the memory limitation of the capacity: https://whitepages.unlimitedviz.com/2017/09/understanding-the-power-bi-capacity-based-skus/

You can install the Premium Capacity Metrics app to monitor your capacity limits: https://docs.microsoft.com/en-us/power-bi/service-admin-premium-monitor-capacity

Best Regards,
Qiuyun Yu

[dearwicker]

@@v-qiuyu-msft Hi, thanks for responding. I tried adding the app. At first it doesn't appear. I refreshed a few times. Then it appeared. When I opened it, it displayed a white banner with a circular icon and the rest of the page was empty grey-ish space. I refreshed a few more times and it became no longer available:

   Sorry, we couldn't find that app

     It may have been deleted or you may not have access to it.

Is this expected behaviour? I tried adding it a second time and the whole experience repeated.

If I can't get this app to work, is there some other way to diagnose memory usage?

What is the expected memory usage for purely Direct Query reports in workspaces? Is it expected to grow in proportion with the number of reports that exist across all workspaces?

[dearwicker]

@v-qiuyu-msftI found the Memory metric in Azure portal. It was at 3.2 GB in the dedicated capacity resource that was failing, and the limit is 3 GB.

I've created a separate resource to support our most urgent needs, and it has about 5 workspaces on it (a tenth of what we had on the failing resource). But it's already at 2 GB usage according to the Memory metric, i.e. a 10th of the workspaces already at 60% of memory usage

Is that metric an accurate guide to the usage? What should I expect the relationship to be between the number of workspaces (which are all basically the same) and the memory usage?

[dearwicker]

@v-qiuyu-msft   by the way, the Premium Capacity Metrics app still hasn't re-appeared, 24 hours after I last tried to add it.

[dearwicker]

@v-qiuyu-msftI've attached a screen grab of the Memory metric from Azure Portal:

The strange thing about this is that we had 50+ workspaces for many months without problems. Then on December 22nd 23:00 UTC the memory usage started to rise significantly until it hit a maximum and stayed there indefinitely. But this was during a time when our business had effectively shutdown for the holidays so I don't understand why this suddenly started happening at that time.

Has anything changed in how PowerBI utilises memory?

[dearwicker]

Even more strange, here's a screengrab from another entirely separate dedicated capacity resource, with far fewer workspaces and again no one using it over the period in late December:

It looks like another steady rise in memory usage when we are not using the service. It seems strangely coincidental that two separate such resources show the same pattern at the same time.

[v-qiuyu-msft]

Hi @dearwicker,

I would suggest you create a support ticket to let engineers look into the issue on your side. 

Best Regards,
Qiuyun Yu