Custom Connector heavy load

[Gottegubben]

So i have developed a custom connector that connects to an REST API. I have about 10 tables that uses the same Custom Connector for 10 diffrent API endpoints. Each table holds up to 3.5 milion rows of data and the totalt amount of data for all tables is about 10 milion rows. The custom connector uses paging and requests 5000 rows at the time from the API.

It takes about 1 hour to refresh all the tables when developing on the Power Bi Desktop, all good.

When the report is uploaded to Power Bi Web cannot refresh the dataset. The Gateway is setup the right way with no errors.

I get this error message:

Underlying error message: The request was aborted: The request was canceled.

Microsoft.Data.Mashup.ValueError.Reason: DataSource.Error

Any ideas?

[Anonymous]

Hi @Gottegubben ,

According to your statement, I know you can succeed to refresh your report in Power BI Desktop. However, when you publish your report to Power BI Service and configure correctly the gateway, you will get error message  when you refresh the dataset.

I think refresh your report will make a heavy workload. What is the capacity of the workspace where your dataset is located, Shared Capacity or Premium Capacity?

If you only have Pro license, your workspace should be in Shared Capacity. 

Since all the users of Power BI are sharing the same set of machines (also referred to as the “shared capacity”), controls are in place to make sure that a few users cannot monopolize all the shared capacity in the cluster and disrupt the quality of service for others. Many of the published restrictions of Power BI, like the 1GB model size or the limitation on the number of refreshes per day, are driven by the need to secure excellent quality of service for everyone. 

So your refresh with heavy workload may be failed, due to the limitations of resources.

Since the dedicated resources for Power BI Premium will be used only by you, all the throttles and limitations that we at Microsoft put in to protect you and the other tenants can now be removed.

I think you can try Power BI Premium to get better performance in refresh.

For reference:

Microsoft Power BI Premium whitepaper

What is Power BI Premium?

Power BI Premium FAQ

Best Regards,
Rico Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Gottegubben]

How many times can I do a request in my custom connector? I do about 1500 request under 1h.

How do i know how much space i need or how much space my dataset is using?

Thanks!