How to schedule data refresh with a custom connector data source

[jasonyeung87]

Hi,

I have a Power BI report that accesses an API to grab data. This API has pagination, so I had to add some code in my power query to simulate going through all the pages to grab. I was able to solve this issue and then I published the report to Power BI Service. The report works fine, but I can't schedule a data refresh. From the link below, it mentions there could be a limitation on what we can do in Power BI service:

https://www.thepoweruser.com/2017/09/04/guide-and-resources-for-web-data-extraction-with-power-bi-po...

It recommended I create a custom connector. I created a simple custom connector to query the same API. Using the extension https://github.com/Microsoft/DataConnectors  , I was able to create the custom connector.. After deploying this to Power BI Desktop, changing the data source to use this and then publishing to Power BI Service, I get the following when I try to schedule a data refresh:

It's still not allowing me to schedule a data refresh. I was wondering how do I allow data refreshes for custom connectors, or calls to API's that have pagination?

[Anonymous]

https://community.fabric.microsoft.com/t5/Desktop/Custom-Zendesk-Connector-Power-Query-Code/td-p/240... maybe useful. It has an example custom connector, links to how to use a personal gateway. & a summary of the options in the space for custom connectors. 

[Anonymous]

Hi @jasonyeung87 ,

When you install the on-premises data gateway, you see a Connectors tab in the on-premises data gateway app. In the Load custom data connectors from folder box, select a folder that can be accessed by the user running the gateway service. The default user is NT SERVICE\PBIEgwService. The gateway automatically loads the custom connector files located in that folder. They appear in the list of data connectors.

For the on-premises data gateway, you need to create a data source for your custom connector. On the gateway settings page in the Power BI service, you should see an option when you select the gateway cluster to allow using custom connectors with this cluster. Select that option to enable using custom connectors with this cluster.

When this option is enabled, you see your custom connectors as available data sources that you can create under this gateway cluster. After you create a data source that uses your new custom connector, you can refresh Power BI reports by using that custom connector in the Power BI service.

For more you may refer to :

Use custom data connectors with the on-premises data gateway - Power BI | Microsoft Learn

Best Regards,
Community Support Team _ Caitlyn

If this post helps then please consider Accept it as the solution to help the other members find it more quickly.

[jasonyeung87]

Hi @Anonymous ,

I followed the instructions on the link you sent. I installed the on prem data gateway and added my custom connector to it. After doing this, I went to Power BI Service and checked the "Allow user's custom data connectors..." checkbox in the Gateway Cluster Settings section.

I then tried to create a data source. However there is no data source type named "My Custom Connector". I was wondering what may be the cause of this option not appearing? Would it be a licensing problem? (I currently have Power BI Pro).

Jason

[jasonyeung87]

Hi @Anonymous ,

thanks for the info. The API the connector is accessing is a third party site hosted in the cloud. I'm not accessing any data that is hosted on prem. Would I still need to setup an on prem gateway?

Sincerely,

Jason