Extract data from multiple URLs without having a valid static URL.

[Javiklauta]

Hello everyone!

I am trying to extract data from a list of more than one hundred urls. The goal is to be able to append all the data from all urls in a single table. The problem is that the structure of the urls is like the following: http://<IPaddress>/status.xml. beign the <IPaddress> twhat makes each url different. 

A sample of the urls I need to fetch data from are: 

http://10.204.18.300/status.xml

http://10.204.18.400/status.xml

http://10.204.18.500/status.xmll

...

If I try to have a table with all the urls and create a Custom Colum with Web.Contents([url]), Power BI service will not refresh it as it detects a dynamic url as the source of the query. 

I also tried Chris Webb's proposed solution, but there is nothing static in my urls that can be serve as a valid url to use in the baseurl parameter. 

Does anyone have any potential solution to this?

Thank you very much in advance!

[Javiklauta]

Thank you very much for your support @artemus !

Kindly, could you provide me with an example or hint on how to make the custom connector bypass Power BI credentials? There are tutorials on how to build a custom connector (which I did), but I am still trying to figure out how to make it bypass credentials. 

Should this be enough for this purpose? 

Thanks! 

[artemus]

You would need to do something like:

Extension.InvokeWithCredentials((datasource) => [ AuthenticationKind = "Anonymous" ], () => Web.Contents("https://10.204.18.300/status.xml"))

[Javiklauta]

Thank you very much @artemus ! I haven't fully figured it out yet, but I'll get there thanks to your help. 

[artemus · Accepted solution]

The issue is that we capture the credentials used for each website seperatly, and while you can use different credentials for different levels of the url, there isn't any for the base url.

Your only option is to create a custom connector, which can bypass the credential check. Note that if you do this you must use Power Bi Desktop (not Excel), or if you deploy online, you will need to use a gateway.