Dynamic API scrapping (Python script)?

[Anonymous]

Hello all,

I'm starting PBI and Python.

Requirements: 

I would like to scrap all json data coming for API url:

https://udex1z31.eu.****.corp/1z31/api/subscriptions/{subscriptionId}/exports/{launchDate}/logs

where  {subscriptionId} = text

where {launchDate} = number

=> for each "SubscriptionId" and "launchDate", I would like to get json:

[
{
"utcdate": "2020-06-10T15:21:26.147Z",
"beat.hostname": "string",
"username": "string",
"message": "string",
"loglevel": "Debug",
"errortype": "Functional",
"subscriptionId": "string",
"reminderkey": "string",
"data": "string",
"idExport": 0,
"idContext": "string"
}
]

The goal is to avoid entering these two parameters.

I would like to fill a table with all these data to be used into PBI report.

I was thinking about Python script. If anyone as an idea, it would 

[mahoneypat]

You don't need Python to do this one in Power BI.  YOu can pass parameters into a web API call.  You can start with a table of your id and date values and concatentate them into the web call, and then expand/append all the results together.  Here are a couple example.

https://community.powerbi.com/t5/Desktop/Passing-dynamic-date-parameters-to-Web-query/td-p/221349

https://community.powerbi.com/t5/Power-Query/Create-google-API-to-lookup-latitude-and-longitute-of-z...

If this works for you, please mark it as the solution.  Kudos are appreciated too.  Please let me know if not.

Regards,

Pat

[lbendlin]

Do you want to run this just on your PC or are you planning to publish it to the Power BI service at some point?  I am asking because there are some limitations for Python script on the service that might prevent you from running dynamic web calls.

You will want to look into pulling the data via Power Query instead. It has web connectors, and JSON parsing.

Assuming you have a table with subscription id and launch date columns as your data source, you can then iterate through that table, call a custom function to retrieve the data, and then parse the result into additional columns as needed.