PowerBi API Threshold limit error in Python

[manasa1923]

I have a requirement to fetch all the Reports in PowerBi and their respective data source details(like, connection string, source query/table used etc.) using Synapse Pipeline/Notebook(pyspark). I have developed a Pyspark(Synapse notebook) as follows.

1. Fetch all the reports data using Powerbi admin API
   1. API used : https://api.powerbi.com/v1.0/myorg/admin/reports
2. Collect the report data into a list
3. Loop through the list and in the loop I used following API to fetch datasource details.
   1. API Used : https://api.powerbi.com/v1.0/myorg/admin/datasets/'+ datasetId +'/datasources'
4. capture the required details and write to a blob storage

The notebook ran fine without any issues for a week. And after that, I'm getting the error "Threshold limit exceeded, please try after XYZ seconds" in the step-3 (where I'm looping through all the available list of reports and fetching their respective connection details). Could you please help me in fixing this issue.

[lbendlin]

you are likely hitting a 429.  Introduce a delay in your loop to avoid that.  And think about using incremental refresh.