Forum Discussion
Limit on get request (Admin - Datasets GetDatasourcesAsAdmin)
Hi,
I recently started to discover Power BI Admin APIs. And I wanted to bring in the GetDatasources, to get all the data sources for the datasets In my tenant. I do it using Python.
Although when I try to get all the data sources, it stops after around 1700 datasets. Even though I have inserted sleep into the script.
Does anyone know if I am hitting a limit for requests? Or does anyone have an idea to solve this? We have around 11000 datasets, so we are far from home with 1700.
Michael
With 11,000 datasets, you have definitely moved past the "standard" Admin APIs limit.
The GetDatasourcesAsAdmin explicitly states a limit of 300 requests per hour.
The Solution: The Scanner API (Metadata Scanning)
For a tenant of your size, you should switch to the Scanner API. It is designed specifically to handle metadata for up to 10,000+ workspaces efficiently.
Even I use the Scanner APIs to scan our entire tenant, since we also have a large tenant with more than 17k reports and more than 10k datasets.
Microsoft calls this Metadata Scanning. Refer to the link - https://learn.microsoft.com/en-us/fabric/governance/metadata-scanning-overview for detailed steps, I have tried to give a brief oveview below -
Unlike the standard "Loop and Fetch" method, the Scanner API is asynchronous and works in four stages:
-
Get Modified Workspaces: Call
admin/workspaces/modified. This gives you a list of workspace IDs that have changed (or all of them if it's your first run). -
Trigger the Scan: Call
admin/workspaces/getInfo. You post a list of Workspace IDs (in batches of 100). This returns aScanID. -
Check Status: Poll
admin/workspaces/scanStatus/{ScanID}until it says "Succeeded." -
Get the Result: Call
admin/workspaces/scanResult/{ScanID}. This returns a massive JSON containing everything: datasets, tables, and—crucially for you—the Datasource details.
I would also recommend referring to Rui Ramano's awesome Power BI Monitor Solution that makes use of Scanner APIs for Admin Monitoring - https://github.com/RuiRomano/pbimonitor
Let me know if this helps or if you need anything else!
Thanks & Regards,
Bipin
-
2 Replies
- Bipin-LalaSolution Sage
With 11,000 datasets, you have definitely moved past the "standard" Admin APIs limit.
The GetDatasourcesAsAdmin explicitly states a limit of 300 requests per hour.
The Solution: The Scanner API (Metadata Scanning)
For a tenant of your size, you should switch to the Scanner API. It is designed specifically to handle metadata for up to 10,000+ workspaces efficiently.
Even I use the Scanner APIs to scan our entire tenant, since we also have a large tenant with more than 17k reports and more than 10k datasets.
Microsoft calls this Metadata Scanning. Refer to the link - https://learn.microsoft.com/en-us/fabric/governance/metadata-scanning-overview for detailed steps, I have tried to give a brief oveview below -
Unlike the standard "Loop and Fetch" method, the Scanner API is asynchronous and works in four stages:
-
Get Modified Workspaces: Call
admin/workspaces/modified. This gives you a list of workspace IDs that have changed (or all of them if it's your first run). -
Trigger the Scan: Call
admin/workspaces/getInfo. You post a list of Workspace IDs (in batches of 100). This returns aScanID. -
Check Status: Poll
admin/workspaces/scanStatus/{ScanID}until it says "Succeeded." -
Get the Result: Call
admin/workspaces/scanResult/{ScanID}. This returns a massive JSON containing everything: datasets, tables, and—crucially for you—the Datasource details.
I would also recommend referring to Rui Ramano's awesome Power BI Monitor Solution that makes use of Scanner APIs for Admin Monitoring - https://github.com/RuiRomano/pbimonitor
Let me know if this helps or if you need anything else!
Thanks & Regards,
Bipin
-
- Kagiyama_yutakaResponsive Resident
I think that call just stops its own enum around ~1700… if u need the full set, run a Scanner API workspace‑scan and pull the scanResult — it’s the only route that never drops on large tenants.