Forum Discussion
Limit on get request (Admin - Datasets GetDatasourcesAsAdmin)
- 5 months ago
With 11,000 datasets, you have definitely moved past the "standard" Admin APIs limit.
The GetDatasourcesAsAdmin explicitly states a limit of 300 requests per hour.
The Solution: The Scanner API (Metadata Scanning)
For a tenant of your size, you should switch to the Scanner API. It is designed specifically to handle metadata for up to 10,000+ workspaces efficiently.
Even I use the Scanner APIs to scan our entire tenant, since we also have a large tenant with more than 17k reports and more than 10k datasets.
Microsoft calls this Metadata Scanning. Refer to the link - https://learn.microsoft.com/en-us/fabric/governance/metadata-scanning-overview for detailed steps, I have tried to give a brief oveview below -
Unlike the standard "Loop and Fetch" method, the Scanner API is asynchronous and works in four stages:
-
Get Modified Workspaces: Call
admin/workspaces/modified. This gives you a list of workspace IDs that have changed (or all of them if it's your first run). -
Trigger the Scan: Call
admin/workspaces/getInfo. You post a list of Workspace IDs (in batches of 100). This returns aScanID. -
Check Status: Poll
admin/workspaces/scanStatus/{ScanID}until it says "Succeeded." -
Get the Result: Call
admin/workspaces/scanResult/{ScanID}. This returns a massive JSON containing everything: datasets, tables, and—crucially for you—the Datasource details.
I would also recommend referring to Rui Ramano's awesome Power BI Monitor Solution that makes use of Scanner APIs for Admin Monitoring - https://github.com/RuiRomano/pbimonitor
Let me know if this helps or if you need anything else!
Thanks & Regards,
Bipin
-
With 11,000 datasets, you have definitely moved past the "standard" Admin APIs limit.
The GetDatasourcesAsAdmin explicitly states a limit of 300 requests per hour.
The Solution: The Scanner API (Metadata Scanning)
For a tenant of your size, you should switch to the Scanner API. It is designed specifically to handle metadata for up to 10,000+ workspaces efficiently.
Even I use the Scanner APIs to scan our entire tenant, since we also have a large tenant with more than 17k reports and more than 10k datasets.
Microsoft calls this Metadata Scanning. Refer to the link - https://learn.microsoft.com/en-us/fabric/governance/metadata-scanning-overview for detailed steps, I have tried to give a brief oveview below -
Unlike the standard "Loop and Fetch" method, the Scanner API is asynchronous and works in four stages:
-
Get Modified Workspaces: Call
admin/workspaces/modified. This gives you a list of workspace IDs that have changed (or all of them if it's your first run). -
Trigger the Scan: Call
admin/workspaces/getInfo. You post a list of Workspace IDs (in batches of 100). This returns aScanID. -
Check Status: Poll
admin/workspaces/scanStatus/{ScanID}until it says "Succeeded." -
Get the Result: Call
admin/workspaces/scanResult/{ScanID}. This returns a massive JSON containing everything: datasets, tables, and—crucially for you—the Datasource details.
I would also recommend referring to Rui Ramano's awesome Power BI Monitor Solution that makes use of Scanner APIs for Admin Monitoring - https://github.com/RuiRomano/pbimonitor
Let me know if this helps or if you need anything else!
Thanks & Regards,
Bipin