Forum Discussion

mhani's avatar
mhani
Frequent Visitor
3 years ago
Solved

Limit on get request (Admin - Datasets GetDatasourcesAsAdmin)

Hi,

 

I recently started to discover Power BI Admin APIs. And I wanted to bring in the GetDatasources, to get all the data sources for the datasets In my tenant. I do it using Python.

 

Although when I try to get all the data sources, it stops after around 1700 datasets. Even though I have inserted sleep into the script.

 

Does anyone know if I am hitting a limit for requests? Or does anyone have an idea to solve this? We have around 11000 datasets, so we are far from home with 1700.

 

Michael

  • With 11,000 datasets, you have definitely moved past the "standard" Admin APIs limit.

     

    The  GetDatasourcesAsAdmin explicitly states a limit of 300 requests per hour.

     

    The Solution: The Scanner API (Metadata Scanning)

    For a tenant of your size, you should switch to the Scanner API. It is designed specifically to handle metadata for up to 10,000+ workspaces efficiently.

     

    Even I use the Scanner APIs to scan our entire tenant, since we also have a large tenant with more than 17k reports and more than 10k datasets.

     

    Microsoft calls this Metadata Scanning. Refer to the link - https://learn.microsoft.com/en-us/fabric/governance/metadata-scanning-overview for detailed steps, I have tried to give a brief oveview below - 

     

    Unlike the standard "Loop and Fetch" method, the Scanner API is asynchronous and works in four stages:

     

    1. Get Modified Workspaces: Call admin/workspaces/modified. This gives you a list of workspace IDs that have changed (or all of them if it's your first run).

    2. Trigger the Scan: Call admin/workspaces/getInfo. You post a list of Workspace IDs (in batches of 100). This returns a ScanID.

    3. Check Status: Poll admin/workspaces/scanStatus/{ScanID} until it says "Succeeded."

    4. Get the Result: Call admin/workspaces/scanResult/{ScanID}. This returns a massive JSON containing everything: datasets, tables, and—crucially for you—the Datasource details.

    I would also recommend referring to Rui Ramano's awesome Power BI Monitor Solution that makes use of Scanner APIs for Admin Monitoring - https://github.com/RuiRomano/pbimonitor

     

    Let me know if this helps or if you need anything else!

     

    Thanks & Regards,

    Bipin

2 Replies

  • With 11,000 datasets, you have definitely moved past the "standard" Admin APIs limit.

     

    The  GetDatasourcesAsAdmin explicitly states a limit of 300 requests per hour.

     

    The Solution: The Scanner API (Metadata Scanning)

    For a tenant of your size, you should switch to the Scanner API. It is designed specifically to handle metadata for up to 10,000+ workspaces efficiently.

     

    Even I use the Scanner APIs to scan our entire tenant, since we also have a large tenant with more than 17k reports and more than 10k datasets.

     

    Microsoft calls this Metadata Scanning. Refer to the link - https://learn.microsoft.com/en-us/fabric/governance/metadata-scanning-overview for detailed steps, I have tried to give a brief oveview below - 

     

    Unlike the standard "Loop and Fetch" method, the Scanner API is asynchronous and works in four stages:

     

    1. Get Modified Workspaces: Call admin/workspaces/modified. This gives you a list of workspace IDs that have changed (or all of them if it's your first run).

    2. Trigger the Scan: Call admin/workspaces/getInfo. You post a list of Workspace IDs (in batches of 100). This returns a ScanID.

    3. Check Status: Poll admin/workspaces/scanStatus/{ScanID} until it says "Succeeded."

    4. Get the Result: Call admin/workspaces/scanResult/{ScanID}. This returns a massive JSON containing everything: datasets, tables, and—crucially for you—the Datasource details.

    I would also recommend referring to Rui Ramano's awesome Power BI Monitor Solution that makes use of Scanner APIs for Admin Monitoring - https://github.com/RuiRomano/pbimonitor

     

    Let me know if this helps or if you need anything else!

     

    Thanks & Regards,

    Bipin

  • Kagiyama_yutaka's avatar
    Kagiyama_yutaka
    Responsive Resident

    I think that call just stops its own enum around ~1700… if u need the full set, run a Scanner API workspace‑scan and pull the scanResult — it’s the only route that never drops on large tenants.