How to access Defender for Cloud Apps data and store it for long-term Power BI analysis?

[gulce]

🔹 1. First — How do you access Defender data at all?

I'm trying to pull discovery data (e.g. which users accessed which apps, like ChatGPT) using the API. I generate a valid access token using client credentials (Azure AD App Registration), and I tried both:

• Classic MCAS endpoint:
  https://<tenant>.portal.cloudappsecurity.com/api/discovery_users/
• Microsoft Graph endpoint:
  https://graph.microsoft.com/beta/security/cloudAppSecurity/discoveryUsers

So my first question is:

Which endpoint and method actually returns usable Defender for Cloud Apps data?

---

🔹 2. Then — How do you store this data for history tracking?

Once I get access working, I’d like to:

• Pull data regularly using an ETL tool (e.g. Talend)
• Save it to a database (e.g. SQL Server)
• Connect Power BI to that database for long-term trend analysis

My second set of questions:

• Has anyone built an ETL flow for Defender data?
• What challenges did you face with throttling, authentication, paging?
• What permissions or scopes are required in Azure?

Any working examples, architectures, or documentation would be really appreciated. Thanks so much!

Best,

Gülce

[johnbasha33]

Hi @gulce 

Python Example: Defender for Cloud Apps (MCAS) API

import requests
import json
import csv

# === Azure AD App Registration ===
tenant_id = "<your-tenant-id>"
client_id = "<your-client-id>"
client_secret = "<your-client-secret>"
mcas_domain = "<yourtenant>"  # without .onmicrosoft.com or any suffix

# === Token Endpoint for OAuth2 ===
token_url = f"https://login.microsoftonline.com/{tenant_id}/oauth2/token"

# === Get OAuth Token for MCAS API ===
token_data = {
    'grant_type': 'client_credentials',
    'client_id': client_id,
    'client_secret': client_secret,
    'resource': f'https://{mcas_domain}.portal.cloudappsecurity.com'
}

response = requests.post(token_url, data=token_data)
access_token = response.json().get('access_token')

if not access_token:
    print("Failed to get token:", response.text)
    exit()

headers = {
    'Authorization': f'Bearer {access_token}',
    'Content-Type': 'application/json'
}

# === Discovery API Endpoint ===
api_url = f"https://{mcas_domain}.portal.cloudappsecurity.com/api/discovery_users/"

all_data = []

while api_url:
    print(f"Fetching page: {api_url}")
    resp = requests.get(api_url, headers=headers)
    if resp.status_code != 200:
        print("Failed to fetch data:", resp.text)
        break

    data = resp.json()
    all_data.extend(data.get('data', []))

    api_url = data.get('nextLink')  # MCAS pagination

# === Save to CSV ===
csv_file = 'discovery_users.csv'
if all_data:
    keys = all_data[0].keys()
    with open(csv_file, 'w', newline='', encoding='utf-8') as f:
        dict_writer = csv.DictWriter(f, fieldnames=keys)
        dict_writer.writeheader()
        dict_writer.writerows(all_data)
    print(f"Saved {len(all_data)} records to {csv_file}")
else:
    print("No data received.")

Permissions Required in Azure AD App Registration:

• Directory.Read.All (under Microsoft Graph)

• Grant MCAS API access permissions in the Defender for Cloud Apps portal

• Admin consent is required

Did I answer your question? Mark my post as a solution! Appreciate your Kudos !!

[gulce]

Thank you for your reply.

I’m able to pull data via the API, but I noticed that not all records are returned.

For example, in the Generative AI category, I see 127 records on the portal, but the API only returns 8–9 records.
Also, not all users are returned either.

Do I need to implement pagination?
If so, how exactly should it be done?

Here are the URLs I used:

• https://{mcas_domain}.portal.cloudappsecurity.com/api/v1/discovery/discovered_apps/

• https://{mcas_domain}.portal.cloudappsecurity.com/api/v1/discovery/users/

And here is the script I used:

import requests
import pandas as pd

# 🔐 Token and domain info
access_token = "....."
mcas_domain = "<......>"

headers = {
'Authorization': f'Token {access_token}',
'Content-Type': 'application/json'
}

# 🔄 API URL: discovered_apps
api_url = f"https://{mcas_domain}.portal.cloudappsecurity.com/api/v1/discovery/discovered_apps/"
all_data = []

while api_url:
resp = requests.get(api_url, headers=headers)
print("Status code:", resp.status_code)

if resp.status_code != 200:
print("Error:", resp.text)
break

data = resp.json()
print("Number of records on page:", len(data.get("data", [])))
all_data.extend(data.get("data", []))
api_url = data.get("nextLink")

df = pd.json_normalize(all_data)

# 💾 Save to CSV
df.to_csv("discovered_apps.csv", index=False)
print("✅ discovered_apps.csv file created.")

[v-achippa · Accepted solution]

Hi @gulce,

Thank you for reaching out to Microsoft Fabric Community.

Thank you @johnbasha33 for the prompt response.

The API is not returning all the apps or users you see in the defender because,

• The API call without a streamId only returns data for one stream, that is why there are only few records. And full pagination is not implemented, here the API returns results in pages so by default it may be showing only the first page records. 

Please follow below steps:

• Get all stream IDs from /api/discovery/streams/
• For each streamId, call /api/v1/discovery/discovered_apps with that streamId in the request body.
• Implement pagination by checking hasNext and use the nextQueryFilters until no more pages remain.
• Combine results from all the streams to match the portal’s total count.

This will return all records from all reports, matching what you see in the portal.

Thanks and regards,

Anjan Kumar Chippa

[gulce]

hi @v-achippa

Thanks. I currently call /api/v1/discovery/discovered_apps to collect the appIds, then use those appIds together with a single streamId to call /api/v1/discovery/app_users for each appId. However, I only ever get a maximum of ~101 rows.

Based on your guidance, here’s the plan I will follow:

• GET https://{your-domain}.portal.cloudappsecurity.com/api/discovery/streams/
  Header: Authorization: <your_token>
  → First, I’ll fetch all streamIds from here.
• For each streamId, I’ll call /api/v1/discovery/discovered_apps.
• I’ll implement pagination by reading hasNext and nextQueryFilters from the response, and for the next request I’ll send them back in the body under queryFilters (repeat until the last page).
• After that, for each appId I’ll call /api/v1/discovery/app_users/ using the same pagination logic, iterating across all streams.

I’d like to confirm two details:

1. When moving to the next page, should the request body field indeed be queryFilters (i.e., pass the response’s nextQueryFilters back as raw JSON, without quotes)?
2. Is the per-page limit of ~100/101 a hard cap, or can it be increased? (We consistently see 101.)

Thanks again for the guidance

[v-achippa]

Hi @gulce,

Yes, you should pass the nextQueryFilters object back in the request body under filters exactly as JSON, do not wrap it in the quotes.
And the 100/101 rows per page is a fixed limit and it cannot be increased, so pagination is required to get all results.

Thanks and regards,

Anjan Kumar Chippa

[gulce]

Hi @v-achippa 

Testing Cloud Discovery – app_users API with Postman

Endpoint

POST https://<TENANT>.portal.cloudappsecurity.com/api/v1/discovery/app_users/

Headers

Authorization: Token <ACCESS_TOKEN> Content-Type: application/json

Request body I used

Works either flat or wrapped in filters. The wrapped form is the “official” one.

1. A) Wrapped in filters (recommended)

{"filters": {"streamId": "......................","appId": 11394,"timeframe": 90},"sortField": "trafficTotalBytes","sortDirection": "desc" }

1. B) Flat (also worked for me)

{"streamId": "......................","appId": 11394,"timeframe": 90,"sortField": "trafficTotalBytes","sortDirection": "desc" }

Response (summary)

{"total": 101,"hasNext": false,"data": [ ...users... ] }

• With these filters I get 101 users and hasNext: false.
• Sorting by trafficTotalBytes desc works and mirrors the Top 100 users view in the portal (https://security.microsoft.com/cloudapps/discover).
• The portal caps the visual list at 100 for that view, but the API returns all matches for the filter/page. In my case, there’s no next page, so hasNext is false.

Have you personally managed to fetch page 2+ from POST /api/v1/discovery/app_users? When the response returns hasNext: true with a nextQueryFilters object, do you paste that object verbatim inside filters for the next request (i.e., { "filters": <nextQueryFilters> }), rather than sending a top-level nextQueryFilters (which throws “Invalid filter params”)? If yes, could you share a minimal working body (redact IDs) and whether nextQueryFilters contained skip/page or a different token? Also, is the page size fixed (~101) or can it be increased?

[v-achippa]

Hi @gulce,

Yes when hasNext: true is returned, you need to take the nextQueryFilters object from the response and paste it in the verbatim inside filters for the next request, like for example
{
"filters": { ...nextQueryFilters... },
"streamId": "<streamId>",
"appId": <appId>,
"limit": 100
}
Do not send the nextQueryFilters as a top level field because that will fail.
And the page size is fixed at 100/101, it cannot be increased. The only way to get all results is to keep looping with hasNext or nextQueryFilters until there are no more pages.

Thanks and regards,

Anjan Kumar Chippa

[gulce]

Hi  @v-achippa 

Thanks a lot for the detailed pointers.

We tried both pagination patterns you suggested on /api/v1/discovery/app_users, but we’re still stuck:

What we did

We have 2 streamIds. One returns “no data”. The other returns total: 101 and "hasNext": false on the very first call.

Request #1 (first page):

POST /api/v1/discovery/app_users

{

"filters": {},

"streamId": "<streamId>",

"appId": 11394,

"timeframe": 90,

"sortField": "trafficTotalBytes",

"sortDirection": "desc",

"limit": 100

}

Response (excerpt):

 {

"results": [ ... 101 items ... ],

"total": 101,

"hasNext": false

}

So there’s no nextQueryFilters returned, and hasNext is already false on page 1.

What we tried next

1.Putting only the nextQueryFilters keys inside filters (as recommended).

When we attempt a second call with:

{

"filters": { "skip": 101, "page": 2 },

"streamId": "<streamId>",

"appId": 11394,

"timeframe": 90,

"sortField": "trafficTotalBytes",

"sortDirection": "desc",

"limit": 100

}

…we get:

"detail": "Invalid filter params were sent to API: {'skip': 101, 'page': 2}"

2.Nesting nextQueryFilters inside filters (alternative shape):

{

"filters": {

"nextQueryFilters": { "skip": 101, "page": 2 }

},

"streamId": "<streamId>",

"appId": 11394,

"timeframe": 90,

"sortField": "trafficTotalBytes",

"sortDirection": "desc",

"limit": 100

}

This also fails with a similar “Invalid filter params” error.

 Because of the above, the REST path isn’t letting us advance beyond the first 101 users.

 Questions on REST

 Could you confirm the exact expected JSON shape for filters in app_users pagination?

– Should filters contain only the returned nextQueryFilters keys (e.g., skip, page), without any other fields?

– Or should we nest them as { "filters": { "nextQueryFilters": { ... } } }?

– Is there a chance page isn’t supported in our tenant and we must send only skip?

Given our first-page response already has "hasNext": false, is it expected that some apps/streams truly have only 101 users? (If so, that would explain things; just want to double-check.)

Graph API path (works) & follow-ups

In Graph Explorer we can list users for the app via (example):

GET /beta/security/dataDiscovery/cloudAppDiscovery/uploadedStreams/{streamId}/aggregatedAppsDetails(period=duration'P90D')/{appId}/users?$skip=300

This returns the next page of users correctly.

What we still need is guidance on other fields beyond the user list:

Which Graph endpoints/properties expose the aggregated metrics we see in MCAS (e.g., trafficTotalBytes, trafficUploadedBytes, lastSeen, etc.) per app and/or per user?

– Is it via the {appId} resource of aggregatedAppsDetails(...) with $select?

– Is there a way to get per-user traffic totals via Graph, or only app-level aggregates?

Any recommendations on pagination best practices here: should we prefer $top + @odata.nextLink over $skip, or is $skip the intended approach for these endpoints?

Which permissions/scopes do you recommend for production (delegated/app)? (e.g., which exact Security/Cloud App Discovery scopes are required.)

Finally, is there a v1.0 equivalent for these endpoints, or is /beta currently the only available surface?

Thanks again for your help—any sample queries/snippets showing how to pull users + their traffic metrics for a given {streamId, appId, period} would be greatly appreciated.

[v-achippa]

Hi @gulce,

Thank you for the detailed response. Since you are consistently seeing hasNext:false at 101 rows even when more users exist, I recommend you to please raise a Microsoft support ticket with all the details, so the product team can review the backend logs for your tenant and confirm whether it is a limitation or a tenant specific issue.

To raise a support ticket, kindly follow the steps outlined in the following guide:

How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

Thanks and regards,

Anjan Kumar Chippa

[v-achippa]

Hi @gulce,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution I have provided for the issue worked? or let us know if you need any further assistance.

Thanks and regards,

Anjan Kumar Chippa

[v-achippa]

Hi @gulce,

We wanted to kindly follow up to check if the solution I have provided for the issue worked? or let us know if you need any further assistance.

Thanks and regards,

Anjan Kumar Chippa

[v-achippa]

Hi @gulce,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution I have provided for the issue worked? or let us know if you need any further assistance.

Thanks and regards,

Anjan Kumar Chippa