Forum Discussion
GetScanResult endpoint does not return "datasourceInstances" list
Hello!
I'm collecting company's PBI metadata via PBI API (Scan Jobs to be precise).
Following documentation, I call PostWorkspaceInfo endpoint with additional params
getArtifactUsers=True&lineage=True&datasourceDetails=True&datasetSchema=True&datasetExpressions=TrueSCAN_GETINFO_PARAMS = {
"getArtifactUsers": "true",
"lineage": "true",
"datasourceDetails": "true",
"datasetSchema": "true",
"datasetExpressions": "true",
}
PBI_SCOPE = "https://analysis.windows.net/powerbi/api/.default"
class PowerBIOAuth2ClientCredentialsAuth(AuthBase):
"""HttpHook auth adapter around dlt OAuth2 client credentials."""
def __init__(self, login: str, password: str) -> None:
tenant_id = "..."
client_id = "..."
if not password:
raise AirflowException("Power BI client secret is empty")
self._auth = OAuth2ClientCredentials(
access_token_url=f"https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token",
client_id=client_id,
client_secret=password,
access_token_request_data={"scope": PBI_SCOPE},
)
def __call__(self, request: PreparedRequest) -> PreparedRequest:
return self._auth(request)
def create_scan(
batch_index: int,
ch_conn_id: str,
select_ids_sql: str,
powerbi_conn_id: str,
batch_size: int = 100,
) -> str:
"""Create a Power BI scan for one mapped workspace batch.
Args:
batch_index: Batch number produced by dynamic task mapping.
ch_conn_id: Airflow ClickHouse connection id used to fetch workspace ids.
select_ids_sql: SQL query that returns workspace ids and supports
`limit` and `offset` parameters.
powerbi_conn_id: Airflow HTTP connection id for Power BI API access.
batch_size: Number of workspace ids requested from ClickHouse per batch.
Returns:
The created Power BI `scan_id`.
Raises:
AirflowException: If the batch is empty, API response is non-202, or the
response payload contains an API error or misses the `id` field.
"""
...
http = HttpHook(
method="POST",
http_conn_id=powerbi_conn_id,
auth_type=PowerBIOAuth2ClientCredentialsAuth,
)
response = http.run(
endpoint=f"v1.0/myorg/admin/workspaces/getInfo?{urlencode(SCAN_GETINFO_PARAMS)}",
data=json.dumps({"workspaces": chunk}),
headers={"Content-Type": "application/json", "Accept": "application/json"},
extra_options={"check_response": False},
)
body: dict[str, Any] = response.json()
if body.get("error"):
raise AirflowException(f"getInfo API error for batch {batch_index}: {body['error']}")
scan_id = body.get("id")
if not scan_id:
raise AirflowException(f"getInfo response missing `id`: {body}")
logging.info("Created scan %s for batch %s (%s workspaces)", scan_id, batch_index, len(chunk))
return scan_id POST request is on screenshot below (tried both 'true' and "True" - no changes)
INFO - Calling Power BI getInfo endpoint for batch 0: v1.0/myorg/admin/workspaces/getInfo?getArtifactUsers=True&lineage=True&datasourceDetails=True&datasetSchema=True&datasetExpressions=TrueIn repsonse for GET GetScanResult I expect to use "datasourceInstances" list data later instead of use Datasets GetDatasourcesAsAdmin to get exact the same info, but I didn't get this list. Only related data I have it's "datasourceInstanceId" in "datasourceUsage" list in "datasets" list.
"datasets": [
{
"id": "",
"name": "",
"tables": [],
"configuredBy": "",
"configuredById": "",
"isEffectiveIdentityRequired": false,
"isEffectiveIdentityRolesRequired": false,
"refreshSchedule": {
"days": [
"Sunday",
"Monday",
"Tuesday",
"Wednesday",
"Thursday",
"Friday",
"Saturday"
],
"times": [
"05:30"
],
"enabled": true,
"localTimeZoneId": "UTC",
"notifyOption": "MailOnFailure"
},
"targetStorageMode": "Abf",
"createdDate": "",
"contentProviderType": "",
"datasourceUsages": [
{
"datasourceInstanceId": "005f54d6-6e55-496c-93da-e2a96356b72a"
}
],
"tags": []
},
I've read that datasourceDetails=True might be an issue, but I pass this param into requst url.
Could you help me please figure out why 'datasourceInstances' list is not in the GetScanResult response?
Hi 1ng4lipt ,
Thank you for providing the detailed request flow. I have reviewed your implementation, and your PostWorkspaceInfo call appears to be correct. so the parameters are set appropriately. The use of true versus True will not affect the response in this context.
Notably, your scan result returns datasourceUsages with a valid datasourceInstanceId, indicating that the scan is successfully identifying the relationship between the dataset and its datasource. If there were an issue with the request, this information would typically not be present.
The absence of the datasourceInstances section generally occurs when the scan API cannot fully resolve the underlying datasource metadata. As outlined in the Microsoft documentation for GetScanResult, the API may only return certain properties depending on the metadata available in the Power BI service. As a result, the response can vary based on datasource type, connection configuration, gateway setup, or how the dataset was created.
This behavior is common with specific cloud connections, custom connectors, parameterized connections, dataflows, and certain Fabric related sources where the relationship is detected, but complete datasource details are not provided in the scan response.
Since your request is properly configured and the scan completes successfully, this appears to be related to the API’s metadata return for that datasource, rather than an issue with your code.
If you require comprehensive datasource connection details, I recommend using the Get Datasources As Admin endpoint, which is generally more reliable when datasourceInstances is missing.
For further reference, please see the following Microsoft documentation:
GetScanResult API documentation
https://learn.microsoft.com/rest/api/power-bi/admin/workspace-info-get-scan-resultPostWorkspaceInfo API documentation
https://learn.microsoft.com/rest/api/power-bi/admin/workspace-info-post-workspace-infoGet Datasources As Admin documentation
https://learn.microsoft.com/rest/api/power-bi/admin/datasets-get-datasources-as-admin
Thank you.
8 Replies
- v-tejramaCommunity Support
Hi 1ng4lipt ,
Thank you for providing the detailed request flow. I have reviewed your implementation, and your PostWorkspaceInfo call appears to be correct. so the parameters are set appropriately. The use of true versus True will not affect the response in this context.
Notably, your scan result returns datasourceUsages with a valid datasourceInstanceId, indicating that the scan is successfully identifying the relationship between the dataset and its datasource. If there were an issue with the request, this information would typically not be present.
The absence of the datasourceInstances section generally occurs when the scan API cannot fully resolve the underlying datasource metadata. As outlined in the Microsoft documentation for GetScanResult, the API may only return certain properties depending on the metadata available in the Power BI service. As a result, the response can vary based on datasource type, connection configuration, gateway setup, or how the dataset was created.
This behavior is common with specific cloud connections, custom connectors, parameterized connections, dataflows, and certain Fabric related sources where the relationship is detected, but complete datasource details are not provided in the scan response.
Since your request is properly configured and the scan completes successfully, this appears to be related to the API’s metadata return for that datasource, rather than an issue with your code.
If you require comprehensive datasource connection details, I recommend using the Get Datasources As Admin endpoint, which is generally more reliable when datasourceInstances is missing.
For further reference, please see the following Microsoft documentation:
GetScanResult API documentation
https://learn.microsoft.com/rest/api/power-bi/admin/workspace-info-get-scan-resultPostWorkspaceInfo API documentation
https://learn.microsoft.com/rest/api/power-bi/admin/workspace-info-post-workspace-infoGet Datasources As Admin documentation
https://learn.microsoft.com/rest/api/power-bi/admin/datasets-get-datasources-as-admin
Thank you.- 1ng4liptRegular Visitor
Hello v-tejrama ,
Thank you for this detailed explanation.
I want to add that dataRetrievalState column has valuesUpstreamLineageErrors; DatasetSchemaDisabledByAdmin; DatasetExpressionsDisabledByAdmin
and
DatasetSchemaDisabledByAdmin; DatasetExpressionsDisabledByAdmin
which is obviously because of the situation you mentioned.
Still, I want to ask does it make sense to go to PBI admin/app settings to check what metadata is allowed to collect or something like that (if it is possible at all. Unfortunately, I am not a PBI admin and have no clue what is inside PBI admin panel). If so, could you please suggest which settings should I ask to check.
- lbendlinSuper User
- 1ng4liptRegular Visitor
In case you, like me, use dlt to get response from PBI endpoints you can select datasourceInstances as selector to get datasource data.
{ "name": "get_scan_datasource_instances", "primary_key": "datasourceId", "write_disposition": "append", "max_table_nesting": 0, "endpoint": { "path": "v1.0/myorg/admin/workspaces/scanResult/{scan_id}", "params": {"scan_id": scan_id}, "data_selector": "datasourceInstances", "paginator": {"type": "single_page"}, }, },response
"value": [ { "datasourceType": "AnalysisServices", "connectionDetails": { "server": "My-As-Server", "database": "My-As-Database" }, "datasourceId": "33cc5222-3fb9-44f7-b19d-ffbff18aaaf5", "gatewayId": "0a2dafe6-0e93-4120-8d2c-fae123c111b1" }- v-tejramaCommunity Support
Hi 1ng4lipt ,
Thank you for sharing the solution and updating the thread.
This is a significant insight for others working with dlt and the Power BI Scan APIs.
Although the datasourceInstances data is available in the Scan Result response, it must be explicitly requested using the datasourceInstances data selector. With this setting enabled, the response provides the expected datasource metadata, including datasourceType, connectionDetails, datasourceId, and gatewayId.Thank you.