GraceChang's avatar
GraceChang
Microsoft Employee
4 years ago
Status:
New

Kusto Dataset Refresh using Service Principal

We have deployed Kusto dataset using a service principal in Azure DevOps. The deployment works. But when we want to refresh the dataset (manually or with powershell Rest API), we always receive the following error message:


"Data source error: It looks like scheduled refresh failed because at least one data source is missing credentials. To start the refresh again, go to this dataset's settings page and enter credentials for all data sources."


We also searched all possible workarounds to this issue. I will provide the complete list we tried at the bottom of my request. None of them works for Kusto data source. To name the most referred one:


  1. Referred to a solution described in below article from Power BI Community back in 2020. The source of the dataset is a Azure SQL database using a SQL Server login. According to the solution it is working for SQL Dataset via using a REST API call:


Solved: Re: Dataset refresh error Service principal - Microsoft Power BI Community


Invoke-PowerBIRestMethod -Url "gateways/$($BounGateway.value.gatewayId)/datasources/$($BounGateway.value.id)" -Method PATCH -Body $UpdateUserCredential | ConvertFrom-Json    


However, the above REST API call does not work for Kusto Dataset. Below is the error from fiddler trace:

 {"error":{"code":"DM_GWPipeline_Gateway_MashupDataAccessError","pbi.error":{"code":"DM_GWPipeline_Gateway_MashupDataAccessError","parameters":{},"details":[{"code":"DM_ErrorDetailNameCode_UnderlyingErrorCode","detail":{"type":1,"value":"-2147467259"}},{"code":"DM_ErrorDetailNameCode_UnderlyingErrorMessage","detail":{"type":1,"value":"The data source kind doesn't support the credential type. Data source kind: Kusto. Credential type: UsernamePassword."}},{"code":"DM_ErrorDetailNameCode_UnderlyingHResult","detail":{"type":1,"value":"-2147467259"}}],"exceptionCulprit":1}}}


Business Justification: Automation, Security


We would like to use Azure Ops to automate report deployment and refresh, instead of using individual account. This is a good practice as someone may leave his job next day and report will be broken. Having one Service Principal also enables us to monitor and alert if there is any report refresh failure, so we can keep our SLA to our customers.


Resource: Here are the list of solutions that I tried but none of them work for Kusto data source. What's bizarre is they seem working for SQL data source, I assume this is a bug of how Power BI Service handles Kusto data source connection. There is even an ICM closed on this topic as non-supported, see at the last one of below:

  1. Refresh your data in Power BI WITHOUT a user??? - YouTube --> HIGHLY RECOMMENDED
  2. Refresh Power BI Datasets with PowerShell – Insight Quest (insightsquest.com)
  3. Announcing Data Refresh APIs in the Power BI Service | Microsoft Power BI-bloggen | Microsoft Power BI
  4. powerbi-powershell/manageRefresh.ps1 at master · Azure-Samples/powerbi-powershell · GitHub
  5. Incident-267938103 Details - IcM (microsofticm.com) --> A KNOWN BUG

Recent ideas