Randomly receiving 503, server unavailable, when retrieving activities?

[Dragon496]

We have some power shell scripts that retrieves details regarding our Power BI tenant.

The actual lines that retrieves the activity details are:-
if (!$result.continuationUri)
{
$result = Invoke-PowerBIRestMethod -Url $activityAPIUrl -method Get | ConvertFrom-Json
}
else {
$result = Invoke-PowerBIRestMethod -Url $result.continuationUri -method Get | ConvertFrom-Json
}

if ($result.activityEventEntities)
{
$audits += @($result.activityEventEntities)
}

These scripts were downloaded from Internet that have been amended for our own purposes. When I was developing the changes, I quite often received errors about the server being unavailable. I put this down to run the scripts, too much while developing them.

I deployed as a schedule task, last Tuesday. Everything was fine until this morning when I received the usual error (even though it would have only been trying to retrieve the activities in the previous 24 hours). The error is shown below, can anybody advise?

Thanks

Error on '.\Fetch - Activity.ps1' - System.AggregateException: One or more errors occurred. --->
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The remote server returned an error: (503)
Server Unavailable.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.<InvokeRestMethod>d__35.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.ExecuteCmdlet()
at Microsoft.PowerBI.Commands.Common.PowerBICmdlet.ProcessRecord()
---> (Inner Exception #0) System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The remote server
returned an error: (503) Server Unavailable.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.<InvokeRestMethod>d__35.MoveNext()<---
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Fetch - Run.ps1

[v-shex-msft]

Hi @Dragon496,

What type of authorizations methods are you used with rest API? OAuth, token or service principal? 

If you mean the service principal mode, it requires you to re-invoke the authorization step with service principal every time before the API usages when you work with powershell scripts.

Regards,

Xiaxin Sheng

[Dragon496]

As far as I can see, the scripts are using the oAuth method.

It appears to fail at random. For example, the scripts are executed by a scheduled task in the morning and it has failed twice this week and succeeded twice this week. One of the failures, I had to rerun once and the other needed five attempts.

Please note, the issue is not the schedule task. Random failures are generated, when I run the script, within Windows PowerShell ISE.

Thanks

[v-shex-msft]

HI @Dragon496,

Perhaps you can take a look at the rest API limitation if your scenario meets to these limitations, the remote server will reject or not respond with these requests which exceed to the limitations.

Power BI REST APIs push datasets limitations - Power BI | Microsoft Learn
Regards,

Xiaoxin Sheng

[Dragon496]

Hello

I do no believe the problem is, exceeding the limitations. Take yesterday as an example… The job that runs the power shell scripts, failed at 6 AM. I then had to run the job, a couple of hours later, 7 times before the job ran successfully.

It appears to be failing on retrieving activities and I should point out that only last 24 hours of activities are being executed.

It is using the service principal to login
$pbiAccount = Connect-PowerBIServiceAccount -ServicePrincipal -Tenant $config.ServicePrincipal.TenantId -Credential $credential -Environment $config.ServicePrincipal.Environment

Part of the code that retrieves the activities is
$activityAPIUrl = "admin/activityevents?startDateTime='$($pivotDate.ToString("s"))'&endDateTime='$($pivotDate.AddHours(24).AddSeconds(-1).ToString("s"))'"

$audits = @()
$pageIndex = 1
$flagNoActivity = $true

do
{
if (!$result.continuationUri)
{
$result = Invoke-PowerBIRestMethod -Url $activityAPIUrl -method Get | ConvertFrom-Json
}
else {
$result = Invoke-PowerBIRestMethod -Url $result.continuationUri -method Get | ConvertFrom-Json
}

surely if it was a limitation issue, it would not work after seven attempts (or two attempts this morning).

An example of the error I am getting is below

31/01/2023 16:21:46
3
  Next execute .\Fetch - Activity.ps1
next activity call
 next activity continuation call
Script error on '.\Fetch - Activity.ps1' - System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
   at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.<InvokeRestMethod>d__35.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.ExecuteCmdlet()
   at Microsoft.PowerBI.Commands.Common.PowerBICmdlet.ProcessRecord()
---> (Inner Exception #0) System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
   at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.<InvokeRestMethod>d__35.MoveNext()<---

3
  Next execute .\Fetch - Catalog.ps1
  successfully executed .\Fetch - Catalog.ps1
3
  Next execute .\Fetch - Graph.ps1
  successfully executed .\Fetch - Graph.ps1

Please note, the numbers and comments simply allowed me to see how far the code is reaching.

Any suggestions would be greatly appreciated.

Thanks