topic Re: Randomly receiving 503, server unavailable, when retrieving activities? in Developer

Randomly receiving 503, server unavailable, when retrieving activities?

Dragon496 — Mon, 23 Jan 2023 15:38:36 GMT

We have some power shell scripts that retrieves details regarding our Power BI tenant.

The actual lines that retrieves the activity details are:-
if (!$result.continuationUri)
{
$result = Invoke-PowerBIRestMethod -Url $activityAPIUrl -method Get | ConvertFrom-Json
}
else {
$result = Invoke-PowerBIRestMethod -Url $result.continuationUri -method Get | ConvertFrom-Json
}

if ($result.activityEventEntities)
{
$audits += @($result.activityEventEntities)
}

These scripts were downloaded from Internet that have been amended for our own purposes. When I was developing the changes, I quite often received errors about the server being unavailable. I put this down to run the scripts, too much while developing them.

I deployed as a schedule task, last Tuesday. Everything was fine until this morning when I received the usual error (even though it would have only been trying to retrieve the activities in the previous 24 hours). The error is shown below, can anybody advise?

Thanks

Error on '.\Fetch - Activity.ps1' - System.AggregateException: One or more errors occurred. --->
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The remote server returned an error: (503)
Server Unavailable.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.<InvokeRestMethod>d__35.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.ExecuteCmdlet()
at Microsoft.PowerBI.Commands.Common.PowerBICmdlet.ProcessRecord()
---> (Inner Exception #0) System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The remote server
returned an error: (503) Server Unavailable.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.<InvokeRestMethod>d__35.MoveNext()<---
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Fetch - Run.ps1

Re: Randomly receiving 503, server unavailable, when retrieving activities?

Anonymous — Fri, 27 Jan 2023 02:33:27 GMT

Hi @Dragon496,

What type of authorizations methods are you used with rest API? OAuth, token or service principal?

If you mean the service principal mode, it requires you to re-invoke the authorization step with service principal every time before the API usages when you work with powershell scripts.

Regards,

Xiaxin Sheng

Re: Randomly receiving 503, server unavailable, when retrieving activities?

Dragon496 — Fri, 27 Jan 2023 08:42:49 GMT

As far as I can see, the scripts are using the oAuth method.

It appears to fail at random. For example, the scripts are executed by a scheduled task in the morning and it has failed twice this week and succeeded twice this week. One of the failures, I had to rerun once and the other needed five attempts.

Please note, the issue is not the schedule task. Random failures are generated, when I run the script, within Windows PowerShell ISE.

Thanks

Re: Randomly receiving 503, server unavailable, when retrieving activities?

Anonymous — Mon, 30 Jan 2023 08:17:46 GMT

HI @Dragon496,

Perhaps you can take a look at the rest API limitation if your scenario meets to these limitations, the remote server will reject or not respond with these requests which exceed to the limitations.

Power BI REST APIs push datasets limitations - Power BI | Microsoft Learn
Regards,

Xiaoxin Sheng

Re: Randomly receiving 503, server unavailable, when retrieving activities?

Dragon496 — Wed, 01 Feb 2023 09:03:43 GMT

Hello

I do no believe the problem is, exceeding the limitations. Take yesterday as an example… The job that runs the power shell scripts, failed at 6 AM. I then had to run the job, a couple of hours later, 7 times before the job ran successfully.

It appears to be failing on retrieving activities and I should point out that only last 24 hours of activities are being executed.

It is using the service principal to login
$pbiAccount = Connect-PowerBIServiceAccount -ServicePrincipal -Tenant $config.ServicePrincipal.TenantId -Credential $credential -Environment $config.ServicePrincipal.Environment

Part of the code that retrieves the activities is
$activityAPIUrl = "admin/activityevents?startDateTime='$($pivotDate.ToString("s"))'&endDateTime='$($pivotDate.AddHours(24).AddSeconds(-1).ToString("s"))'"

$audits = @()
$pageIndex = 1
$flagNoActivity = $true

do
{
if (!$result.continuationUri)
{
$result = Invoke-PowerBIRestMethod -Url $activityAPIUrl -method Get | ConvertFrom-Json
}
else {
$result = Invoke-PowerBIRestMethod -Url $result.continuationUri -method Get | ConvertFrom-Json
}

surely if it was a limitation issue, it would not work after seven attempts (or two attempts this morning).

An example of the error I am getting is below

31/01/2023 16:21:46
3
Next execute .\Fetch - Activity.ps1
next activity call
next activity continuation call
Script error on '.\Fetch - Activity.ps1' - System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
   at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.<InvokeRestMethod>d__35.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.ExecuteCmdlet()
   at Microsoft.PowerBI.Commands.Common.PowerBICmdlet.ProcessRecord()
---> (Inner Exception #0) System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
   at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.<InvokeRestMethod>d__35.MoveNext()<---

3
Next execute .\Fetch - Catalog.ps1
successfully executed .\Fetch - Catalog.ps1
3
Next execute .\Fetch - Graph.ps1
successfully executed .\Fetch - Graph.ps1

Please note, the numbers and comments simply allowed me to see how far the code is reaching.

Any suggestions would be greatly appreciated.

Thanks