Forum Discussion
On-premises Power BI Enterprise Data Gateway logs - missing (expected) log fields
Hi Folks
We are running several gatways (version 3000.186.14, August 2023, clustered too) and are just noticing that both the DatasetId and WorkspaceID are mostly (99.99% of the time) missing?
We are doing log shipping (and almost real-time monitoring), and tracking this down to support actionable code/things is difficult without this information. (since we run at least 4 different application environments, and multiple gateways).
Are we missing anything as far as gateway setup goes, to ensure we have this information?
Thanks 🙂
15 Replies
- TomMartens
Super User
Hey rkaratzas ,
in addition to what lbendlin already mentioned, maybe updating you gateways to the October 2023 release will help, according to this article artifact details are already part of the basic logging: https://powerbi.microsoft.com/en-us/blog/on-premises-data-gateway-october-2023-release/
Hopefully, this helps to tackle your challenge.
Regards,
Tom
- lbendlin
Super User
Those "EvaluationContext" details are only available for a subset of data sources. What are the main data sources processed on your gateways?
We see the dataset details for 370000 out of 560000 log events.
- rkaratzas
Helper I
ok, so having those two IDs is not reliable I gather.
Maybe the next question I would have is, since we are trying to make log errors an actionable thing, is there a way to trace anything back to the running "application" (application being dashboards, apps, reports, automations, etc.)?
We've contemplated writing a utility which can go out into the "Azure landscape" (various workspaces associated with different environments) and finding where the query portion (or anything else identifiable of the error) exists. (A very complicated process for searching, needing crawls, unpacking the objects, etc.)
- lbendlin
Super User
we are trying to make log errors an actionable thingOh, good. Let me know when you find something. I have been at it for a couple months and have yet to find a meaningful metric. For example
looks really pretty but is also meaningless since none of these reported issues seem to affect actual refreshes, or indicate any real gateway issues.
We had more luck with telemetry that we collected ourselves. Sadly the SystemCounter reports are not usable as they are only flushed to disk very occasionally.
The EvaluationContext data is still your best bet. Raise a ticket with Microsoft to figure out why it doesn't populate for you (make sure to use a recent-ish gateway version, post June 2023).
- rkaratzas
Helper I
yes, so it's pretty easy:
1. We grabbed a copy of this (and have been using it as one of our monthly gateway reports https://github.com/RuiRomano/pbigtwmonitor (we also use the vanilla one developed by the gateway team)
2. I wrote a PowerShell to log ship every X minutes (via task scheduler on EACH gateway server):
Set-ExecutionPolicy -Scope LocalMachine -ExecutionPolicy Unrestricted -Force
# Set the UNC path to the directory
$uncPath = "\\yourserver\c$\Users\PBIEgwService\AppData\Local\Microsoft\On-premises data gateway\*.log"# Set the UNC path to the log destination directory
$destinationPath = "\\NASS1\UW_Workstation\PowerBIEnterpriseDataGatewayLogs24Hour\yourserver"# Calculate the cutoff date and time for files modified in the last 24 hours
$now = Get-Date
$cutoff = $now.AddHours(-24)# List files modified in the last 24 hours
$filesToCopy = Get-ChildItem -Path $uncPath -Recurse | Where-Object { $_.LastWriteTime -ge $cutoff }# Copy the selected files to the destination
$filesToCopy | ForEach-Object { Copy-Item -Path $_.FullName -Destination $destinationPath }# Delete files older than the cutoff date in the destination directory
Get-ChildItem -Path $destinationPath | Where-Object { $_.LastWriteTime -lt $cutoff } | Remove-Item -Force3. Register a Certified Dataset in Azure
4. PowerShell to refresh the dataset @ X minutes (similar to this):
https://stackoverflow.com/questions/52697381/power-bi-dataset-refresh-using-powershell
(note that clientid is really applicationid in the Azure portal)
understand?
PS Note that I already submitted a feature request, such that log shipping functionality becomes part of the Gateway interface (and it being configurable via what's already there in the config files).