Forum Discussion
Helper IOn-premises Power BI Enterprise Data Gateway logs - missing (expected) log fields
Super UserMonthly?!?! What's the point?
Almost Realtime ? Dream on. As I said the logs are flushed to disk infrequently.
What we really need is the ability to push log events into streaming datasets - as they are happening.
Fun story: We have a situation where we get DOZENS of new QueryStart log files EVERY DAY due to the unintended consequences of the inclusion of the query text in the log. In itself that's a good thing but we have an app that fires a ginormous SQl query at a Direct Query source every 10 seconds on average. You would look at 60GB of log data for a monthly report.
Helper IMonthly: It works for us (using this data helps us understand and forecast resource usage), i.e. having the ability to do things like, understand when we will need to add more gateways to the cluster.
Almost Realtime: Support can isolate errors within an hour window (that's our hope). Which helps us (obviously) with errors, but data egress from Azure (and failures along that scenario) is not free. Failures are costly, and worth the effort to make customers happy and save the company $. Often, the things that fail are aspects we can clearly identify from the logs, like we see a notably high percentage pattern of failures when the mashup/etc. is not completed within 1.5 minutes.
Last I looked (we do monthlies tonight/tomorrow), we were around 430 GB.
Streaming sounds great, when the gateway team offers us something with the ability to connect directly with our Synapse Warehouse. (But overall, we lean on the cheap/conservative side, trying to keep much of our "typical" reporting on local PBIRS servers. It gets expensive and difficult if everything is in Azure (since if we can't monitor the gateways, and what's going on, we can't manage it 😉
- lbendlin
Almost Realtime: Support can isolate errors within an hour window (that's our hope).As I said, this does not match our experience. Daily at best with the current flushing mechanism (unless you resort to brute force service restarts which is not a good idea).
- rkaratzas
grabbing a copy of those items (what's shown in the PowerShell) works just fine. (remember, it's working with the file system where the logs are stored, so you are just dealing with it in a read-only context, so grab what's changed within your modified window, load it into the dashboard mentioned and it works - or at least it has for the last 3 weeks for us 🙂 (We are still ramping up on all of the technical details in the logs AND I've registered a support call with Microsoft to get further clarity on the ActivityType codes, so we can translate this into laymen terms for support.
- lbendlin
grabbing a copy of those items (what's shown in the PowerShell) works just fineThat is only half of the story. A last modified timestamp on a file does not reveal the actual last row of data written into that file. Again - our experience is that the gateway service only flushes data to the files occasionally, every couple hours at best and every 24 hours at worst.
- rkaratzas
Help me understand, log files are just that (simple file system things that work like anything else)?
You can pull (a copy of) those "things" at will, so you just look at the property (of the file) that they were last modified. (easy peasey, but I'd really like to understand what it is that you are experiencing that requires any restarting or flushing). - Rob