When dataflows run into errors and Microsoft Support is involved, the feedback is always the same. Resumed: "we need logs to do any investigation. We don't collect anything ourselves, so your only option is to use your browser developer tools to record a run".
This is frustrating and quite limited when considering that:
- Some issues are intermittent and only/mostly happening on scheduled dataflows (sometimes in the middle of the night)
- Some issues are related to internal connections not transiting via the browser (e.g. connection issues between a dataflow and the underlying staging lakehouse) which will never be captured via browser logs.
Please enable some background logging for dataflows to enable better support for issues here.
dm-p
on:
Allow standard Focus Mode for custom visuals that ...
on:
Data Pipelines - Run only selected activities
on:
More granular control and better design experience...
Would you mind letting us know more a bit what kind of background logging you'd like to see?
We do have some telemetry that we use to help troubleshoot scenarios, but there are scenarios in which we simply do not store detailed information as it doesn't run under our platform. The primary example of such cases are evaluations that happen through the Gateway and this is the reason why advanced logging is a feature that you can enable at the Gateway level to capture them and go deeper into the investigation, but the logging only happens at the gateway and such information is not sent or captured by the Fabric platform.