Fabric Engineering - Dataflow Gen2 - Enhancement - add diagnostic + error tracing

[JohnMiner3]

The image shown below represents a meta data driven design which call dataflow gen 2 (ci/cd).

The 040 pipeline (schedule) contains multiple calls to the 035 pipelines (execute job steps).  The 025 data flow has failed.  There is no error message related to the dataflow refresh.

We are lucky that a data pipeline is calling the dataset refresh.  We can see the error code of 20302 is a catch all error.  Did the dataflow get termined due to the gateway timing out or did the fabric capacity throttle the job?

Errors like this are very hard to trace without diagnostics.  The above image shows all the steps in this dataflow.  Would love to see which step the job failed at, how many records were processed before the error, and a better error message.

Without additional details, tickets have to be created for MSFT support to help trace down the issue.

https://learn.microsoft.com/en-us/power-bi/fundamentals/desktop-diagnostics

The above link shows how diagnostics can be turned on in PBI to get trace of the mashup language execution.  Something like this feature would be invaluable or at least give a better error message in the monitor window.  None currently is returned.  Thus calling a dataflow for a data pipeline is required to get the error code.  Right now, when a dataflow fails, one can refresh and hope for the best.