Get error messages from pipeline activities

[Mnnleus]

Hello,

I retrieve information on failures (name and type of activity that failed) from Logs Analytics, in the "SynapseIntegrationActivityRuns" table. I use Logs Analytics, because I have a lot of failures due to connectivity problems (SQL, PySpark, ...). In this case, my monitoring tables fed into the pipelines are not updated.

I would like to classify my errors (data error, connectivity error, other), but I can't find a way to retrieve the error details (error message) for each activity. These details exist when Irun the pipeline, since they are visible, but I don't know how to "capture" them.

I saw in a blog response that it was possible to customize the logs. In that case, how do I go about it? Or do you have another method?

Thank you in advance.
Have a nice day,
Mnnleus

[NandanHegde]

you can do a custom designing wherein map all the activity to a common set variable activity and get the error messages in case if there are any failures.

Below blog would help with brief understanding of the same:

https://datasharkx.wordpress.com/2021/08/19/error-logging-and-the-art-of-avoiding-redundant-activiti...

[Mnnleus]

Hello,
Thank you for taking the time to answer.
My problem is that I can't put these "capture" errors in a table if the failure is due to a connectivity problem (SQL or PySpark), because that would require using a SQL script or a notebook. That's why I wanted to use the logs.

[v-cboorla-msft]

Hi @Mnnleus 

Apologies for the inconvenience and delay in response.

Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thank you.

[v-cboorla-msft]

Hi @Mnnleus 

We haven’t heard from you on the last response and was just checking back to see if you've had a chance to submit a support ticket. If you have, a reference to the ticket number would be greatly appreciated. This will allow us to track the progress of your request and ensure you receive the most efficient support possible.

Thank you.

[Mnnleus]

Hello,

At the moment I'm trying to find a solution on my own. If I can't, I'll use a ticket.

Have a nice day.
Thanks,
Mnnleus