topic Re: Get error messages from pipeline activities in Data Engineering

Get error messages from pipeline activities

Mnnleus — Thu, 13 Jun 2024 13:44:56 GMT

Hello,

I retrieve information on failures (name and type of activity that failed) from Logs Analytics, in the "SynapseIntegrationActivityRuns" table. I use Logs Analytics, because I have a lot of failures due to connectivity problems (SQL, PySpark, ...). In this case, my monitoring tables fed into the pipelines are not updated.

I would like to classify my errors (data error, connectivity error, other), but I can't find a way to retrieve the error details (error message) for each activity. These details exist when Irun the pipeline, since they are visible, but I don't know how to "capture" them.

I saw in a blog response that it was possible to customize the logs. In that case, how do I go about it? Or do you have another method?

Thank you in advance.
Have a nice day,
Mnnleus

Re: Get error messages from pipeline activities

NandanHegde — Fri, 14 Jun 2024 04:18:11 GMT

you can do a custom designing wherein map all the activity to a common set variable activity and get the error messages in case if there are any failures.

Below blog would help with brief understanding of the same:

https://datasharkx.wordpress.com/2021/08/19/error-logging-and-the-art-of-avoiding-redundant-activities-in-azure-data-factory/

Re: Get error messages from pipeline activities

Mnnleus — Fri, 14 Jun 2024 12:45:21 GMT

Hello,
Thank you for taking the time to answer.
My problem is that I can't put these "capture" errors in a table if the failure is due to a connectivity problem (SQL or PySpark), because that would require using a SQL script or a notebook. That's why I wanted to use the logs.

Re: Get error messages from pipeline activities

v-cboorla-msft — Thu, 27 Jun 2024 12:22:47 GMT

Hi @Mnnleus

Apologies for the inconvenience and delay in response.

Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thank you.

Re: Get error messages from pipeline activities

v-cboorla-msft — Mon, 01 Jul 2024 05:22:12 GMT

Hi @Mnnleus

We haven’t heard from you on the last response and was just checking back to see if you've had a chance to submit a support ticket. If you have, a reference to the ticket number would be greatly appreciated. This will allow us to track the progress of your request and ensure you receive the most efficient support possible.

Thank you.

Re: Get error messages from pipeline activities

Mnnleus — Mon, 08 Jul 2024 09:28:00 GMT

Hello,

At the moment I'm trying to find a solution on my own. If I can't, I'll use a ticket.

Have a nice day.
Thanks,
Mnnleus