topic Understanding the EventProcessedUtcTime and EventEnqueuedUtcTime columns to add the destination in Eventstream

Understanding the EventProcessedUtcTime and EventEnqueuedUtcTime columns to add the destination

pmscorca — Tue, 22 Oct 2024 13:02:45 GMT

Hi,

when for an Evenstream I use an Azure Event Hubs as a source and a KQL database as a destination, I can add the EventProcessedUtcTime and EventEnqueuedUtcTime columns to the KQL table.

I'd like to understand better the usefulness of these two data.

Any suggests to me, please? Thanks

Re: Understanding the EventProcessedUtcTime and EventEnqueuedUtcTime columns to add the destination

Anonymous — Wed, 23 Oct 2024 01:42:07 GMT

Hi pmscorca ,

1. EventProcessedUtcTime:
This column records the exact time when the event was processed by your system. It is useful for monitoring the processing latency and ensuring that your event processing pipeline is functioning as expected. By comparing this timestamp with the EventEnqueuedUtcTime, you can measure the time it takes for an event to be processed after it has been received.

2. EventEnqueuedUtcTime:
This column captures the time when the event was received by Azure Event Hubs. It is crucial for tracking the arrival time of events and can help in identifying any delays or bottlenecks in the event ingestion process. This timestamp is particularly useful for time-based analysis and for ensuring that events are processed in the correct order.

For more details, please refer:

Stream data as input into Azure Stream Analytics - Azure Stream Analytics | Microsoft Learn

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly.

Re: Understanding the EventProcessedUtcTime and EventEnqueuedUtcTime columns to add the destination

pmscorca — Wed, 23 Oct 2024 16:24:02 GMT

Hi, thanks for your reply.

I understand that EventProcessedUtcTime represents the Eventstream processing time of an event entered from Azure Event Hubs: it is ok for me.

I have a doubt about the meaning of EventEnqueuedUtcTime: is it the time when the event enters in Azure Event Hubs or is it the time when the event enters in Eventstream from Azure Event Hubs?

Both these two times could be very very close to each other, but however they are different between them.

Thanks

Re: Understanding the EventProcessedUtcTime and EventEnqueuedUtcTime columns to add the destination

Anonymous — Thu, 24 Oct 2024 06:16:25 GMT

Hi pmscorca ,

This timestamp represents the exact time when the event is received by Azure Event Hubs, not when it enters Eventstream from Azure Event Hubs.

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly.

Re: Understanding the EventProcessedUtcTime and EventEnqueuedUtcTime columns to add the destination

pmscorca — Thu, 24 Oct 2024 07:18:17 GMT

Hi,

ok, but is EventEnqueuedUtcTime registered inside Azure Event Hubs?

Having this timestamp in Eventstream it seems having a redundant information, isn't it?

Re: Understanding the EventProcessedUtcTime and EventEnqueuedUtcTime columns to add the destination

Anonymous — Fri, 25 Oct 2024 01:46:27 GMT

Hi pmscorca ,

Yes, the EventEnqueuedUtcTime is indeed registered inside Azure Event Hubs. This timestamp is set by Azure Event Hubs when the event is enqueued in the Event Hub partition.

While it might seem redundant to have this information in Eventstream, it serves a specific purpose. By including the `EventEnqueuedUtcTime` in your Eventstream data, you can directly compare it with the `EventProcessedUtcTime` to measure the end-to-end latency of your event processing pipeline. This comparison helps you identify and troubleshoot any delays or bottlenecks between the time an event is received by Azure Event Hubs and the time it is processed by your application.

Understand time handling in Azure Stream Analytics - Azure Stream Analytics | Microsoft Learn

EventData.EnqueuedTime Property (Azure.Messaging.EventHubs) - Azure for .NET Developers | Microsoft Learn

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly.

Re: Understanding the EventProcessedUtcTime and EventEnqueuedUtcTime columns to add the destination

pmscorca — Fri, 25 Oct 2024 06:46:34 GMT

Hi, thanks.

This is an important phrase: the EventEnqueuedUtcTime timestamp is registered in Azure Event Hubs and reported in Eventstream, maintaning the same value.

Having this timestamp in Eventstream can be convenient in order to avoid to check the relative value in Azure Event Hubs, that is out of Eventstream.