Understanding the EventProcessedUtcTime and EventEnqueuedUtcTime columns to add the destination

[pmscorca]

Hi,

when for an Evenstream I use an Azure Event Hubs as a source and a KQL database as a destination, I can add the EventProcessedUtcTime and EventEnqueuedUtcTime columns to the KQL table.

I'd like to understand better the usefulness of these two data.

Any suggests to me, please? Thanks

[Anonymous]

Hi @pmscorca ,

1. EventProcessedUtcTime:
This column records the exact time when the event was processed by your system. It is useful for monitoring the processing latency and ensuring that your event processing pipeline is functioning as expected. By comparing this timestamp with the EventEnqueuedUtcTime, you can measure the time it takes for an event to be processed after it has been received.

2. EventEnqueuedUtcTime:
This column captures the time when the event was received by Azure Event Hubs. It is crucial for tracking the arrival time of events and can help in identifying any delays or bottlenecks in the event ingestion process. This timestamp is particularly useful for time-based analysis and for ensuring that events are processed in the correct order.

For more details, please refer:

Stream data as input into Azure Stream Analytics - Azure Stream Analytics | Microsoft Learn

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly. 

[pmscorca]

Hi, thanks for your reply.

I understand that EventProcessedUtcTime represents the Eventstream processing time of an event entered from Azure Event Hubs: it is ok for me.

I have a doubt about the meaning of EventEnqueuedUtcTime: is it the time when the event enters in Azure Event Hubs or is it the time when the event enters in Eventstream from Azure Event Hubs?

Both these two times could be very very close to each other, but however they are different between them.

Thanks

[pmscorca]

Hi,

ok, but is EventEnqueuedUtcTime registered inside Azure Event Hubs?

Having this timestamp in Eventstream it seems having a redundant information, isn't it?

[Anonymous · Accepted solution]

Hi @pmscorca ,

Yes, the EventEnqueuedUtcTime is indeed registered inside Azure Event Hubs. This timestamp is set by Azure Event Hubs when the event is enqueued in the Event Hub partition.

While it might seem redundant to have this information in Eventstream, it serves a specific purpose. By including the `EventEnqueuedUtcTime` in your Eventstream data, you can directly compare it with the `EventProcessedUtcTime` to measure the end-to-end latency of your event processing pipeline. This comparison helps you identify and troubleshoot any delays or bottlenecks between the time an event is received by Azure Event Hubs and the time it is processed by your application.

Understand time handling in Azure Stream Analytics - Azure Stream Analytics | Microsoft Learn

EventData.EnqueuedTime Property (Azure.Messaging.EventHubs) - Azure for .NET Developers | Microsoft ...

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly. 

[pmscorca]

Hi, thanks.

This is an important phrase: the EventEnqueuedUtcTime timestamp is registered in Azure Event Hubs and reported in Eventstream, maintaning the same value.

Having this timestamp in Eventstream can be convenient in order to avoid to check the relative value in Azure Event Hubs, that is out of Eventstream.

[Anonymous]

Hi @pmscorca ,

This timestamp represents the exact time when the event is received by Azure Event Hubs, not when it enters Eventstream from Azure Event Hubs.

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly.