Forum Discussion
watermark delay in data insights of eventstream's data source
Hello, everyone.
When I looked at the "data insights" of the "data source", I found that the "watermark delay" is always around 10 seconds.
Does this mean that the data is being loaded with a delay to the data destination?
Even with the powerBI screen auto-refresh set to 1 second, data is not being loaded every second.
I have set IoT Hub (standard, partition=16) as the data source and KQL data base as the data destination.
What is the best way to eliminate or reduce the watermark delay?
thank you.
Hi bxstnk ,
The 1-second auto-refresh you set means that the Real-Time dashboard checks every second to see if the data displayed by the tiles needs to be updated, but there is a small delay in loading the data from the data source into the Eventstream and from the Eventstream into the Destination you set, and there is a small delay in loading the data in the Transformation you set in Evenstream can also cause delays. The length of the delay depends on the size of your data and the complexity of the transformation.
I conducted the following test myself. I connected to a data source without setting any transformation operation and set the destination directly, but it still has a watermark delay of 4 seconds:
So the latency between the data source, eventstream and destination is almost impossible to eliminate, all you can do is: simplify the data in the data source and simplify the complexity of the transformation steps in the eventstream.
Best Regards,
Dino Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
2 Replies
Hi bxstnk ,
The 1-second auto-refresh you set means that the Real-Time dashboard checks every second to see if the data displayed by the tiles needs to be updated, but there is a small delay in loading the data from the data source into the Eventstream and from the Eventstream into the Destination you set, and there is a small delay in loading the data in the Transformation you set in Evenstream can also cause delays. The length of the delay depends on the size of your data and the complexity of the transformation.
I conducted the following test myself. I connected to a data source without setting any transformation operation and set the destination directly, but it still has a watermark delay of 4 seconds:
So the latency between the data source, eventstream and destination is almost impossible to eliminate, all you can do is: simplify the data in the data source and simplify the complexity of the transformation steps in the eventstream.
Best Regards,
Dino Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.thank you, Anonymous.
I accepted your reply as the solution.
