Push Dataset is empty when reached the 200K limit

[Anonymous]

Hi,

I'm using a Push Dataset which received data from Stream Analytics.

The Dataset is created with de default values (basicFIFO). But from the weekend I have some problems, it works well before this weekend.

Now, when the Dataset reach the 200K records limit (a bit more of that 200K), the Dataset became empty. All content is deleted, not only the oldest as FIFO had to do.

Is this the normal functioning?

Thanks

Raúl

[lbendlin]

yes, there are a number of events that can cause the dataset to flush out. If you need the history then you should use a hybrid streaming dataset instead.

[Anonymous]

How can I do it from Stream Analytics?

I thought this is the type created by default from Stream Analytics, a mix between push and streaming.

[v-yiruan-msft]

Hi @Anonymous ,

Please check if the solution in the following thread can help you.

What happens when you reach the size limit of a Push Dataset with Historical Analysis Enabled

Best Regards

Rena

[Anonymous]

Thanks @v-yiruan-msft,

I saw theis thread before ask this question. But I think is not my case.

The retention policy of my Dataset is basicFIFO, I have enabled the 'Historical data analysis', and each record has few information, something like name-value, then I never reach the size limit before reach the 200K records limit.

Last week the dataset is working as the documentation said, when the 200K records limit is reached, the oldest are deleted but keep information in the dataset.

But now, the same dataset, with the same configuracion, when reach the 200K limit, delete all data, let the dataset without any record. This is not what I read in the documentation for basicFIFO, and I cannot delete old records of the dataset from Stream Analytics because it is not allowed to write 'delete' in que query.

I think my dataset has the same configuration, but it seems that it does not work the same as the solution in the other thread.

My dataset receive information every minute from Stream Analytics, and I need to have the last hour of records (around 60K records and size less than 10MB). Deleting all records in the dataset, It's an hour with wrong informacion in the reports.

Regards

Raul