Stream Analytics/Power BI Integration Clearing PushStreaming Dataset

[Anonymous]

We have been using Stream Analytics w/the Power BI Connector to stream real-time data to Power BI for a little over a year.  We are streaming Contact Center data and then running some DAX formulas against it to calculate different totals for the day; which we cannot calculate in Stream Analytics.

Since the last week of October when our dataset hits the row limit for the Streaming Dataset instead of purging old rows it is now clearing the dataset entirely.  It is doing this on both of our datasets.  We have since recreated 1 of the dataset connections and the issue is still occurring.  I have the same solution deployed for another customer and the datasets are not behaving like this.

Did anything change with PushDatasets and how the data retention policy is applied?

[ralmeida]

@Anonymous

I have the same problem with you, three weeks have passed and I have not found the solution.

When my streaming data set has 225-230k rows it goes back to 215-220k rows, deleting the first 10k rows I sent. I have two other streaming dataset with more than 2 million rows, and they work perfectly. 

I tried create a new test account to test with another streaming dataset on another account to eliminate the chances of reaching the pro account data limit, and I had the same problem.

Could it be a problem generated after the October update? Or does my dataset need some configuration that I don't know about?

[Anonymous]

 @ralmeida 

The behavior of the 250k back down to 215k is the default behavior for PushStreaming datasets when FIFO is the Default Retention Mode.

When SA creates a dataset it creates the Default Retention as FIFO which enforces the 200k limit.  If you create a PushStreaming dataset via the API it does not default to FIFO which allows you to push well over 250k but then you hit API data streaming restrictions and your real-time Dashboards will start to slow down.

[ralmeida]

@Anonymous  

I created a new dataset via API and now it works, thanks man!!!

But I don't understand why the other streaming data sets work and this one doesn't. All data sets were created using the same method.

[GilbertQ]

Hi there

Are you storing the historical data?

If so I do recall a limit of 5M rows

Here are more details: https://docs.microsoft.com/en-us/power-bi/connect-data/service-real-time-streaming#using-azure-strea...

[Anonymous]

Hi Gilbert,

We are storing historical data but the limit before data is purged is between 200k - 225k.  We did delete the dataset again and let SA recreate it.  Since doing that when the count approaches 220k it is back to running as it did before.  It starts to purge the older records.

Not sure why the issue randomly started but we have another dataset that we did not fix yet; so we are going to use that to troubleshoot.

Jeremy