Streaming data limitations

[Anonymous]

We have a requirement of generating reports on PowerBI for real time transactions. We have roughly 2000,000 transactions flowing in 1 day and we would like reports generated atleast for these number of rows. The reports don't have to necessarily be updated as the data is pushed, but should show as close a real time view of the current situation as possible.

I understand that the push streaming API has a limitation of 200,000 rows for FIFO datasets and 5,000,000 for "none retention policy" Link

My questions are as follows:

1. If we create a streaming data set push API via the PowerBI service, what dataset is created by default in the background? FIFO or the none retention policy dataset?
2. For a none-retention policy dataset, what happens when we cross the 5000,000 limit? If there is a failure, does that mean we need to delete old rows via an API call on a frequent basis? An example API to do this will help. Deleting all rows is not an option as business would like reports like KPIs over the last 24 hrs for example.
3. Since we don't need real time updated reports, would it make more sense to just connect PowerBI to an SQL server or a CosmosDB instead so the point in time data is fetched?

[v-jingzhang · Accepted solution]

Hi @Anonymous 

Real-time datasets are mainly designed to display real-time data in tiles on dashboards rather than for power bi reports. If you want to utilize more features of power bi reports and show real-time data, you may consider using DirectQuery mode to create your dataset in Power BI Desktop application.

If you don't need real-time data in reports, you can also consider using Import mode to create a dataset and setting incremental refresh rule for it. You can decide for which period to store the data and only refresh the latest data. For example, store the data in latest 5 days and refresh the newly added data in latest day. And configure Scheduled refresh for this dataset.

For a push dataset, it is created as a none retention policy dataset by default. You can use the defaultRetentionPolicy query parameter  to change this behavior. But for a streaming dataset, data is temporarily stored for only one hour to render the visuals on dashboard. Reference: Streaming-dataset-matrix

Regards,
Community Support Team _ Jing Zhang