Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified in Microsoft Fabric—for free! For a limited time, the Microsoft Fabric Community team will be offering free DP-600 exam vouchers. Prepare now

Reply
Anonymous
Not applicable

Streaming data limitations

We have a requirement of generating reports on PowerBI for real time transactions. We have roughly 2000,000 transactions flowing in 1 day and we would like reports generated atleast for these number of rows. The reports don't have to necessarily be updated as the data is pushed, but should show as close a real time view of the current situation as possible.

I understand that the push streaming API has a limitation of 200,000 rows for FIFO datasets and 5,000,000 for "none retention policy" Link

My questions are as follows:

  1. If we create a streaming data set push API via the PowerBI service, what dataset is created by default in the background? FIFO or the none retention policy dataset?
  2. For a none-retention policy dataset, what happens when we cross the 5000,000 limit? If there is a failure, does that mean we need to delete old rows via an API call on a frequent basis? An example API to do this will help. Deleting all rows is not an option as business would like reports like KPIs over the last 24 hrs for example.
  3. Since we don't need real time updated reports, would it make more sense to just connect PowerBI to an SQL server or a CosmosDB instead so the point in time data is fetched?
1 ACCEPTED SOLUTION
v-jingzhang
Community Support
Community Support

Hi @Anonymous 

Real-time datasets are mainly designed to display real-time data in tiles on dashboards rather than for power bi reports. If you want to utilize more features of power bi reports and show real-time data, you may consider using DirectQuery mode to create your dataset in Power BI Desktop application.

 

If you don't need real-time data in reports, you can also consider using Import mode to create a dataset and setting incremental refresh rule for it. You can decide for which period to store the data and only refresh the latest data. For example, store the data in latest 5 days and refresh the newly added data in latest day. And configure Scheduled refresh for this dataset.

 

For a push dataset, it is created as a none retention policy dataset by default. You can use the defaultRetentionPolicy query parameter  to change this behavior. But for a streaming dataset, data is temporarily stored for only one hour to render the visuals on dashboard. Reference: Streaming-dataset-matrix

 

Regards,
Community Support Team _ Jing Zhang

View solution in original post

1 REPLY 1
v-jingzhang
Community Support
Community Support

Hi @Anonymous 

Real-time datasets are mainly designed to display real-time data in tiles on dashboards rather than for power bi reports. If you want to utilize more features of power bi reports and show real-time data, you may consider using DirectQuery mode to create your dataset in Power BI Desktop application.

 

If you don't need real-time data in reports, you can also consider using Import mode to create a dataset and setting incremental refresh rule for it. You can decide for which period to store the data and only refresh the latest data. For example, store the data in latest 5 days and refresh the newly added data in latest day. And configure Scheduled refresh for this dataset.

 

For a push dataset, it is created as a none retention policy dataset by default. You can use the defaultRetentionPolicy query parameter  to change this behavior. But for a streaming dataset, data is temporarily stored for only one hour to render the visuals on dashboard. Reference: Streaming-dataset-matrix

 

Regards,
Community Support Team _ Jing Zhang

Helpful resources

Announcements
OCT PBI Update Carousel

Power BI Monthly Update - October 2024

Check out the October 2024 Power BI update to learn about new features.

September Hackathon Carousel

Microsoft Fabric & AI Learning Hackathon

Learn from experts, get hands-on experience, and win awesome prizes.

October NL Carousel

Fabric Community Update - October 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors