topic Re: What happens when you reach the size limit of a Push Dataset with Historical Analysis Enabled in Developer

What happens when you reach the size limit of a Push Dataset with Historical Analysis Enabled

flennon — Fri, 06 Jul 2018 14:00:47 GMT

I thought up until recently that push datasets followed the 200k first in first out (basciFIFO) retention policy. I realise now that this is just for streaming datasets as the 200k rows is stored in cache.

I am pushing into the dataset using the RESTful API but I have historical analysis enabled so that I could avail of all the different types of charts for my dashboards.

I see here that datasets are limited to 1gb. What happens when that limit is reached. Is there any FIFO system here? I know you cant selectively delete from a dataset, its either all or nothing. Would that mean that when I hit the 1gb limit, I'd have to delete out all the data and start again?

Re: What happens when you reach the size limit of a Push Dataset with Historical Analysis Enabled

az2451 — Fri, 06 Jul 2018 15:06:24 GMT

The answer to your question is right below: https://docs.microsoft.com/en-us/power-bi/service-admin-manage-your-data-storage-in-power-bi#what-happens-when-you-hit-a-limit

You simple have to reduce the size to create new dashboards etc.

There is not other way around, thats the rule.

You can try it out by your self..

Greetings

Re: What happens when you reach the size limit of a Push Dataset with Historical Analysis Enabled

flennon — Fri, 06 Jul 2018 15:48:37 GMT

My apologies, I did not communicate my point across well enough.

I understand that I have to reduce the size of the dataset. However, when attempting to find a way to reduce the number of records in the dataset, I found that there is no method to delete specific rows (i.e. You cant delete rows older than a certain date), you can only delete all rows.

Therefore, I'm asking is there something I'm missing, or do I really have to delete all the rows in the dataset and start again when I hit the maximum capacity? Rather than being able to delete 50mb of the oldest records in the dataset.

Re: What happens when you reach the size limit of a Push Dataset with Historical Analysis Enabled

v-jiascu-msft — Mon, 09 Jul 2018 07:26:29 GMT

Hi @flennon,

You can set a retention policy. Then it will run automatically. Please refer to api-automatic-retention-policy-for-real-time-data.

Best Regards,

Dale

Re: What happens when you reach the size limit of a Push Dataset with Historical Analysis Enabled

flennon — Mon, 09 Jul 2018 08:42:31 GMT

Hi Dale,

Are you certain you can set the retention policy for a dataset when 'Historic Data Analysis' is enabled? The solution to this forum post states that PowerBI will store everything without the basicFIFO retention policy if Historic Data Analysis is enabled.

This would make sense as other streaming datasets arent stored, they're just held in memory.

I've also been trying to see what other retention policies there are, but there only seems to be basicFIFO and no retention policy, are there any others?

Re: What happens when you reach the size limit of a Push Dataset with Historical Analysis Enabled

v-jiascu-msft — Mon, 09 Jul 2018 15:02:28 GMT

Hi @flennon,

Yes, I'm quite sure. I have tested it. Please refer to the snapshot below.

1. If it's only a streaming dataset, I can't create a report.

2. If it's only a push data set, it won't have an API info.

3. I pushed 300K rows into it from ID 1 to ID 300000. (I intended to do so. It's obvious my code doesn't that precise. But the 300K data is pushed successfully)

There are only 226K rows exist. And the minimum ID is 80001.

Best Regards,

Dale

Re: What happens when you reach the size limit of a Push Dataset with Historical Analysis Enabled

flennon — Mon, 09 Jul 2018 15:07:59 GMT

Thank you very much Dale