Knowing the uncompressed dataset size on the gateway

[Blame73]

Since with datasets that always refreshed overnight as scheduled I'm now getting the "received uncompressed data on the gateway client has exceeded limit" error - without many changes from the previous datasets size - I'd like to know if there's a way to know the actual uncompressed data size.

(And I wonder why what always used to work, doesn't work anymore...)

Thanks,

Davide

[v-qiuyu-msft]

The same issue reported internally: CRI 39670466. And I already share your .saz file internally. Will update here once I get information.

Best Regards,
Qiuyun Yu

[DarylM]

Hi, 

Following this issue. I am having the same problem. Reports that refreshed fine before now are erroring with this same error. 

Interested in any solution. 

Something went wrong

The received uncompressed data on the gateway client has exceeded limit.
Please try again later or contact support. If you contact support, please provide these details.

Cluster URIWABI-US-EAST2-redirect.analysis.windows.net

Activity ID55fca6e0-b265-462e-94bb-3477c7e51da5

Request IDc336b24f-d41b-6ada-b6b4-32c571cc02d7

Time2017-06-22 12:03:56Z

[v-qiuyu-msft]

Hi all,

I got the information internally below:

This is due to the cutoff of data volume/size <= 10GB as enforced by DS

Best Regards,
Qiuyun Yu

[Blame73]

Thanks but... what does it mean?

Also, it doesn't answer the questions:

why it used to work before with the same dataset size?

How can I know the uncompressed dataset size?

I'm going to lose customers because of this and you're goint to lose customers too. That's not the correct way to push the Premium edition.

Don't mark this issue as 'solved' as it's really not.

[DarylM]

This also does not make any sense in my case either. My dataset is only 500MB and is well under the 1GB dataset limit. And, it was refreshing just fine until I updated the gateway. I have at lease 4GB of space in the power BI service, so that can't be the issue either. 

Maybe I am missing something here. 

[iJuancho]

I think this is a step on the part of microsoft to push us to its premium service.

i found this article where it says: 

Uncompressed data limits for refresh
The maximum size for datasets imported into the Power BI service is 1 GB. These datasets are heavily compressed to ensure high performance. In addition, in shared capacity, the service places a limit on the amount of uncompressed data that is processed during refresh. This limit accounts for the compression, and therefore is much higher than 1 GB. Datasets in Power BI Premium are not subject to this limit. If refresh in the Power BI service fails for this reason, please reduce the amount of data being imported to Power BI and try again.

https://powerbi.microsoft.com/en-us/documentation/powerbi-refresh-troubleshooting-refresh-scenarios/

[DarylM]

Thanks iJuancho.

Here is what I still don't understand. My uncompressed dataset is only 500MB in size. 

Compressed its around 300MB. I don't understand why I get these errors when my dataset is well under the iGB limit as stated in the article. 

[Burningsuit]

As I understand it, the size of the datamodel depends on (at least) three things. Firstly, obviously, the number of records. Then, the cardinality of the columns in those records. Because Power BI uses Run Length Encoding, columns with lots of common data will compress better than columns with unique data (which may not compress at all). Finally the number of calculated columns is important, as calculated columns may not be compressed (depending on how long it takes to compress the original columns in the data model). So if I were seeing this type of problem, it might be because the data has changed. No new records need be introduced, but colums of data may change from a few repeated records to unique records, which will have an impact on the model size. Also, adding lots of calculated columns may also increase the size of the datamodel without adding any records. Calculated columns are tempting, because they're easy to understand, but they can be inefficient, as they are calculated for every row in the dataset, whether thay are needed or not, and measures (which are only calculated when required) may be better.

This complex corrolation with the data explains why it's difficult to know how much your data will compress by, and can explain why a dataset that earlier did compress to less the 1GB will no longer compress. If you have added some calculated columns, or changed the cardinality of any rows you'll see an increase in datamodel size. Equally, compressing the datamodel take computing cycles, which Microsoft provides, and may limit. But if I had this problem I'd look first to my data to see what was not compressing, or what was taking a long time to compress first.

[DarylM]

Thanks! That is helpful! 

I will look into the dataset. I do have a fair amount of calculations that may be the issue as you stated.

Thanks again!