Data limit and incremental refresh

[jitpbi]

Hi,

My data source is web where i need to bring last 1 year data into power bi report. I want to store rows for the last 1 year and refresh the rows for last 15 days based on datetime field. 

My problem is when trying to load last 1 year data into power bi desktop it gets fail everytime with timeout error. I am not sure it may be due to datalimit size in signle dataset of 1 GB. I just tested by loading 1 day data and dataset size for 1 day data is 1 MB.

I need experts advise how to handle this where i can bring 1 year data into my report and can set inremental refresh.

Thanks

[ibarrau]

Hi. The Power Bi limitations for pro/free are:

- Data model of 1GB.

- One tabla can't size more than 10GB before the engine compressit.

- Two hours windows for refresh.

If you want to keep a good track of an API data, i wouldn't recommend loading it in Power bi directly. Unless the are in the first two limitations, the best appoarch is to store that data somewhere. My recommendations are Azure DataLake Gen2 or Azure Synapse (this last one would be in case you want to start building models for the company and not just for one table).

APIs take time. If you are requesting one year information it might be a timeout of how much time is taking to respond.

Hope that helps,

[jitpbi]

Thank you @ibarrau  for the above explanation.

In addition to my original post, I am getting data through API but the data source is Cosmos DB. This is because some internal guidelines but can be explore to connect directly to cosmos db if required.

so just wanted to understand, is there any workaround possible to store 1 year data if i directly connect to Cosmos DB instead taking it through API?

[ibarrau]

Interesting. I haven't tried cosmos db. You can try the Power Bi connector to get your data. I don't know if you can query it like "last 365 days" or something like that. If you can't, consider filter the last 1 year of data from Power Query (transform data menu) that run before close & apply.

Let's see how that works before considering moving data to a different place.

Regards,