Help working with big data: limit rows to reduce file size

[cris1196]

Hi guys, I need help with the following:

I have a file with many rows, apart from being heavy, I can't finish applying some changes in Power Query due to "memory problems". So what I'm wondering is if there is a way to: limit the rows that I use but, when I publish the dataset, all the information is still used.

I found something similar in this video:

https://www.youtube.com/watch?v=_zYvybVMk7k

But I read that it has some disadvantages, example one user said:
"...this approach may get us into trouble as if we pick only TOP X rows for the fact table and the remaining tables might be pulling ALL or lesser rows (after applying the same TOP logic to them), we might end up with missing related records and the magical BLANK rows too..."

I guess Microsoft foresaw this and there is another less problematic option. I have Premium and access to everything, is there an easier way? I read about using the deployment pipeline but I couldn't find anything on the internet that tells me how to do it.

Thank you!

[Idrissshatila · Accepted solution]

Hello @cris1196 ,

Try using incremental refresh, so you would archive a specific period of data and only refresh the lat month or 2 or even 6.

check this to know more about incremental refresh and how you could set it up https://www.youtube.com/watch?v=Kui_1G6kQIQ&t=148s

If I answered your question, please mark my post as solution, Appreciate your Kudos 👍

Follow me on Linkedin