Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
cris1196
Helper I
Helper I

Help working with big data: limit rows to reduce file size

Hi guys, I need help with the following:

 

I have a file with many rows, apart from being heavy, I can't finish applying some changes in Power Query due to "memory problems". So what I'm wondering is if there is a way to: limit the rows that I use but, when I publish the dataset, all the information is still used.

 

I found something similar in this video:

https://www.youtube.com/watch?v=_zYvybVMk7k

But I read that it has some disadvantages, example one user said:
"...this approach may get us into trouble as if we pick only TOP X rows for the fact table and the remaining tables might be pulling ALL or lesser rows (after applying the same TOP logic to them), we might end up with missing related records and the magical BLANK rows too..."


I guess Microsoft foresaw this and there is another less problematic option. I have Premium and access to everything, is there an easier way? I read about using the deployment pipeline but I couldn't find anything on the internet that tells me how to do it.


Thank you!

1 ACCEPTED SOLUTION
Idrissshatila
Super User
Super User

Hello @cris1196 ,

 

Try using incremental refresh, so you would archive a specific period of data and only refresh the lat month or 2 or even 6.

 

check this to know more about incremental refresh and how you could set it up https://www.youtube.com/watch?v=Kui_1G6kQIQ&t=148s

 

If I answered your question, please mark my post as solution, Appreciate your Kudos 👍

Follow me on Linkedin



Did I answer your question? Mark my post as a solution! Appreciate your Kudos
Follow me on LinkedIn linkedIn
Vote for my Community Mobile App Idea

Proud to be a Super User!




View solution in original post

1 REPLY 1
Idrissshatila
Super User
Super User

Hello @cris1196 ,

 

Try using incremental refresh, so you would archive a specific period of data and only refresh the lat month or 2 or even 6.

 

check this to know more about incremental refresh and how you could set it up https://www.youtube.com/watch?v=Kui_1G6kQIQ&t=148s

 

If I answered your question, please mark my post as solution, Appreciate your Kudos 👍

Follow me on Linkedin



Did I answer your question? Mark my post as a solution! Appreciate your Kudos
Follow me on LinkedIn linkedIn
Vote for my Community Mobile App Idea

Proud to be a Super User!




Helpful resources

Announcements
Europe Fabric Conference

Europe’s largest Microsoft Fabric Community Conference

Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.

Power BI Carousel June 2024

Power BI Monthly Update - June 2024

Check out the June 2024 Power BI update to learn about new features.

RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.