Forum Discussion

lowdman87's avatar
lowdman87
Frequent Visitor
3 years ago
Solved

Full Data Table Loading instead of just the Query defined in Power Query

Due to confidentiality within my company I cannot share my dataset, but I will do my best to share some details.  I have a table that gets approximately 2.5 million rowas added to it each day as an archiving process.  As part of the archiving process, a timestamp is applied to each new row.  The table is currently around 350 million rows.  For a specific project, I only need about 15 million of those rows. 

In Power Query, I am filtering the rows by the days needed, then I am also filtering on a few other conditions to yield the desired 15 million rows.  The refresh in Power Query takes a long time, but eventually it shows the desired result with only the dates intended.  When I close Power Query and Apply Settings, the file starts the load.  After some time, the table is showing 293 million rows has been loaded so far, so it seems that ALL of the data in that table is loaded, then it is filtered.  I have not noticed it with past tables, but it could be because they are smaller.  

  1. Is this how Power BI Desktop normally works?  Load the full table then filter to defined requirements?
  2. If so, why does the same table need to load completely again when I change my date condition to only pull 10 million of the rows?  It seems if the entire data set is already loaded then this would be a quick operation, but it is not.  

2 Replies