Forum Discussion

kbearman's avatar
kbearman
New Member
3 years ago
Solved

Filtering performance within power query referencing dataflow

Hello

 

We have worked around this for now but I would appreciate a greater understanding of the performance issue we found

 

We have two dataflows that each create a table based on a folder connector (so each table will contain say 6 month's worth of data). These dataflows are separate because the refresh schedules for each a different (due to the batch timing creating each set of files)  

 

I add connectors to each of these dataflow entities into my PowerBI data model and load the data. All good. No problem

 

Now we find out that we need to filter some of the data out of each set of these files so I have a few options:

  1. Filter out in Dax or by virtue of our data model - would rather avoid because why load data I will throw away
  2. Have a hardcoded filter table in PBI M query and do a "not List.Contains" - performance is great, functionally good but I don't want to touch my PBIX file if I need to add to this list
  3. Put the list in a meta data file, pop it in a dataflow and then reference that dataflow entity in my PBI dataset - operationally this is perfect but the performance is horrific (gave up on apply changes when hit 2 hours - refresh would normally be 20 minutes)

I understand that there is a difference between powerbi.dataflows and powerplatform.dataflows in terms of folding but both performed terribly. In the end we added the table referencing the meta data exclusion file in both of the dataflows, did the filtering there and then it was fine but it feels very clunky so there must be a better way

 

Data volumes here are not massive so one DF produces 600k rows and the other 10m

 

Thanks

Kerry

  • Please share the M code from your #3 scenario. I suspect you could use List.Buffer or Table.Buffer to load your meta file contents only one time to be used in your Dataflow filter.

     

    Pat

2 Replies

  • ppm1's avatar
    ppm1
    Solution Sage

    Please share the M code from your #3 scenario. I suspect you could use List.Buffer or Table.Buffer to load your meta file contents only one time to be used in your Dataflow filter.

     

    Pat

    • kbearman's avatar
      kbearman
      New Member

      List.Buffer does the trick. Thank you.

      Now you've said that I can find loads of posts with this detail on but couldn't see for looking before!