Hello
We have worked around this for now but I would appreciate a greater understanding of the performance issue we found
We have two dataflows that each create a table based on a folder connector (so each table will contain say 6 month's worth of data). These dataflows are separate because the refresh schedules for each a different (due to the batch timing creating each set of files)
I add connectors to each of these dataflow entities into my PowerBI data model and load the data. All good. No problem
Now we find out that we need to filter some of the data out of each set of these files so I have a few options:
- Filter out in Dax or by virtue of our data model - would rather avoid because why load data I will throw away
- Have a hardcoded filter table in PBI M query and do a "not List.Contains" - performance is great, functionally good but I don't want to touch my PBIX file if I need to add to this list
- Put the list in a meta data file, pop it in a dataflow and then reference that dataflow entity in my PBI dataset - operationally this is perfect but the performance is horrific (gave up on apply changes when hit 2 hours - refresh would normally be 20 minutes)
I understand that there is a difference between powerbi.dataflows and powerplatform.dataflows in terms of folding but both performed terribly. In the end we added the table referencing the meta data exclusion file in both of the dataflows, did the filtering there and then it was fine but it feels very clunky so there must be a better way
Data volumes here are not massive so one DF produces 600k rows and the other 10m
Thanks
Kerry
