Forum Discussion
Create Custom Function to Filter Main Table and use it as filter reference
Hello,
I'm connecting to a > 10 GB dataflow and it's problematic as sometimes I just need to some specific range in dates and filter some categorical columns. Trying my luck here but it is possible to create a function that filters a table and use that same table as a way to create some level of data validation for me to filter first in the function before I load the humongous data?
Example:
*Imagine the data has this column
| Date | Category 1 | Category 2 |
| 1/18/2021 | Cat | NS |
| 10/27/2021 | Dog | NS |
| 10/7/2022 | Dog | NS |
| 11/13/2021 | Dog | US |
| 11/25/2022 | Dog | US |
| 2/4/2022 | Dog | US |
| 3/14/2021 | Cat | US |
| 3/3/2022 | Cat | US |
| 4/22/2022 | Cat | RU |
| 5/9/2021 | Cat | US |
| 7/15/2022 | Dog | RU |
| 7/17/2021 | Dog | US |
| 7/27/2021 | Cat | US |
| 8/29/2021 | Cat | US |
| 9/14/2022 | Cat | US |
The Function should Dynamically create these following fields (or if not possible, reference it):
Then "Run" the function and output the filter data
Edit 1: If this is not possible, can anyone provide a workaround to have the data filtered first before actually loading the data from dataflow?
2 Replies
If this Dataflow in in a Workspace backed by Premium (and you use the new Dataflows connector), you can turn on the Enhanced Compute Engine so that query folding can take place.
Pat
Unfortunately, I'm only on Pro License. It would be great to have tho.
But thanks, this is informative! Hopefully there's just a way to reduce the data without needing to filter it in power query as I'm planning to deploy it publicly and those who will be utilizing the data might need to really wait for things to load before they can actually get the data they need
