Filtering and Loading 1.7 TB Dataset

[AravindhPi]

Hey everyone!

I hope you're all doing well. I'm currently facing a bit of a challenge and could really use some guidance.

I'm working on a project where I need to load a dataset around 1.7 TB in size from Databricks into Power BI and filter upto 50 values by using any custom filter visual or any alternate method.

what I did for the above Problem:- I have created semantic model and connected as a Direct quey and used filter by list custom visual by devscope, but the dataset is not getting loaded in the visual and also the search in the filter is not working, since I am getting the error "too many values..."

Now, I am looking for any other method to filter the data before loading the entire dataset and also please guide me how can we set some default value in filter to load the datset faster, afterthat the user wants to change the filters values and they use to provide the values as a list upto 50 values. (now BU wants to provide 50k values in the filter visual, is it even possible?)

If anyone has experience or tips on how to efficiently handle such a large dataset transfer, I would greatly appreciate your insights. 

Thank you so much in advance for your assistance. Your expertise could really make a difference here!

Best regards,
AravindhPi

[AravindhPi]

Hey @Anonymous 
Thanks for your reply!

Actually I have connected as a live connection to the semantic model created from Tabular editor and I am not able to pass the paramenters. 

The BU wants to provide 50k values to filter the table visual and I am looking for any other alternate approach, since it can't be done only with the help of power BI, if I am not wrong.

I used "filter by list" custom visual by devscope to filter the table but the problem with this is we can't provide 50k values and also it accepts only around 20 values with the warning message "Too many Values...."

Can it be done with any other approach, If so, Please guide me through,

your help on this matter is very much appreciated!

Regards!

AravindhPi

[Anonymous]

Hi @AravindhPi ,

Based on the description, please try to use the parameter and set default for the parameter.

You can also view the following documents to learn more information.

DirectQuery in Power BI - Power BI | Microsoft Learn

Use parameters to visualize variables - Power BI | Microsoft Learn

Incremental refresh for semantic models and real-time data in Power BI - Power BI | Microsoft Learn

Best Regards,

Wisdom Wu

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.