Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
AravindhPi
New Member

Filtering and Loading 1.7 TB Dataset

Hey everyone!

 

I hope you're all doing well. I'm currently facing a bit of a challenge and could really use some guidance.

 

I'm working on a project where I need to load a dataset around 1.7 TB in size from Databricks into Power BI and filter upto 50 values by using any custom filter visual or any alternate method.

 

what I did for the above Problem:- I have created semantic model and connected as a Direct quey and used filter by list custom visual by devscope, but the dataset is not getting loaded in the visual and also the search in the filter is not working, since I am getting the error "too many values..."

 

Now, I am looking for any other method to filter the data before loading the entire dataset and also please guide me how can we set some default value in filter to load the datset faster, afterthat the user wants to change the filters values and they use to provide the values as a list upto 50 values. (now BU wants to provide 50k values in the filter visual, is it even possible?)

 

If anyone has experience or tips on how to efficiently handle such a large dataset transfer, I would greatly appreciate your insights. 

 

Thank you so much in advance for your assistance. Your expertise could really make a difference here!

 

Best regards,
AravindhPi

2 REPLIES 2
AravindhPi
New Member

Hey @v-jiewu-msft 
Thanks for your reply!

 

Actually I have connected as a live connection to the semantic model created from Tabular editor and I am not able to pass the paramenters. 

 

The BU wants to provide 50k values to filter the table visual and I am looking for any other alternate approach, since it can't be done only with the help of power BI, if I am not wrong.

 

I used "filter by list" custom visual by devscope to filter the table but the problem with this is we can't provide 50k values and also it accepts only around 20 values with the warning message "Too many Values...."
AravindhPi_1-1716541190061.png

Can it be done with any other approach, If so, Please guide me through,

 

your help on this matter is very much appreciated!

 

Regards!

AravindhPi

v-jiewu-msft
Community Support
Community Support

Hi @AravindhPi ,

Based on the description, please try to use the parameter and set default for the parameter.

You can also view the following documents to learn more information.

DirectQuery in Power BI - Power BI | Microsoft Learn

Use parameters to visualize variables - Power BI | Microsoft Learn

Incremental refresh for semantic models and real-time data in Power BI - Power BI | Microsoft Learn

 

Best Regards,

Wisdom Wu

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Helpful resources

Announcements
Europe Fabric Conference

Europe’s largest Microsoft Fabric Community Conference

Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.

Power BI Carousel June 2024

Power BI Monthly Update - June 2024

Check out the June 2024 Power BI update to learn about new features.

PBI_Carousel_NL_June

Fabric Community Update - June 2024

Get the latest Fabric updates from Build 2024, key Skills Challenge voucher deadlines, top blogs, forum posts, and product ideas.

RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

Top Solution Authors