Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
DonRitchie
Frequent Visitor

Is it Possible to Filter Data from a Dataflow before Import While Building A Dataset?

Title says it all.  I have two 5 GB tables I am trying to import from a dataflow.  I want to manipulate the tables in Power Query, but it takes the better part of an hour to load and apply every time I change a step.  Is it possible to limit the columns before import (not Remove Columns after import) or to filter to a smaller subset of rows (perhaps by a date field - Sept 22 data only, for example?)?

1 ACCEPTED SOLUTION
DonRitchie
Frequent Visitor

That would probably work, but I have not explored the datamart feature much.

What I ended up doing is exporting the dataflow and adding a filter for the PreviousNMonths(2) to the tables I wanted to filter and set up a new QA dataflow that I only need to refresh once a week (or month, if MS gave us the option) in Notepad++.  Tried doing it in the PQ Online in the PBI Service and it was taking forever, so I just edited the JSON manually.  Took 15 minutes to filter 30 or so really large tables to a workable size for me to build datasets from.  I can then reconnect them in Desktop PBI to the full dataflow after I have the dataset built.

 

PBI is such garbage.  My old employer used Tableau and I miss it so much.  Tableau wouldn't even blink at a 1m row table, much less need an hour to join ("merge" WTF) it to another.

 

Thank you for the idea, though!  I appreciate the time and brainpower you gave me.

View solution in original post

2 REPLIES 2
DonRitchie
Frequent Visitor

That would probably work, but I have not explored the datamart feature much.

What I ended up doing is exporting the dataflow and adding a filter for the PreviousNMonths(2) to the tables I wanted to filter and set up a new QA dataflow that I only need to refresh once a week (or month, if MS gave us the option) in Notepad++.  Tried doing it in the PQ Online in the PBI Service and it was taking forever, so I just edited the JSON manually.  Took 15 minutes to filter 30 or so really large tables to a workable size for me to build datasets from.  I can then reconnect them in Desktop PBI to the full dataflow after I have the dataset built.

 

PBI is such garbage.  My old employer used Tableau and I miss it so much.  Tableau wouldn't even blink at a 1m row table, much less need an hour to join ("merge" WTF) it to another.

 

Thank you for the idea, though!  I appreciate the time and brainpower you gave me.

hnguy71
Memorable Member
Memorable Member

@DonRitchie I don't think it's possible at this current time, but what about instead of using dataflows you create a datamart. You can can build an entire model and if you still need to do additional transformations on top of it, you can connect to it via Azure SQL to return relevant fields and records.



Did I answer your question?
Please help by clicking the thumbs up button and mark my post as a solution!

Helpful resources

Announcements
Fabcon_Europe_Social_Bogo

Europe’s largest Microsoft Fabric Community Conference

Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.

Power BI Carousel June 2024

Power BI Monthly Update - June 2024

Check out the June 2024 Power BI update to learn about new features.

PBI_Carousel_NL_June

Fabric Community Update - June 2024

Get the latest Fabric updates from Build 2024, key Skills Challenge voucher deadlines, top blogs, forum posts, and product ideas.

RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.