Forum Discussion
Merge two tables with different source using query
- 1 year ago
Hi, thank you for your response. I managed to make the second option working, but time was not reduced. At first it load the whole DB and after that executing filtering. As for the first option I had something similar, but there is still time problem. Is there a power app that could help me prepare data from sql, which could be then passed to powerbi? Or maybe some power app which would display data in the form of table, that would be filtered based on input value? Thanks in advance.
- 1 year ago
Ok, so first thing: You're never going to get this query to fold as it is because you're using a native query as your source i.e. you're using Sql.Database() with an SQL SELECT script.
Based on the simplicity of the SQL query (from what I can see, anyway) it's totally unnecessary to have this query as a native connection to begin with.
Assuming your tables are all coming from the SAME DB, you can set up a foldable query set like this:
Create a new SQL query and connect to [DBName].[TEST] from within the connection dialog. Then multi-select (Ctrl+click) the [Test_ID] and [Place] columns and go to the Home tab > Remove Columns (dropdown) > Remove Other Columns. If you right-click on your 'Remove Other Columns' query step now, you should see 'View Native Query' lit up. Selecting this will show you the native ("folded") query sent to the source.
Rinse and repeat in new queries for [DBName].[RESULT] and any other tables you need to join. For these other tables, you can right-click on the query name and DE-select 'Enable Load'.
Perform the joins between these tables in Power Query and, when you right-click on the query steps in your first query, you should, again, see 'View Native Query' lit up. This will now be sending the full optimised query back to the source for your base table, all the join tables, and your transformations so far. Once you confirm this is working, you can then use my technique(s) to join your SP query in a foldable way.
This is a pretty big topic so I can't go into the Nth degree of detail here, but this is the basic setup of a foldable query set. There's TONS of resources online about Query Folding as a topic, and I've provided a couple of links previously to get you started - please read these as they basically explain what I've described here.
I can tell you this: Once you understand query folding and learn to implement it correctly, it will completely change your Power Query mindset, and have you producing very efficient PBI models.
Pete
Ok. Try adding another foldable step after the filter step (just for testing) e.g. Remove Other Columns or something. Sometimes Power Query doesn't 'know' it can fold the query at certain query steps. You might find that, by adding a basic foldable step after this one, your final query step does actually fold, including the filter step.
You may also decide that the overall query now runs fast enough with everything else folding and just this one step not folding at the end of the query - that's an acceptable outcome too.
Pete
I've added a new step, where I have removed one column, but it is still not folding. Well it take too much time to refresh because of size of data it has to import. Maybe this is simply not possible to perform in Power BI effectively, I'll try different program. Thank you so much for helpful advices, I will definitely use them in different projects.