Forum Discussion
Query Folding and Access Native Query for Incremental Refresh
What happens if instead of fancy M logic like this, you query the table directly and then build your transformation steps in Power Query? Does that allow query folding at ALL levels? Does it even do query folding at the top SELECT step?
My thought process was to do all transformations prior to loading it into PBI to lessen the load. Then just load the query in using import mode and apply incremental refreshing to the report to make it even swifter
That is what Query Folding IS! It pushes data transformation steps (like filtering) and many others down to the source , letting those operations be handled by the server. The result is that the server is probably better equipped to handle the steps natively, and it hopefully results in fewer rows returned by the query, and less processing overhead by the client (Power BI).
Below is a very simple example of query folding on my Customer table for customers in the city of Buffalo:
Let the query folding happen naturally. Don't try to force it. If your source is capable (things like Excel and flat files are NOT capable) then it just happens automagically.
Hope that helps.
Gotcha. Well, some of my queries have some pretty complex transformations. I'd like to keep these transformations as is in the SQL. However, I want to add incremental refreshing to my table which requires query folding to be enabled. I was hoping to just add a filter for the Date range ontop of my original SQL and have this filter be equal to parameters labeled "RangeStart" and "RangeEnd".
>> some of my queries have some pretty complex transformations<<
How complex? I don't know about Teradata, but SQL Server can handle a lot of the transformations listed, even GROUPING and PIVOT/UNPIVOT. I chokes on things like splitting a field based on the instance of a delimiting character, or things like FILL DOWN.
If you absolutely MUST have your custom transformations and they preclude query folding in Power BI, then the best bet is to wrap all that in a view in the source database and simply reference the view instead of a base table.
But Power BI needs A) Query Folding and B) appropriate date column in order to support Incremental Refresh.