Hello,
We use Dataflows Gen 2 to build our data transformations, and use Snowflake as our database. We run all our transformations on one single Snowflake database and try to utilize query folding as much as possible to minimize data movement. Inherently though, query folding only supports source side of the operations, and Dataflows brings the transformed, final set of data from source to its memory, only to write it back to the very same destination.
If Dataflows can distinguish it is running over a single data store/database for a set of operations, it can convert all logic to SQL (or data store's native language) and let actual execution completely happen on the database itself, preventing an unnecessary dat journey to its memory and back to the database itself.
In the past, when cloud was not a thing, we used Informatica for complete query folding (they call it push query) , and that improved ETL performance drastically.
Query folding is a great feature, and if it can cover complete DML execution on the database itself, that can bring the performance to next level.
Thank you.
on:
Multi-item & folder-level delete
on:
Replace All that are Not a Value
on:
Introducing Staging in Power Query: Smarter Valida...
SMATIN
on:
Support non-deterministic functions in scalar UDFs...
on:
Edit Top N Rows” (grid editor) for Fabric Warehous...
d_gosbell
on:
RDL Subscription Options in Power BI Service