Forum Discussion
Bring Your Own Lake and Dataflow Speed
Hello Everyone,
We have have dire need of an ETL solution, and the Power Query Dataflows are begining to seem like an insufficient solution. We began to move our ETL work over to Dataflows (instead of doing them in each report) in the hopes that it would allow us to scale faster, and take on more clients. The idea was that we could use set naming conventions for the types of data we work with. A set of flows based on data from a system that one client used could be then used as a template for another client and speed up implementation.
The problem we have run into is that the Dataflows through Power Platform are unusably slow. They refresh fine, but the validation time is too long for our use case since we have short implementation times and waiting 10-100 minutes for each step of a query doesn't work.
My question is, is building Dataflows any faster if you bring your own lake? We already use Azure Blob to store our data. I wasn't sure if brining your own lake would mean not using the same Power Platform dataflow capacity that seems to be chronically overstressed. We use embedded capacity to deliver our reports, so we can't turn up the workload for the dataflows, since this appears to be a premium only option.
If that isn't faster, is doing ETL in Azure Data Factory any quicker? I was thinking that we could do Power Query work in Data Factory, and have the transformed files sent back to blob for use by BI datasets. It appears that Data Factory is more limited in its Power Query functionality though, but is it a major difference? We would need to be able to filter by parameters and lists, execute various merges, unpivots, group bys, etc.
Not sure if there are any other options we are missing. We need a way to standardize ETL to grow, and we also need to do more work outside our BI reports because our embedded SKU capacity is becoming an issue, but the Dataflows seem like they might not be a solution due to speed.
On a side note, is there any way to store DAX tables for use in multiple BI reports? For example, creating summary tables that can be queried by multiple reports, or using DAX calculated columns that calculate on refresh and can be accessed by multiple reports? I know reports can share a dataset, but that seems like it could lead to more data being included in a report than it needs. M is great, but DAX does certain things better, so it's a shame to have to create the same DAX tables manually for each report.
Any info or insights are appreciated.
Thanks!
1 Reply
- lbendlin
Super User
You may need to differentiate between the Power BI visuals and the DAX queries that are used to feed (and ultimately render) said visuals. There is such a thing as a "DAX Table" but that is merely DAX code that returns a calculated table.
Are you planning to build a library of reusable DAX code snippets? Maybe look at Quick Measures, or at DAXPatterns.com . Those examples are tested and hardened.
If you think about going on-prem then maybe you should invest in a super fast file storage for CSV files, or a fast SQL server. Keep in mind that deep down dataflows are just glorified Parquet files.