Forum Discussion
Develop Power Query outside Power BI or Excel
- 4 months ago
Power Query can struggle is when it’s used for heavy data engineering workloads, for example:
-
Very large datasets (hundreds of millions of rows)
-
Complex iterative logic or procedural transformations
-
Transformations that don’t fold back to the source
-
Large multi-stage pipelines where each step materializes intermediate data
In those situations, tools like SQL, Spark, or Python are often better because they are built for set-based processing at scale.
-
- 4 months ago
Hi Lajouac
Currently, Power Query development is primarily designed to work within Power BI Desktop, Excel, or Power Query Online (Dataflows).
While it is technically possible to use tools like the Power Query SDK with VS Code to write M code, the setup can be complex especially when dealing with authentication and data source connections.
In practice, most users still rely on Power BI Desktop for development and debugging, and then reuse the logic in Dataflows or Fabric for production scenarios.
At this time, there isn’t a fully supported standalone IDE experience for Power Query outside Microsoft tools.
Power Query deserves a little more appreciation, I think.
Rule of thumb for ETL: as far upstream as possible, and as far downstream as necessary.
Especially with SharePoint Online, performance often gets much better when using SharePoint.Contents or the Web.Contents connector instead of SharePoint.Files
- Lajouac4 months agoAdvocate I
Thanks for the SharePoint.Contents tip, I'll check if this makes things faster!