Should I use SQL or Power Query for Data Transformations? Or something else?

[analyticsyc]

I work with Syspro (an ERP system) which sits on top of an SQL Database. It records all of our company's sales information, and I use PowerBI to tie together multiple tables and create reports for different departments. I currently have 1 main underlying model, which I adjust visuals for different end users. As the number of reports has grown, I am struggling to keep changes consistent throughout all my models. I make my data transformations in Power Query, which is great, until I need to make the same change across 10+ different files. 

I'm hoping someone has advice on how to create a sustainable method of data transformations rather than using Power Query. My initial thought was using SQL to transform my data, and then linking all my models to that transformed data. I would store the transformed data in Azure. That way, when I need to make a change (say adding a calculated column), I only have to make the change in SQL and all my PowerBI files are automatically updated the next time they refresh. One requirement is that wherever this transformed data is stored, I need to be able to schedule refreshes through power automate. Using SQL is just my initial idea though, and I'm interested in hearing other's opinions on what works best.

[Rickmaurinus]

Good question, not per se an easy answer.

Power Query and SQL are not adversaries; they're allies, each offering unique advantages.

Power Query is known for its user-friendly interface, making it great for beginners and quick tasks. It also serves as a bridge for Excel users, allowing more sophisticated data transformations, especially in on-the-spot situations.

On the other hand, SQL is unmatched in its performance, scalability, and in executing precise and secure operations. It provides superior control, robust security, and ensures data reliability.

Depending on your project, one may fit better, but understanding and leveraging the strengths of both can be a great combination.

I wrote an article on this to further explain the concepts: 

https://gorilla.bi/power-query/power-query-vs-sql/

[Daryl-Lynch-Bzy · Accepted solution]

Hi @analyticsyc - Roche's Maxim - do transformation as far upstream as possible and downstream when necessary.

By creating SQL views for the basis of your all your extracts is advantageous.  When changes are made to the main database, the views can be subject to System Testing before production deployment.  That said, it is unlikely that tables in ERP system to be changed.  But, how easy is it for you to include and make changes to views in the main ERP.  Also, when you make changes to these views, the changes won't necessary load into your Power BI Data models.

I would create pipeline before you import into Power BI Dataset files to pull the ERP data and then transform it.  

Personally, I like Dataflows for this it does things in a No-Code/Low-Code way and uses query folding against the source.  This allow you centralise the Power Query in one place, but you won't have the flexibilty or scale of other Azure data lake solutions.

So, it makes sense to replicate the ERP into a lakehouse for bronze, silver and gold transformations.  I would recommend using Azure pipeline to replicate the ERP database.  Here are couple of videos to watch:
Modern Data Lakehouse? Where do you even start??? Azure Synapse of course! - YouTube. 
Merging your data in a modern lakehouse data warehouse - YouTube.