Forum Discussion
Should I use SQL or Power Query for Data Transformations? Or something else?
Hi analyticsyc - Roche's Maxim - do transformation as far upstream as possible and downstream when necessary.
By creating SQL views for the basis of your all your extracts is advantageous. When changes are made to the main database, the views can be subject to System Testing before production deployment. That said, it is unlikely that tables in ERP system to be changed. But, how easy is it for you to include and make changes to views in the main ERP. Also, when you make changes to these views, the changes won't necessary load into your Power BI Data models.
I would create pipeline before you import into Power BI Dataset files to pull the ERP data and then transform it.
Personally, I like Dataflows for this it does things in a No-Code/Low-Code way and uses query folding against the source. This allow you centralise the Power Query in one place, but you won't have the flexibilty or scale of other Azure data lake solutions.
So, it makes sense to replicate the ERP into a lakehouse for bronze, silver and gold transformations. I would recommend using Azure pipeline to replicate the ERP database. Here are couple of videos to watch:
Modern Data Lakehouse? Where do you even start??? Azure Synapse of course! - YouTube.
Merging your data in a modern lakehouse data warehouse - YouTube.
Hi analyticsyc - Roche's Maxim - do transformation as far upstream as possible and downstream when necessary.
By creating SQL views for the basis of your all your extracts is advantageous. When changes are made to the main database, the views can be subject to System Testing before production deployment. That said, it is unlikely that tables in ERP system to be changed. But, how easy is it for you to include and make changes to views in the main ERP. Also, when you make changes to these views, the changes won't necessary load into your Power BI Data models.
I would create pipeline before you import into Power BI Dataset files to pull the ERP data and then transform it.
Personally, I like Dataflows for this it does things in a No-Code/Low-Code way and uses query folding against the source. This allow you centralise the Power Query in one place, but you won't have the flexibilty or scale of other Azure data lake solutions.
So, it makes sense to replicate the ERP into a lakehouse for bronze, silver and gold transformations. I would recommend using Azure pipeline to replicate the ERP database. Here are couple of videos to watch:
Modern Data Lakehouse? Where do you even start??? Azure Synapse of course! - YouTube.
Merging your data in a modern lakehouse data warehouse - YouTube.