Hey guys,
I'd like to ask some questions about CI/CD best practices now that git is available and we still have deployment pipelines around.
Setting:
I do have a dataset which connects to a SQL server. Using a parameter called 'Schema', I can connect to different environments: DEV, TEST, PROD - and therefore to different data. The dataset is hosted in a git repository which is connected to a Power BI workspace.
I do have a fabric capacity / PPU capacities available.
I'd like to move all such datasets to an Incremental Refresh policy, thus applying partitions & allowing for much shorter refresh cycles, reducing load and cost on the SQL server.
However, what is the best way of doing this keeping the CI/CD approach in mind?
The problems I see are the following:
When using a git repository with branches dev, test, main for the three different environments, I could connect each branch to a corresponding workspace. However, whenever I make changes to the data model in DEV and merge into TEST/PROD (in git), I will have to override the 'Schema' parameter in the dataset as well.
This would then cause the partitions to crash - the dataset to be empty, no?
Allow deployment pipelines (with stages DEV, TEST, PROD) for a different scenario as on each deloyment you can specify the parameters it should keep? Or do deployment pipelines override the whole dataset as opposed to only the changes made as git does?
Ultimately, most of the work can be automized with the REST API when using pipelines as opposed to the git approach, as you still must manually sync incoming changes. Are pipelines therefore perferable?
How do you guys handle your CI/CD approach with incremental refresh?
Thanks for your help and input 😄
