PowerBI - CICD using Azure DevOps

[Jayendran]

Introduction

Power BI is a business analytics service by Microsoft. It aims to provide interactive visualizations and business intelligence capabilities with an interface simple enough for end-users to create their own reports and dashboards. Currently, there is no direct way of Implementing CICD in Power BI. Using the Publish menu from the Power BI Desktop is the easiest way to deploy/publish a report as of now. In this article, we are going to see how we can implement source control and CICD for Power BI using Azure DevOps.

                                                   

Difficulties using direct publishing from Power BI Desktop

There are some difficulties of publishing a Power BI Report directly from Power BI Desktop, that includes

• Power BI Desktop (Client Tool) needs to be installed in all the Environment (Production,Dev,QA Servers)
• Without having a proper Source Version Control, the changes couldn't be tracked.
• The developer needs access to all the data sources across every environment. For E.g, if we have 2 environments say Dev and PROD and the data source is SQL server, then the developer needs access to both Dev SQL Server and PROD SQL Server.
• The developer needs to manually change the data source settings for every environment

Best Practice

• If you using Cloud Data sources you don't need a gateway, but for on-premises data sources, we should use the gateway. Another Advantage of using the gateway for cloud data sources is your credentials were encrypted and stored in the gateway server and not with Power BI
• Try to reduce hardcoding your datasources in your reports (or) In other words, try to use Parameters for your data source as much as possible, Also keep in the mind updating parameter using Power BI REST API don't support for the datasources like SQL,Oracle,Teradat...
• Organzie your Source Control folder structure based on your requirements/client-specific /data sources.

CICD Process for Power BI Reports 

As soon as we say the automation(CICD) the things that would come to mind be like using API / Cmdlets. Likewise,we can use Power BI Rest API /  PowerBI cmdlets for automating Power BI reports.

Maik van der Gaag created a great (Azure DevOps) extension called Power BI Actions, which makes things easier to handle the CICD for Power BI. We are going to see how this extension would be helpful for us in the CICD process., This extension will be handly in most way but in sometimes we need to do some additional steps that won't support by this extension for now,so we also going to use some PowerShell Scripts.

✔ Sample Power BI Reports and Environment Specification

For the demo purpose here you will see two different powerbi reports deploying in two different environments (workspace), keep this as a reference you can do use your own reports.

Dev Environment Workspace  :  PowerBI_CICD

Prod Environment Workspace :  PowerBI_CICD_PROD

PowerBI Reports:

1. AdventureReports.pbix
2. Products.pbix

Dev Environment Data Sources

AdventureReports => SQL =>devenvironment.database.windows.net;devdb

Products => OData => https://services.odata.org/V3/OData/OData.svc/

Prod Environment Data Sources

AdventureReports => SQL =>prodenv.database.windows.net;proddb

Products => OData => https://services.odata.org/V3/(S(2z5tfelmekddgffwed3sq4ve))/OData/OData.svc/

✔ Power BI Reports in Azure Repo 

The initial step here is to set up a source control version for our Power BI Reports (.pbix files). In this article, we will see on how to setup the Azure Repo as version control for our Power BI Reports. Here you can also use Version control like BitBucket,TFVC,GitHub ,SubVersion etc., instead of Azure Repo.

• Create a new Azure Repo

• Clone the repo in your local and commit your pbix files in the Azure Repo. Here you can see how you can do that using VS Code
  

Commit the changes




Push the Commit

•  Once you pushed the commit, you will see your commit in the azure repo (like below)

✔ Azure Build Pipeline (CI) for Power BI

Now we had our PBI reports in the Azure Repo's. It's time to setup the CICD.Let see how we can setup the Continous Integration

Setup the CI is actually very easy, We just need to include 2 task in the Build pipeline

• Copy Task
• Publish Task

Classic Editor with-out YAML

Copy Task to Staging Artifact



Publish Task

Using YAML

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
 
trigger:
- master
 
pool:
  vmImage: 'ubuntu-latest'
 
steps:
- task: CopyFiles@2
  displayName: 'Copy Files to: Staging Artifact'
  inputs:
    SourceFolder: Reports
    TargetFolder: '$(Build.ArtifactStagingDirectory)'
    OverWrite: true
     
- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: drop'

✔ Azure Release Pipeline (CD) for Power BI

This release pipeline is just a demo purpose, actual release pipeline and tasks may varies depends on the various use-cases based on the end-user.So this section is just gives you a basic understanding on how you can use the existing features in azure devops to implement the Continous deployment of Power BI Reports.

Autentication for Power BI

The autentication for Power BI can be done in 2 ways 

1. Master Account : This basically username and password autentication along you need to reg an application in AAD to access Powerbi API's
2. SPN (Service Principal): This is basically reg you application in AAD and add your app in powerbi portal. Read more 

Kindly do remember that there are many difference between master account vs service principal autentication

Using Power BI Action Extension (Master Account Autentication)

As a first step you need to install this extension from marketplace (https://marketplace.visualstudio.com/items?itemName=maikvandergaag.maikvandergaag-power-bi-actions). Using Power BI Action Extension we can easily do the operations like below from Azure Pipeline, 

1. Create Workspace
2. Delete Workspace
3. Add a new Admin User to worksapce
4. Refresh a Dataset
5. Update DataSource Connection
6. Upload Power BI Report

As of now this extension is using the legacy approach of autentication using username and password, if you want to use the Service Principal you need to use your own powershell scripting which you can see in the next section.

For this demo we are planning to implement the CD for 2 Environments 1. Dev and 2. PROD

DEV Environment

After adding the Power BI Action Task into our Release Task, we need to configure the Power BI Service connection, as this is using master account approach, we need to provide username,password and clientid

PROD Environment

For the Production Environment we need to deploy the reports as well as need to change the datasources.

Deploy Reports





Update SQL DataSource 



Update Odata DataSource



With this you can able to deploy the reports and update the datasource, but the restriction is you couldn't able to update the credentials for the updated datasource

Current limitation of this extension: (As of 18 Aug 2019)

The following features are not yet supported by this extension, but we can expect this to include in near future.

• SPN Autentication
• Parameter updation
• DataSet TakeOver
• Update Credentials for the datasource

Using PowerShell Scripting in Azure DevOps (SPN Autentication)

Using Powershell scripts we can do all the possible deployments and break the limitations of the Power BI Action Extension.

Here we are not going to see all the possbile actions from Power BI instead we will see how we can use Powershell script to overcome the limitations of the above extension

Install the Required Modules

TakeOver DataSet

Update Datasource Credentials for a DataSet

Update Parameters for a DataSet

Variables Used from Azure DevOps

Download Full Script

You can download the full script from https://gallery.technet.microsoft.com/AzureDevOps-CICD-for-fefd58b8

[Anonymous]

Hi @Jayendran,

We are using Paginated reports (.rdl) in our project. Can we create CICD using azure devops for .RDL files as well? Or it is just supported for .PBIX files?

Thanks,

Sumit

[Jayendran]

Hi @Anonymous 

Yes it is possible see the below reference

https://dev.to/kenakamu/upload-rdl-file-s-to-power-bi-via-api-28h

[cherven]

Hi @Jayendran,

Do you mind sharing YAMLs for release pipelines?

Thank you.

[Anonymous]

this link is not work

You can download the full script from https://gallery.technet.microsoft.com/AzureDevOps-CICD-for-fefd58b8

Can you check again and provide new link

Thank you

[Jayendran]

@Anonymous  here you go 

https://github.com/jayendranarumugam/PowerBI-CICD

[shyammayhs]

Really helpful post.

One question I have is around the Gateway setup when connected to a cloud data source. How do I go around setting this up? Any information would be great!!

[Jayendran]

@shyammayhs For the master account/user account approach for CICD, the gateway can be bind to the dataset after the deployment by calling bind-to-gateway-in-group

[shyammayhs]

@Jayendranthanks. Would you have a tutorial to how I can setup a Gateway connection so all my Cloud connections go through the Gateway?

Thanks.

[renaudmontes]

I struggled initially but this article helped me out:

Automate your Power BI reports deployment using Azure DevOps | by Ivan Porta | The Startup | Medium

You do not need to see the video, and the video itself is low-res so is really hard to follow anyhow. 

I used Master account instead of Service Principal.

[Ramana_Ch]

Hi @Jayendran ,

This is very helpful as I am currently working on the Automation deployment activity of Power BI reports. I needed help from you, as you mentioned we can do "Using PowerShell Scripting in Azure DevOps (SPN Authentication)" also, but you created the tasks for Install-Module, Take over, Update Data source Credentials and Update Parameters for Dataset only. But can't we publish the Power BI report to Workspace in Power BI Service using PowerShell scripting. If yes, can you please help me out with how to do that and any code if you have for the same?

[mjc543]

---

@Jayendran wrote:

Best Practice

• If you using Cloud Data sources you don't need a gateway, but for on-premises data sources, we should use the gateway. Another Advantage of using the gateway for cloud data sources is your credentials were encrypted and stored in the gateway server and not with Power BI

---

What do you mean by "another" advantage - what is the first one? In the previous sentence you said a gateway for connecting to cloud sources is unnecessary...

[shivamshukla123]

Hi Jayendran,

We have a requirement to update the Impala ODBC data connections, but in drop down, the option is not present, could any one suggest workaround of the same?

[Anonymous]

Hi @Jayendran ,

Myself Mahendra. working as a DevOps Engineer in UK. currently automating BI tools like PowerBI in my company.

First of all thank you for the wonderful and insightful article which i thoroughly enjoyed and it helped alot.

in my case

My build pipeline CI has successfully ran and generated the artefact. 

For Release pipeline CD i have used SPN mechanism. i have created a new service connection using tenantID and clientID as certificate secret is optional. Im facing the following issue. please assist me as it giving me a lot of headache

2022-05-13T18:37:56.9921128Z ##[error]Cannot bind argument to parameter 'CertificateThumbprint' because it is an empty string.

i have used "Upload PowerBI" option and trying to place a "Test.pbix" in my workspace.

ps: my workspace name having spaces like (MY TEST WORKSPACE). is that could be the reason? im jus wondering.

[DileepRajam]

@Jayendran do you have new link for Script. the link ( https://gallery.technet.microsoft.com/AzureDevOps-CICD-for-fefd58b8) you provided is not working

[Jayendran]

@DileepRajam I couldn't able to edit the article. Please find the script at my github repo https://github.com/jayendranarumugam/PowerBI-CICD