how to manage reporting lifecycle and avoid cross-workspace sourcing

[mr_wizard]

I need help determining a best practice for my orgnaization's reporting lifecycle.  My organization manages its reporting lifecycle via workspaces in a single instance of Power BI (Premium capacity).  That is, we have separate workspaces for Development, Test, and Production.  My organization also separates semantic models from reports.  That is, all reports are in Live mode against some semantic model.  Our philosophy for lifecycle is that every successive "layer" in our data platform sources from the Production environment (not a non-Production environment) of the previous layer (e.g., the semantic layer points to the Production environment of the data warehouse).  So, with that philosophy, all reports source from semantic models in Production workspaces, not Development or Test workspaces.

We don't want cross-workspace sourcing (e.g., we don't want a report in workspace A sourcing from a semantic model in workspace B).  However, the combination of report-model separation and lifecycle-via-workspaces seems to force us into cross-workspace sourcing.  Is there a way to avoid this?  Or, is there a best practice around this?

For example, suppose report A sources from semantic model X.  Because we want all reports to source from Production (not non-Production), then report A sources from semantic model X in a Production workspace (as opposed to, say, the instance of semantic model X in the Development or Test workspaces).  So, when report A moves through its lifecycle (i.e., it moves through Development, Test, and Production), the instance of report A in the Development and Test workspaces violates our cross-workspace rule (because the instance of report A in those workspaces will point against a Production workspace).  How can we avoid this?  Or, is there a better pattern for managing the lifecycle?  It's almost like we need separate Power BI instances to avoid the cross-workspace sourcing problem.

[nilendraFabric · Accepted solution]

Hello @mr_wizard 

have you tried deployment pipelines 

Deployment Pipelines allow you to manage content across Development, Test, and Production environments systematically. With deployment pipelines:
• Semantic models and reports can be deployed together across stages.
• Thin reports (live-connected to datasets) will automatically connect to the appropriate dataset as they move through pipeline stages, ensuring consistency without cross-workspace sourcing

[mr_wizard]

Yes.  However, I don't understand how deployment pipelines resolve the problem.  Both the semantic models and the reports need to follow a lifecycle (Development, Test, Production).  As a thin report moves through each stage (workspace) of the lifecycle, to not encounter cross-workspace sourcing, its semantic model needs to be "local" (i.e., in the same workspace as the report).  But, because of our rule that we always point downstream layers to the Production "environment" of previous layers, that forces the semantic model in each stage (workspace) of the report layer to be the Production version.  Whether we deploy manually or in a pseudo-automated way (with deployment pipelines), I don't see any way to avoid cross-workspace sourcing, unless we do something like deploy the Production version of each semantic model four times (first to the Production workspace of the semantic model, then again to the Development workspace of the report, then again to the Test workspace of the report, and lastly to the Production workspace of the report).  What am I missing?

[Tutu_in_YYC]

It sounds like you do not have lifecycle for semantic model, only for report, since all dev/test/prod report will be connecting to the final version of the semantic model.

In this case, as you mentioned, you may have to have the same final version of the semantic model in 3 different workspace within the pipeline. You can still use the deployment pipeline to manage the lifecycle of the report, ie only deploy report through dev > test > prod. The semantic model remain static. As the report is being deployed, it will automatically rebind to the next semantic model in the next environment. 

If any changes is made to the semantic model, it has to be published separately to all 3 workspaces dev/test/prod ; or published the final version to dev, and deploy to test and prod right away.

This way you still have 3 versions of report, and same semantic model in all environment with no cross-workspace

[Tutu_in_YYC]

If you use deployment pipeline, you will be able to have a proper promotion management, with tracking, ie who did it, when and even add notes of the additional features have been added to the prod version. And also you will be able to do some comparisons between versions of different environment.

If all above is not needed, you can just work in 1 workspace, create folders for the different "environments". Not a proper solution, but it works.. You will always publish into the same workspace for "dev/prod/test"

[mr_wizard]

We definitely need (and use) deployment pipelines for both the semantic models and the reports (for many of the reasons that you listed).  In order to enforce the no-cross-workspace-sourcing rule while also enforcing the rule that every report in every stage point to Production, I think it's unavoidable to have 6 workspaces and 2 deployment pipelines: 3 workspaces and 1 pipeline for the semantic model lifecycle, and another 3 workspaces and 1 pipeline for the report lifecycle (where the "semantic model" pipeline deploys to the 3 "report" workspaces after it deploys into the Production "semantic model" workspace).

[Tutu_in_YYC]

The challenge i see here is: how are you going deploy the semantic model from deployment pipeline for Semantic model into deployment pipeline for Report?

Is this what you are thinking?

[mr_wizard]

For the semantic model deployment pipeline, I would assign 6 workspaces, where the last 3 are for the report lifecycle.  So, the semantic model deployment pipeline would look something like:

Dev Model -> Test Model -> Prod Model -> Dev Report -> Test Report -> Prod Report

I don't think deployment pipelines support parallel deployment.  Otherwise, after deployment into workspace Prod Model, I would run parallel deployments into workspaces Dev Report, Test Report, and Prod Report, since the path after Prod Model doesn't need to be linear.

The deployment pipeline for reports would simply be:

Dev Report -> Test Report -> Prod Report

This way, I can uncouple the model lifecycle from the report lifecycle (much like how I uncouple the warehouse lifecycle from the model lifecycle).

[Tutu_in_YYC]

I see. Good luck, let us know if you encounter challenges.

[Tutu_in_YYC]

 @nilendraFabric is correct. Deployment Pipeline is the first recommended step by Microsoft when it comes to lifecycle. Which as your organization matures, then you can proceed to implement other components of CICD. The following link will tell you more about cicd and deployment pipeline. And eventually you can also add a backup repo with git.

https://learn.microsoft.com/en-us/fabric/cicd/