How do Dev/Test/Prod workspaces fit into Fabric architecture?

[arpost]

Greetings, community. I'm continuing to explore Fabric and identify how to use the different components especially in relation to traditional ALM with Dev/Test/Production workspaces. Microsoft has talked a lot about the medallion architecture along with the benefits of OneLake and OneCopy, but I'm trying to figure out how this all aligns in terms of the ALM process in Fabric/Power BI. 

Traditionally, you'd have at least three workspaces (Dev, Test, Production) with each workspace containing a version of an artifact. Production is obviously the final-state version of the artifact for consumption. You'd use a deployment pipeline to push artifacts from one stage to the next.

From a medallion-style approach that uses data warehouses, you'd:

1. Load data to a Lakehouse (Bronze).
2. Load and transform data in a Data Warehouse (Silver).
3. Either persist the data in the Data Warehouse or load it to a Lakehouse (Gold).

In the world of Fabric, is the idea that all of those artifacts would live in a single workspace? So you are running everything through to from Bronze to Gold in that same workspace?

If yes, then does that mean you have to re-run everything through in Test and again in Production, which creates at least three "copies" of data between the Dev, Test, and Production workspaces? Or am I missing something?

[MikeH_SDE]

Consider Dev/Test/Prod and Bronze/Silver/Gold as separate things. B/S/G represents transformations done on your data, whilst D/T/P are stages in safely making changes to those transformations and bringing new ones into existence. 
There is going to be a degree of replication of data across all these areas. We will only replicate across D/T/P what we have responsibility to create, any data set that we only read we access the live/production data for Dev & Test. We will try to minimise the replication of our own data sets in Dev & Test (how to best to this is something we are currently investigating - poss ADF only what we need to Dev & Test)

[nishamarshalll]

Hello

The Dev/Test/Prod workspaces in Fabric architecture are essential to the ALM process. Fabric's medallion-style approach has the potential to combine several artifacts into a single workspace, whereas conventionally, each workspace (Dev, Test, and Production) houses a version of each artifact.

In this case, the data would move within the same workspace from Bronze to Gold. This does not, however, imply that everything in Test and Production must be redone. Because of Fabric's architecture, redundant processes can be avoided and data pipelines may be managed effectively.

Regards

Nisha Marshall

Prestige Camden Gardens

[arpost]

Thank you both for your replies, @Anonymous and @R1k91. That's very helpful. Two additional questions:

1. Am I right in thinking there will still be duplication when doing things like inserts into data warehouses?
2. If an organization is wanting to have a centralized data repository but then pull that data into Dev/Test/Prod workflows, would the idea be that you have a "Data Repository" (perhaps a separate lakehouse) and then shortcut that into each workspace?

Trying to figure out how to work toward that OneCopy vision and ensure we don't end up just storing double/triple/quadruple the exact same data across numerous workspaces.

[R1k91]

1. yes and even creating new files in lakehouses. we could say that each line could be "duplicated" as many layers you have in your architecture. in reailty you need to add layers since your data isn't ready for users consumption yet therefore it's not exactly duplicated and equal to the previous layer. in our projects we tipically hide everything with the exception of "gold/final" layer to final users, therefore it's very close to "one copy". by the way it's pratical vs theory.
2. in my opinion every environment "DEV/TEST/UAT, PROD" should be considered as an atomic and isolated ecosystem with itsown artifacts (lakehouses, warehouses..) and permissions that doesn't communicate with other environments. you're going to duplicate data among different environments but if you are in "test" you shouldn't see what is "prod". again practical vs theory.

[Anonymous]

Hi @arpost ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .

Thanks

[Anonymous]

Hi @arpost ,

Thanks for using Fabric Community.

This is an architecture question and the answer is "it depends" on requirements. Choice of architecture/layout of workspaces could be driven by cost, performance, org structure, isolation/governance boundary requirements etc. 

From product perspective, Fabric is flexible and building blocks (workspaces and items) can be laid out in different ways to meet business requirements. Common patterns which we have seen thus far:

Pattern 1: Separate workspace and separate Fabric Items (WH, LH) for Bronze, Silver and Gold layers.

From ALM perspective, in the current release of Fabric, you would use separate deployment pipelines for Bronze, Silver and Gold layers which are each represented by a separate workspace. 

Bronze Deployment pipeline: [WS <dev-bronze>] -> [WS <test-bronze>] -> [WS <prod-bronze>]

Silver Deployment pipeline: [WS <dev-silver>] -> [WS <test-silver>] -> [WS <prod-silver>]

Gold Deployment pipeline: [WS <dev-gold>] -> [WS <test-gold>] -> [WS <prod-gold>]

Pattern 2: Single workspace with separate LH or WH for Bronze, Silver and Gold layers.

Further reading which I would recommend to help shape your decision:

Implement medallion lakehouse architecture in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

Microsoft Fabric adoption roadmap - Power BI | Microsoft Learn

Deployment patterns for Microsoft Fabric - Azure Architecture Center | Microsoft Learn

Overview of Fabric deployment pipelines - Microsoft Fabric | Microsoft Learn

Hope this is helpful.

[Anonymous]

Hi @arpost ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .

Thanks

[R1k91]

IMHO Bronze/Silver/Gold is a transformation pattern and it's applied inside a project.

a project could be in dev, test or production environments. therefore yes you're going to copy data in multiple environments (but you could decide to have subset of data in dev and test for example).

we put all the layers inside one workspace and then we'll create a workspace per environment.

you could split layers in different workspaces but you'll need to create workspaces per environment also.

3 wrksp dev (gold siver and bronze) 3 wrksp test 3 wrksp prod = 9 wkrsp that is much more complicated to manage even from a deployment perspective.

having different environments with all the layers will duplicate data across workspaces but "one copy" principle will be true for users that should only see the production one.

moreover you can say "one copy" principle is true "at environment level" because in test environment there'll be only one gold layer, as one silver and so on.