Forum Discussion

TK12345's avatar
TK12345
Icon for Resolver II rankResolver II
1 month ago
Solved

OAP for MLVs cross workspace

Hi all,

I have a question regarding OAP implementation across multiple workspaces.

My current setup consists of three layers:

Bronze: OAP enabled

Silver: OAP enabled

Gold: OAP not enabled

In the Bronze layer, I extract all source data using specific endpoints and gateways, with the required connection rules configured. In the Silver layer, I use shortcuts to access the source lakehouses in Bronze. This works as expected.

Within Silver, I then create Materialized Lake Views (MLVs) for my entities. For example, I create a Person entity based on multiple source tables and store the result in LH_Silver. This also works correctly.

The issue occurs in the next step. On top of the Silver entity, I want to create dim_Person. The MLV reads the entity from LH_Silver, but the output should be written to LH_Gold in the Gold workspace.

When OAP is disabled on all workspaces, this cross-workspace setup works and I can successfully write the MLV to Gold. However, as soon as OAP is enabled on the Silver workspace, I can no longer write the MLV to the Gold workspace.

I've already tried several connection rules, including:

FabricMaterializedLakeView

FabricSqlEndpointMetadata

Fabric SQL Analytics Endpoint

Lakehouse

Notebook

Web V2

I've also configured an MPE for the Gold (3000) workspace, but so far none of these options have resolved the issue.

Does anyone know which connection rule, MPE, or configuration is required to allow an MLV running from an OAP-enabled Silver workspace to write to a Lakehouse in another workspace?

Ideally, I would like to keep the notebooks and MLV logic in the Silver layer rather than moving them to the Gold workspace.

Thanks in advance!

Cel 1:

%%pyspark

spark.conf.set("spark.sql.caseSensitive", "true")

workspace = spark.conf.get("trident.workspace.name")

environment = "DEV" if "DEV" in workspace else "TEST" if "TEST" in workspace else "PRD"

spark.sql(f"SET TargetEnvironment = 3000_CURATED ({environment})")

 

Cel 2:

CREATE OR REPLACE MATERIALIZED LAKE VIEW `${TargetEnvironment}`.LH_3000.Algemeen.dim_Persoon AS

SELECT *

FROM test.`02_entity`.Person

5 Replies