topic Re: Single Version of Truth Mapping for Multiple Data Using a Single Notebook in Data Engineering

Single Version of Truth Mapping for Multiple Data Using a Single Notebook

Mirdula — Wed, 23 Apr 2025 11:41:47 GMT

Hello everyone,

My team is in the process of implementing a Medallion architecture in Microsoft Fabric, ingesting data from multiple ERP systems. One of these systems will serve as the “master,” and all dimensions must align with a Single Version of Truth (SVOT) based on its schema.

So far, we have ingested and transformed data from one source system, which includes both Finance and Sales data. Although the core SVOT mapping logic is nearly identical across functions, there are a handful of dimensions that require function-specific handling.

Today, we maintain separate PySpark notebooks for each dataset (Finance, Sales, etc.) to apply the SVOT mapping. We’d like to consolidate this into a single, parameterized PySpark notebook that can handle all functional datasets while still accommodating their slight dimension differences.

Does anyone have recommendations on how we could design and orchestrate one reusable SVOT mapping notebook in Microsoft Fabric?

Re: Single Version of Truth Mapping for Multiple Data Using a Single Notebook

nilendraFabric — Wed, 23 Apr 2025 15:23:44 GMT

Hello Mirdula

not official MS recommendations, but these are few which I try to follow in my deployments

Centralize SVOT Logic: Keep your mapping logic modular and in shared modules to avoid duplication.
• Parameterize Notebooks: Use parameters for dataset type, source/target paths, and configuration settings.
• Handle Function-Specific Needs: Apply conditional logic or configuration-driven mappings for each functional dataset (e.g., Finance, Sales).
• Orchestrate with Pipelines: Use Microsoft Fabric pipelines or notebook widgets to pass parameters and manage execution.
• Follow Medallion Architecture: Structure your data into Bronze (raw), Silver (SVOT-mapped), and Gold (curated) layers.
• Prioritize Maintainability: Use the DRY (Don’t Repeat Yourself) principle so updates are easy and consistent across datasets.

Re: Single Version of Truth Mapping for Multiple Data Using a Single Notebook

Mirdula — Thu, 24 Apr 2025 06:55:47 GMT

Thank you for your response nilendraFabric 🙂 . It helps 👍