Forum Discussion

reddyr2502's avatar
reddyr2502
Regular Visitor
22 days ago
Solved

Code organization for Bronze layer

Hi, I am the only person managing data analytics at a startup. We use a single Microsoft Fabric workspace for our Bronze, Silver, and Gold data layers. For the Bronze layer, we ingest data from mor...
data engineering
environment
notebook
  • ssrithar's avatar
    ssrithar
    21 days ago

    Hi reddyr2502​ ,

    Your ER model is a good starting point for a metadata-driven Bronze ingestion framework. I would keep the design simple by separating orchestration, reusable ingestion logic, configuration, and application-specific exceptions.

    A practical structure could be:

     

    I would use the metadata tables for operational configuration such as source system, connection, source object, load type, watermark, schedule, active status and run history.

    Use JSON/configuration files for more complex or variable rules, particularly JSON paths, flattening/explode rules, schema mappings, API pagination options and application-specific parameters. This avoids making the metadata tables increasingly wide as new source types are added.

    The execution pattern can then remain simple:

    For standard source movement, I would use Fabric Pipeline/Copy Activity and supported Fabric connectors where possible. Notebooks are better reserved for processing that actually requires code, such as complex/nested JSON handling or source-specific logic.

    For unusual applications, keep a Custom area as an escape hatch rather than trying to make the generic framework support every possible scenario.

    I would also keep actual credentials out of the metadata/config files. Store only the appropriate connection/secret reference and resolve credentials securely at runtime.

    The main design principle I would follow is:

    Adding a new source should normally require metadata and configuration changes, not a new pipeline or notebook.

    Application-specific code should be the exception. That keeps the framework scalable while still being practical for a small team to operate.

     

    If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.

    If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!

     

jayaprakash2000's avatar
jayaprakash2000
New Member
17 days ago

Hi, I went through the ER diagram. The overall design looks good for a metadata-driven Bronze ingestion framework.

Since you are handling 10+ source applications, I would avoid creating separate pipelines and notebooks for each source.

I would keep one master pipeline and a few reusable notebooks for API, JDBC, files, and JSON processing.

I would suggest adding a separate SourceEntity table to keep the source tables/files/API objects separate from the connection details.

For JSON, I would keep the flattening rules in a separate JSONMapping table, especially for deeply nested JSON.

The pipeline can pass SourceID and EntityID to the notebook, and the notebook can get the required configuration from the metadata tables.

I would also keep watermark and last-run information separately from the main configuration, using something like an EntityState table.

The existing PipelineRunLog can be used for run status, row counts, errors, start/end time, and watermark details.

For application-specific cases such as unusual authentication, pagination, or JSON structures, I would keep small custom functions instead of adding too many conditions to the common notebook.

For the workspace, I would keep separate folders for 01_Bronze, 02_Silver, and 03_Gold, even if they are currently in the same workspace.

Overall, I would follow a simple approach: metadata for common configuration, reusable notebooks for common processing, and custom code only where a source really needs it.