Forum Discussion

reddyr2502's avatar
reddyr2502
New Member
1 day ago

Code organization for Bronze layer

Hi,

I am the only person managing data analytics at a startup. We use a single Microsoft Fabric workspace for our Bronze, Silver, and Gold data layers.

For the Bronze layer, we ingest data from more than 10 source applications using a metadata-driven framework. Each application has different connection requirements and may return data in formats such as CSV, simple JSON, or deeply nested JSON.

What is the best way to organize the following components?

  • Reusable ingestion code
  • Application-specific connection configurations
  • Metadata tables and configuration files
  • Source tables and extraction settings
  • File-format-specific processing
  • JSON flattening rules for both simple and nested structures
  • Pipelines, notebooks, and supporting utilities

I would like the framework to use metadata and configuration files wherever possible, while still supporting application-specific logic when required. Please review the ER diagram for metadata driven tables. Could you recommend a simple, scalable folder structure and an overall design for organizing the Bronze-layer ingestion framework in Microsoft Fabric?

 

data engineering
Environment
Notebook

2 Replies

  • gslick's avatar
    gslick
    Frequent Visitor
    20 hours ago

    Just looking at the design, could you not use KeyVault instead of putting usernames and passwords in a table?

  • ssrithar's avatar
    ssrithar
    Super User
    17 hours ago

    Hi reddyr2502​ ,

    Your ER model is a good starting point for a metadata-driven Bronze ingestion framework. I would keep the design simple by separating orchestration, reusable ingestion logic, configuration, and application-specific exceptions.

    A practical structure could be:

     

    I would use the metadata tables for operational configuration such as source system, connection, source object, load type, watermark, schedule, active status and run history.

    Use JSON/configuration files for more complex or variable rules, particularly JSON paths, flattening/explode rules, schema mappings, API pagination options and application-specific parameters. This avoids making the metadata tables increasingly wide as new source types are added.

    The execution pattern can then remain simple:

    For standard source movement, I would use Fabric Pipeline/Copy Activity and supported Fabric connectors where possible. Notebooks are better reserved for processing that actually requires code, such as complex/nested JSON handling or source-specific logic.

    For unusual applications, keep a Custom area as an escape hatch rather than trying to make the generic framework support every possible scenario.

    I would also keep actual credentials out of the metadata/config files. Store only the appropriate connection/secret reference and resolve credentials securely at runtime.

    The main design principle I would follow is:

    Adding a new source should normally require metadata and configuration changes, not a new pipeline or notebook.

    Application-specific code should be the exception. That keeps the framework scalable while still being practical for a small team to operate.

     

    If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.

    If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!