Code organization for Bronze layer

Hi reddyr2502​ ,

Your ER model is a good starting point for a metadata-driven Bronze ingestion framework. I would keep the design simple by separating orchestration, reusable ingestion logic, configuration, and application-specific exceptions.

A practical structure could be:

I would use the metadata tables for operational configuration such as source system, connection, source object, load type, watermark, schedule, active status and run history.

Use JSON/configuration files for more complex or variable rules, particularly JSON paths, flattening/explode rules, schema mappings, API pagination options and application-specific parameters. This avoids making the metadata tables increasingly wide as new source types are added.

The execution pattern can then remain simple:

For standard source movement, I would use Fabric Pipeline/Copy Activity and supported Fabric connectors where possible. Notebooks are better reserved for processing that actually requires code, such as complex/nested JSON handling or source-specific logic.

For unusual applications, keep a Custom area as an escape hatch rather than trying to make the generic framework support every possible scenario.

I would also keep actual credentials out of the metadata/config files. Store only the appropriate connection/secret reference and resolve credentials securely at runtime.

The main design principle I would follow is:

Adding a new source should normally require metadata and configuration changes, not a new pipeline or notebook.

Application-specific code should be the exception. That keeps the framework scalable while still being practical for a small team to operate.

If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.

If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!