Delta table in Bronze or Overkill?

[Amir_m_h]

Hi,

I wanted to get some opinion on whether I should create a delta table for Bronze layer?

• We can extract data and land it in our Lakehouse file section. If we think of a copy activity then most probably we will have tabular parsed data. In such cases, loading it into delta table is practically the same with added benefit of delta features.

• However, we do a lot of API extractions and those are landed as files with JSON format. Here, it makes sense to explode those JSON arrays, convert it to string to keep it resembling data as-is, and then load it into delta table.
  Also, having delta table with change data feed enabled helps in doing incremental reload in all layers (although it can be done with timestamp field as well).
  Another benefit would be that, raw parquet files could be moved to some cold storage ADLS GEN 2 for audit.
  In this case, we are ending up with Landing > Bronze > Silver > Gold.

Would be good to hear some opinions on how others have done it and if you recommend a differnet appoach?

Thanks

[v-mdharahman · Accepted solution]

Hi @Amir_m_h,

Thanks for reaching out to the Microsoft fabric community forum and for sharing your thoughtful breakdown of the Lakehouse ingestion strategy. You’ve already outlined a solid rationale for using Delta tables in the Bronze layer, especially considering the mix of tabular and semi-structured data sources. Regarding delta tables in bronze layer, it’s generally a good practice to use Delta tables even in the Bronze layer. This ensures consistency and unlocks features like schema enforcement, time travel, and change data feed (CDF), which are valuable for auditability and incremental processing.

For handling JSON from APIs, your strategy to explode JSON arrays and convert them to strings before loading into Delta is sound. It preserves the raw structure while making the data queryable. Just ensure that the transformation logic is well-documented and version-controlled and fpr Incremental Loads while timestamp-based filtering works, enabling CDF on Delta tables gives you more flexibility and precision, especially when dealing with late-arriving data or schema changes.

As for cold storage strategy, archiving raw Parquet files to ADLS Gen2 for audit is a smart move. Just make sure metadata and lineage are tracked so you can trace back transformations if needed.

In summary, your approach is aligned with best practices. Using Delta tables in the Bronze layer adds value and future-proofs your architecture. If you’re using Microsoft Fabric, also consider leveraging shortcuts and role-based access controls for better governance.

If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

Best Regards,
Hammad.

[v-mdharahman · Accepted solution]

Hi @Amir_m_h,

Thanks for reaching out to the Microsoft fabric community forum and for sharing your thoughtful breakdown of the Lakehouse ingestion strategy. You’ve already outlined a solid rationale for using Delta tables in the Bronze layer, especially considering the mix of tabular and semi-structured data sources. Regarding delta tables in bronze layer, it’s generally a good practice to use Delta tables even in the Bronze layer. This ensures consistency and unlocks features like schema enforcement, time travel, and change data feed (CDF), which are valuable for auditability and incremental processing.

For handling JSON from APIs, your strategy to explode JSON arrays and convert them to strings before loading into Delta is sound. It preserves the raw structure while making the data queryable. Just ensure that the transformation logic is well-documented and version-controlled and fpr Incremental Loads while timestamp-based filtering works, enabling CDF on Delta tables gives you more flexibility and precision, especially when dealing with late-arriving data or schema changes.

As for cold storage strategy, archiving raw Parquet files to ADLS Gen2 for audit is a smart move. Just make sure metadata and lineage are tracked so you can trace back transformations if needed.

In summary, your approach is aligned with best practices. Using Delta tables in the Bronze layer adds value and future-proofs your architecture. If you’re using Microsoft Fabric, also consider leveraging shortcuts and role-based access controls for better governance.

If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

Best Regards,
Hammad.

[v-mdharahman · Accepted solution]

Hi @Amir_m_h,

As we haven’t heard back from you, so just following up to our previous message. I'd like to confirm if you've successfully resolved this issue or if you need further help.

If yes, you are welcome to share your workaround so that other users can benefit as well.  And if you're still looking for guidance, feel free to give us an update, we’re here for you.

Best Regards,

Hammad.