Forum Discussion

SamyAbdul's avatar
SamyAbdul
Frequent Visitor
1 day ago

The Source has too many files

Hi experts, we ingesting the data through API , for some data sources the business has shared nearly 150 files for one source, the schema is different sparing one or two similar columns , by going th...
data engineering
data pipeline
lakehouse
Murtaza_Ghafoor's avatar
Murtaza_Ghafoor
Icon for Super User rankSuper User
22 hours ago

@SamyAbdul

Since you have many files, I suggest below approach:

  • Bronze Layer: Store API responses in their original format, organized by source.
  • Schema handling: Group files with similar schemas into the same table. Keep different entities separate.
  • Silver Layer: Standardize schemas, map common fields, and apply data quality rules.
  • Scalability: Use metadata-driven ingestion to manage multiple sources efficiently.

  • Gold Layer: Clean curated, business friendly tables for analytics and AI ready data.

Please avoid creating a table for every file or combining all possible columns into one wide table.

If this helps, ✓ Mark as Kudos | Help Other
AI Assisted Draft


SamyAbdul's avatar
SamyAbdul
Frequent Visitor
22 hours ago

Thank you Murtaza_Ghafoor​  The files I am talking about particularly because of an API issue being shared on SFTP location in Excel /CVC format.