Forum Discussion
The Source has too many files
Hi experts, we ingesting the data through API , for some data sources the business has shared nearly 150 files for one source, the schema is different sparing one or two similar columns , by going th...
anilgavhane
Super User
Super UserI would recommend avoiding the creation of 150 separate Lakehouse tables without first checking whether the files represent different business entities or variations of the same entity.
Recommended approach:
- Bronze layer: Store the raw API responses or files with source metadata, such as source name, file name, ingestion timestamp, and schema version.
- Schema standardization: If the files represent the same business entity with different columns, use a superset schema with nullable columns for fields that are missing in some files.
- Separate tables when necessary: If the files represent different business entities, keep them separate rather than forcing them into one wide table.
- Silver layer: Standardize column names, data types, and structures, and consolidate compatible datasets into curated tables.
My recommendation: Keep Bronze flexible for raw ingestion and perform schema harmonization in Silver. Avoid creating one massive table containing unrelated fields just to reduce the table count.
AI assistance was used to format this response.