Forum Discussion
Extracting data from External API
- 4 months ago
Hi SaumyaMathai ,
For most Fabric medallion implementations, the recommended approach is usually closest to Option 2.
External APIs are typically ingested into the same Bronze layer as raw data, since Bronze is designed to store source data in its original form regardless of where it comes from (SQL, APIs, files, etc.). From there, the data can follow the same Bronze > Silver > Gold transformation pattern as the rest of your pipeline.
To keep things maintainable when you have many APIs, a common practice is to use separate ingestion pipelines per source system, but land the raw outputs into the shared Bronze layer often partitioned or organized by source system. This avoids maintaining multiple medallion architectures while still keeping ingestion logic modular.
Creating an additional Platinum layer is generally not necessary unless you have a very specific consumption or governance requirement beyond the standard Gold layer.
Overall, a single medallion architecture with source-specific ingestion pipelines feeding Bronze tends to scale well and keeps the architecture simpler to manage.
Hope this helps.
Thank you. - 4 months ago
Hello SaumyaMathai,
Ingest external API data into your existing Bronze → Silver → Gold pipeline. Store raw API responses in Bronze, normalize and clean in Silver, and integrate with internal SQL data in Gold. Use incremental loads, error handling, and schema tracking to keep pipelines scalable and maintainable.
Refs:
Hello SaumyaMathai,
Ingest external API data into your existing Bronze → Silver → Gold pipeline. Store raw API responses in Bronze, normalize and clean in Silver, and integrate with internal SQL data in Gold. Use incremental loads, error handling, and schema tracking to keep pipelines scalable and maintainable.
Refs:
Thank you so much