Extracting data from External API

[SaumyaMathai]

Hi everyone,

I am new to this community and also fairly new to Microsoft Fabric.

I work for a Multi Academy Trust, and we recently implemented Fabric using a Medallion Architecture with our core data coming from a SQL database.

Now we need to bring in external API data (for example: Teaching systems, HR systems, Finance systems) into Fabric so that we can build live dashboards.

I am trying to understand the best architectural approach to ingest and manage these APIs within our existing medallion setup. From my current understanding, there seem to be a few possible options:

Option 1:
Create separate pipelines and a separate medallion architecture for each API, and then integrate the results into the main architecture (SQL-based).
Concern: This could lead to multiple pipelines and higher maintenance.

Option 2:
Ingest the API data directly into the same Bronze layer where our SQL data lands, and process it through the same Bronze → Silver → Gold pipeline.
Benefit: Only one pipeline to maintain.

Option 3:
Create an additional Platinum layer for external APIs, and then connect that layer to the Gold layer of the main architecture.

Has anyone implemented something similar in Fabric?
What would be the recommended pattern for ingesting multiple APIs while keeping the architecture scalable and maintainable?

Any guidance or best practices would be greatly appreciated.

Thanks in advance.

Saumya

[v-echaithra · Accepted solution]

Hi @SaumyaMathai ,

For most Fabric medallion implementations, the recommended approach is usually closest to Option 2.

External APIs are typically ingested into the same Bronze layer as raw data, since Bronze is designed to store source data in its original form regardless of where it comes from (SQL, APIs, files, etc.). From there, the data can follow the same Bronze > Silver > Gold transformation pattern as the rest of your pipeline.
To keep things maintainable when you have many APIs, a common practice is to use separate ingestion pipelines per source system, but land the raw outputs into the shared Bronze layer often partitioned or organized by source system. This avoids maintaining multiple medallion architectures while still keeping ingestion logic modular.
Creating an additional Platinum layer is generally not necessary unless you have a very specific consumption or governance requirement beyond the standard Gold layer.
Overall, a single medallion architecture with source-specific ingestion pipelines feeding Bronze tends to scale well and keeps the architecture simpler to manage.

Hope this helps.
Thank you.

[v-echaithra]

Hi @SaumyaMathai ,

Thank you @Olufemi7 , @deborshi_nag  for your inputs.

We’d like to follow up regarding the recent concern. Kindly confirm whether the issue has been resolved, or if further assistance is still required. We are available to support you and are committed to helping you reach a resolution.

Thank you.

[SaumyaMathai]

Hi , I have got an idea on how to move forward now . Thanks so much for all your inputs and suggestions, they have been really helpful. 

Appreciate the support from this community.

[Olufemi7 · Accepted solution]

Hello @SaumyaMathai,

Ingest external API data into your existing Bronze → Silver → Gold pipeline. Store raw API responses in Bronze, normalize and clean in Silver, and integrate with internal SQL data in Gold. Use incremental loads, error handling, and schema tracking to keep pipelines scalable and maintainable.

Refs:

• Medallion Lakehouse Architecture

• Materialized Lake Views

[SaumyaMathai]

Thank you so much

[deborshi_nag]

Hello @SaumyaMathai 

First, assess whether the source systems (e.g. HR, Finance, ERP platforms) support native Microsoft Fabric Mirroring.

Where available, Mirroring should be the preferred ingestion mechanism to populate the Bronze layer, as it provides managed, incremental synchronisation with minimal engineering effort, preserves source fidelity, and reduces operational complexity.

If Mirroring is not supported, the next option is to implement Change Data Capture (CDC), either using Fabric Pipelines or approved external CDC tooling. CDC enables incremental extraction of inserts, updates, and deletes, which can be landed into a controlled landing area.

From this landing zone, transformation pipelines can be implemented to merge and apply changes in order to construct and maintain the Bronze layer in a consistent, auditable, and replayable manner.

[v-echaithra · Accepted solution]

Hi @SaumyaMathai ,

For most Fabric medallion implementations, the recommended approach is usually closest to Option 2.

External APIs are typically ingested into the same Bronze layer as raw data, since Bronze is designed to store source data in its original form regardless of where it comes from (SQL, APIs, files, etc.). From there, the data can follow the same Bronze > Silver > Gold transformation pattern as the rest of your pipeline.
To keep things maintainable when you have many APIs, a common practice is to use separate ingestion pipelines per source system, but land the raw outputs into the shared Bronze layer often partitioned or organized by source system. This avoids maintaining multiple medallion architectures while still keeping ingestion logic modular.
Creating an additional Platinum layer is generally not necessary unless you have a very specific consumption or governance requirement beyond the standard Gold layer.
Overall, a single medallion architecture with source-specific ingestion pipelines feeding Bronze tends to scale well and keeps the architecture simpler to manage.

Hope this helps.
Thank you.

[SaumyaMathai]

Thanks @v-echaithra . To add on if I need each API as a standalone, reusable data block for redistribution (to the community), then each API might need its own individual medallion architecture flow is self-contained per API.