Hi @rahul_rajesh
Yes, Azure Data Factory does support incremental data loading from an API.
Steps to Implement Incremental Data Loading from an API
Create a Data Factory Instance:Start by creating a new Azure Data Factory instance in the Azure portal.
Set Up a Pipeline:Create a new pipeline within your Data Factory.
Use Web Activity:
Add a Web Activity to your pipeline. This activity will call the REST API.
Pass parameters to the API, such as the last modified date or a watermark, which will help you fetch only the new or updated records since the last load.
Implement Incremental Logic:
You can use an Until Activity to loop through the API responses until all data is loaded. This is useful if the API returns data in pages.
Store Data Temporarily:
Consider loading the data into a temporary or staging table first. This allows you to perform any necessary transformations or checks before merging it into your main table.
Merge Data:
Finally, use a Copy Activity or a stored procedure to merge the new data from the staging table into your main table based on your incremental logic.
How to implement an incremental data load using ADF from a REST API Source - Microsoft Q&A
For questions about the ADF, we suggest you go to the following link, there will be more professional staff to help you:
Azure Data Factory | Microsoft Community Hub
If you have any questions about Fabric, please continue to use this forum to ask!
Regards,
Nono Chen
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
