How to do incremental refresh in data pipeline?

Hi Charlotte_chum 

Thank you for reaching out to the Microsoft Fabric Community Forum.
 

To implement a 24-hour incremental refresh in a Data Pipeline based on the last modified data, consider the below steps:

Make sure you have a source table with a LastModifiedTime column (datetime) to track changes, and a watermark table to store the last successfully processed timestamp.

Required a destination (e.g., Lakehouse or SQL DB) and access to Microsoft Fabric Data Factory to build and run the pipeline.

Create a pipeline parameter, e.g., LastRunTime, that will dynamically retrieve the last watermark value from watermark_table. Use the watermark in your SQL query to filter records modified in the last 24 hours.

Add a copy data activity in a pipeline, after successful data load, update watermark_table with the current UTC timestamp to record the latest successful run.

Use triggers in the Data Factory to schedule the pipeline to run every 24 hours.

For more detailed information, Please refer to the Microsoft official document:

Incrementally copy data from a source data store to a destination data store - Azure Data Factory | Microsoft Learn

If this response resolves your query, kindly mark it as Accepted Solution to help other community members. A Kudos is also appreciated if you found the response helpful.

Thank You!