Forum Discussion
How to do incremental refresh in data pipeline?
Hi all,
I would like to perform a 24-hour incremental refresh based on the last modified data. I found some details from online sources and Microsoft documentation, but I am unable to find specific information for data pipelines. Does anyone know how to do this?
Thank you for reaching out to the Microsoft Fabric Community Forum.
To implement a 24-hour incremental refresh in a Data Pipeline based on the last modified data, consider the below steps:
Make sure you have a source table with a LastModifiedTime column (datetime) to track changes, and a watermark table to store the last successfully processed timestamp.
Required a destination (e.g., Lakehouse or SQL DB) and access to Microsoft Fabric Data Factory to build and run the pipeline.
Create a pipeline parameter, e.g., LastRunTime, that will dynamically retrieve the last watermark value from watermark_table. Use the watermark in your SQL query to filter records modified in the last 24 hours.
Add a copy data activity in a pipeline, after successful data load, update watermark_table with the current UTC timestamp to record the latest successful run.
Use triggers in the Data Factory to schedule the pipeline to run every 24 hours.
For more detailed information, Please refer to the Microsoft official document:
If this response resolves your query, kindly mark it as Accepted Solution to help other community members. A Kudos is also appreciated if you found the response helpful.
Thank You!
5 Replies
Thank you for reaching out to the Microsoft Fabric Community Forum.
To implement a 24-hour incremental refresh in a Data Pipeline based on the last modified data, consider the below steps:
Make sure you have a source table with a LastModifiedTime column (datetime) to track changes, and a watermark table to store the last successfully processed timestamp.
Required a destination (e.g., Lakehouse or SQL DB) and access to Microsoft Fabric Data Factory to build and run the pipeline.
Create a pipeline parameter, e.g., LastRunTime, that will dynamically retrieve the last watermark value from watermark_table. Use the watermark in your SQL query to filter records modified in the last 24 hours.
Add a copy data activity in a pipeline, after successful data load, update watermark_table with the current UTC timestamp to record the latest successful run.
Use triggers in the Data Factory to schedule the pipeline to run every 24 hours.
For more detailed information, Please refer to the Microsoft official document:
If this response resolves your query, kindly mark it as Accepted Solution to help other community members. A Kudos is also appreciated if you found the response helpful.
Thank You!
Can you be more specific what you mean by incremenatl refersh in data pipelines?
Do you mean incremental data sync from a source to sink via copy activity?May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.
Thank you.
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.Hi Charlotte_chum
I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.
Thank you.
