Data Pipeline Incremental load

Hi Hussein_charif ,

 

Thank you for reaching out to Microsoft Fabric Community Forum.

 

Since your Dataverse table has over 30 million rows, doing a full refresh each time is not scalable.

Instead, you should implement an incremental load strategy to pull only new or updated records.Here are some workarounds to do that:

 

1. Use ModifiedOn or CreatedOn Columns

Dataverse tables typically include system fields:

• createdon — the date when the record was created
• modifiedon — the date when the record was last updated

You can filter your data extraction by only pulling records where modifiedon is greater than the timestamp of your last successful load.

 

Example query:

SELECT * FROM account
WHERE modifiedon > [LastSuccessfulPipelineRunTime]

You can apply this logic in Dataverse API queries, FetchXML, or OData.

 

2. Store and Reuse the Last Extraction Timestamp

In your pipeline:

• After a successful load, store the maximum modifiedon value from the extracted data.
• In the next run, filter records starting from this stored timestamp.

You can save the last run timestamp in:

• Azure Key Vault
• A SQL control table
• Pipeline parameters

 

3. Pipeline-Level Implementation Steps

If you are using Azure Data Factory (ADF) or Synapse Pipelines:

• Create a pipeline parameter called LastModifiedDate.
• At the start of the pipeline, retrieve this value (from a config table, Key Vault, or elsewhere).
• Pass it dynamically into your Dataverse query.
• After loading the data, update the stored LastModifiedDate with the latest value from your extraction.

 

4. Handling Deleted Records

Note that using modifiedon alone will not detect deleted records unless:

• You have a Soft Delete setup (records are flagged, not removed), or
• You enable Dataverse Change Tracking (recommended for full delta handling).

 

5. Use Dataverse Change Tracking (Recommended for Large Data)

Dataverse offers Change Tracking, a built-in feature that:

• Tracks inserts, updates, and deletes automatically.
• Allows you to fetch only the changed data since your last sync.
• Removes the need to manually compare timestamps.

If you're using the ADF Dataverse connector, it can automatically leverage Change Tracking when enabled.

 

Please refer the below document:

Use change tracking to synchronize data with external systems (Microsoft Dataverse) - Power Apps | Microsoft Learn

 

Regards,

B Manikanteswara Reddy