Incremental dataset refresh

[PBILover]

How to perform incremental refresh based on modified date in Power BI?

I have a requirement where-in I need Incremental refresh with the ability to handle modifications. i have  record created  date and record modified date. I have INCR dataset refresh policy set on the created date for last 10 days.

For example: can I configure Incremental refresh on date modifed field, and suppose a record that was pulled in previous load is now modified in source thereby having a new Date modifed, then is the Incremental load able to delete the older record from the model when loading the new version of this row?

If not supported by Power BI incremental refresh, then is there any way to handle such scenario in Power BI? 

I am using SYnapse as a Source of the data .

Thanks

[v-achippa]

Hi @PBILover,

As we haven’t heard back from you, we are closing this thread for now. If you have any further issue, please raise a new thread in the community forum, we are ready to assist you on it.

Thank you for being part of Microsoft Fabric Community.

Thanks and regards,

Anjan Kumar Chippa

[NEWATIA123]

I will check the post

[v-achippa]

Hi @PBILover,

Thank you for reaching out to Microsoft Fabric Community.

Thank you @Deku for the prompt response.

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the super user resolved your issue? or let us know if you need any further assistance.
If our super user response resolved your issue, please mark it as "Accept as solution" and click "Yes" if you found it helpful.

Thanks and regards,

Anjan Kumar Chippa

[v-achippa]

Hi @PBILover,

We wanted to kindly follow up to check if the solution provided by the super user resolved your issue.
If our super user response resolved your issue, please mark it as "Accept as solution" and click "Yes" if you found it helpful.

Thanks and regards,

Anjan Kumar Chippa

[PBILover]

It does not resolved my issue, but useful for some insights.

Thank you for your support.

[v-achippa]

@PBILover Thank you for the response, do you need any other assistance from us. If any of the responses were helpful please mark that response as Accept as Solution or please provide any useful insights and accept your own post as solution, so that other community members will find it more easily.

Thanks and regards,

Anjan Kumar Chippa

[PBILover]

Sure, I want to achieve only the partition which updated modified date should get refreshed. Is it possible to acieve this in any way? lets say in detect data changes there are 5 partitions out of which only 2 have changes in it. Power BI refresh all the 5 partitions, but i want to refresh only those 2 partitions.

[v-achippa]

Hi @PBILover,

Yes, power bi supports this case using the ‘Detect data changes’ option in Incremental Refresh. When this option is enabled, power bi tracks the maximum value of your selected change detection column for example like ModifiedDate for each partition. During the next refresh only those partitions where this value has changed will be refreshed and others will be skipped.

So in your case, if only 2 of the 5 partitions have updated records, only those 2 partitions will be refreshed. Make sure that ModifiedDate must be datetime not just date for accurate change detection.

If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it! 

Thanks and regards,

Anjan Kumar Chippa

[PBILover]

Correct me if i am wrong, i observed that it is not working as it is stated here, I observed all 5 partitions are getting refreshed.even though there are changes in only 2 partitions.

DO you have any scenarios on this?

@v-achippa 

[NEWATIA123]

I agree with you 

[v-achippa]

Hi @PBILover,

Looks like some conditions did not fully met, while power bi supports selective partition refresh using detect data changes, but it only works when a few key conditions are met:

• The query must fully support query folding all the way to the data source (in your case, Synapse).
• ModifiedDate must be a datetime column and used as both the range and change detection column.
• Partition specific refresh takes effect only after the first full refresh.
• This feature is supported only in Premium, PPU, or Fabric capacities but not in Pro.

If any of these are not met, power bi will default to refreshing all partitions in the incremental range.

To confirm I recommend checking query folding, you can use the refresh history and verify which partitions were actually refreshed.

If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it! 

Thanks and regards,

Anjan Kumar Chippa

[PBILover]

@v-achippa : I have all above conditions fulfilled

[v-achippa]

Hi @PBILover,

Thank you for confirming that all the required conditions are met. Since you are still seeing all partitions being refreshed, I recommend you to please raise a Microsoft support ticket so that they can analyze the backend telemetry and logs and investigate further.

How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

Thanks and regards,

Anjan Kumar Chippa

[PBILover]

Sure, Thank you

[v-achippa]

Hi @PBILover,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the super user resolved your issue.
If our super user response resolved your issue, please mark it as "Accept as solution" and click "Yes" if you found it helpful.

Thanks and regards,

Anjan Kumar Chippa

[PBILover]

Thanks @Deku , I will check the post

[Deku]

You can only ever refresh a full partition, not a particular row. If the older version of the row is in archived partition it will remain. You can extend the window of active partitions and set a polling expression so that partitions without an updated modified date are not refreshed.

See this Chris webb post on the topic