Forum Discussion

Syndicate_Admin's avatar
Syndicate_Admin
Icon for Administrator rankAdministrator
4 years ago

Incremental refresh saving deleted records in data source

Hello colleagues

First of all, I attach an image with the desired scenario in case the problem is not well understood.

If you have another approach to solve this issue, please feel free to exposed it.

Thanks in advance,

I need some advice on the following problem I have:

From Power BI I am connecting to a SalesForce table. The API has a restriction of 2000 rows, that is, when I update the tables the Power BI will only be able to update 2000 records. This is a problem for me because I would lose historical data.

I came up with the following solution:

1.- Configure the SalesForce report to only contain the previous day's data. With this, I solve the 2000 rows problem because the daily records will never exceed 2000 records.

2.- Configure the incremental update in Power BI to update only the data from the day before the update date.

I tested, from Power BI I connected to the SalesForce report with data only from the day 07/06/2022 then I configured the incremental update to update the data only from the day before the update date, i.e. today is 09/06/2022. When I update the table it will only update the data from the day 08/06/2022 (AND THE DATA I HAD ALREADY LOADED PREVIOUSLY SHOULD BE REMAIN). Therefore, you should have a table with data from 07/06 + 08/06.

This does not happen. When I update the table, it deletes the records of 06/07 and updates those of 06/08.

The question is, how can I keep data that have been deleted in the source and increase with the new data that appear in the source?

Do you have any other ideas on how to address this Salesforce API restriction the 2000 rows and be able to retain the historical data each time we update the query in PBI?


renecernitz_0-1654774681408.png

@amitchandak

@tamerj1

@Vijay_A_Verma

@SpartaBI

1 Reply

  • Anonymous's avatar
    Anonymous
    Not applicable
    4 years ago

    Hi Syndicate_Admin ,

     

    According to your statement, I think you want to load data in two-day period into your data model instead of one-day period.

    I think you can set "Incrementally refresh data starting before refresh date" as you want in Incrementally refresh setting.

    For example, if we specify a refresh period of 3 days, with each refresh operation the service overrides the RangeStart and RangeEnd parameters to create a query for rows with a date/time within a three-day period, beginning and ending dependent on the current date and time. Rows with a date/time in the last 3 days up to the current refresh operation time are refreshed. 

    For reference:

    Incremental refresh and real-time data for datasets

     

    Best Regards,
    Rico Zhou

     

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.