API only delivers one year of data, how to avoid deleting data older than a year?

Hi Bokazoit 

In your scenario, where the API only provides a rolling one-year window of data that shifts daily—meaning each day you lose the oldest day’s data and gain a new day’s data—handling inserts in your database requires a strategy to maintain a complete historical record despite the limited API window. Since the API data window moves forward every day, you cannot rely on the API alone to keep the full dataset, especially for older dates falling outside the current one-year window. To handle inserts effectively, you should implement an incremental load process that fetches the latest available data from the API each day and inserts new records for those new dates. At the same time, your database must maintain all previously collected data for dates outside the current API window so you don’t lose historical data. This often means your ETL or data pipeline should detect which dates are newly available from the API, insert or update those records, and avoid deleting older data that is no longer provided by the API. In other words, you treat the API as a moving snapshot and maintain your own persistent storage that accumulates data over time. This approach ensures that even when the API drops older dates from its response, your database retains a full historical timeline by only inserting and updating new or changed records, and never deleting older data unless explicitly required.