Incremental Data Load for future dates.

rajatkatiyar · ‎04-10-2023

Hi,

I am working on huge dataset its record count is around 200 million records and I am trying to implement incremental load. While applying policy I can see it is available to load the data till todays date or till the month end but as our data is travel data like on which date someone want to travel according to there booking date so we need to include the future date. I already made partitions based on quarter basis and last 3 month data on month basis and load the data but issue occurs now how to incorporate the future dates as effective date concept is also not working.