I have a dataflow that contains data for the past 8 years. Most of the changes to that data are additions that would include a current date in the date column, so an incremental refresh process would capture those additions. However, if an input is made that is to an existing row of data, it makes that change and maintaines the original date. Also, if a deletion is made in the system to an input, which can be done to data that is 5 days old or 2 years old, it completely removes that row from the data source. Over 90% of the data is going to be additions to the existing data table, but enough changes will be made to historical data that it impacts the reports utilizing that data. The data only contains the one date column and doesn't include a seperate modification date column that we could utilize the detect changes option in the incremental refresh. We are needing the data refreshed throughout the day for certain reports that could be done utilizing an incremental refresh, but would also need to have a full refresh done once or twice a day to capture those potential changes to historical information. Is there a way to structure this within a dataflow to do both without having to do a manual process of turning off the incremental refresh? Or is there possibly a way to use power automate flows to do one or the other (incremental or full refresh) depending on another variable? If not, is there anything in development that might account for this scenario?
