Understanding the bucket size setting for the incremental refresh in dataflow gen2

Hi pmscorca ,

Thanks for sharing your incremental refresh settings. Let's break down your points to clarify the concepts further.

• Monthly Bucket and Date Column to Filter By: You're correct that a monthly bucket means the data will be partitioned by month based on the "orderdate" column. If you have orders only in August, the partition will be for August. However, if you have orders in August, September, and October, there will be three partitions, one for each month. The "Extract data from the past" option is indeed relevant for the initial load, ensuring that historical data is included based on the specified range.
• Extract Data from the Past: This setting is primarily for the initial load. For subsequent loads, the data will be loaded based on the current month and moving forward. For example, in October, it will load October's orders, in November, it will load November's orders, and so on. The order date will determine which monthly bucket the data falls into, such as an order with the date 10/20/2024 falling into the October bucket.
• Maximum Value in Modified Date Column: This setting ensures that if any data within a monthly bucket changes, the entire bucket is refreshed. So, if an order within a monthly bucket is modified, the entire bucket for that month will be refreshed, potentially replacing unmodified orders if any changes are detected within that bucket.

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly