Help with incrementally refreshing an entity computed from two separate dataflows

Hi  purple_SP ,

Instead of merging the entire second entity with the grouped first entity, consider filter the second entity firs, you may apply a filter to the second entity to only include rows with codes that exist in the first entity. This can be done using a list of unique IDs from the first entity. If possible, use an inner join instead of a left join, as it can be more efficient when you only need matching rows.

You can use parameters to dynamically filter the data being loaded. Create a parameter for the date range you want to refresh. then apply this parameter to filter the second entity before merging.

Finally you may consider using staging tables to store intermediate results. This can help break down the process into smaller.

You can stage the filtered second entity, create a staging table that only contains the filtered rows from the second entity and then merge with the first entity, which perform the merge operation on the staged data.

Best regards.
Community Support Team_Caitlyn