Categorical Object Properties Issue

I'm trying to create a custom visual where the user should be able to pick colors for each category.

I'm able to generate the color picker as shown in the screenshot, but my problem is that when I update one of the categories, all of the pickers update. Similar code structure seems to work from the Sample Bar Chart, so I'm not sure where setting the color is going wrong.

In the screenshot, you can see the each node corresponds to an object in that list. And when I change only the first one to white (#ffffff) all of the color picker change their property color.

Let me know what additional information would be helpful for troubleshooting. I'm creating a chord chart using categorical data structure with 2 groupings and a measure.