Component tab shows more secondary characteristics


I am currently experiencing an issue with the component tab that has me stumped, and I was hoping for some assistance. Specifically, I have implemented multiple button states - primary (that has black, white, 1 and 2 color option), secondary (ghost and textual which only have black and white options) - and when I switch to the secondary ghost state, I am presented with more color options than I actually require (1 and 2 also shows in the ghost and textual states). I would like to limit my color options to just black and white.

Would someone kindly guide me on how to proceed with resolving this issue? Thank you in advance for your time and support.