Is there an Inverse Mask?

I have this:

If I group red and blue together, and yellow and green together, how do I exclude the two groups? If I do an exclude selection it turns into this:

Not only are the colors unified, but the groups’ elements are also subject to exclusion. Unexpected, but still working as intended

For lines cutting through groups, I figured I could rasterize them. It works in that case although you’ll have to maintain the pre-rasterized group.

But in this case, when I did an exclude selection, the 2nd image(very right box) just disappeared

Is there an easier way to do this? Like an inverse mask? Any help is appreciated, thanks in advance!