I swapped libraries but some of my variants are now assigned to the new library and some to an old one

I am aware that currently when swapping libraries it does not work on variables, but somehow when I swapped libraries, some of the color variables reassigned to the new library variables. So now I have components color variables mixed with the original library and the swapped one. How is this possible? I thought the swapped components variables will remain linked to the original library. Does anyone have similar experience?