Swapping color library for the assets

What do I have:
1 component library
several color libraries

What do i need:
Connect a color library only once so i could get any component from the assets with a specific coloring for the brand.

I dont like the idea that I have to build a layout with a default component colors and then switch a library. And switch it again for the desired colors after i did some work in the design file.

Thanks!