This is currently giving me a whole lot of grief. Swap Library seems like a very useful tool, but in its current form it keeps failing because style and component names don’t match 1:1. It actually looks like the interface is meant to allow you to manually select replacements. (The grey arrow and text almost seems like an empty state) but there’s no option to do this.
Working with different teams and different brands and sub-brands, I don’t always have edit access to the different libraries I’m working with, or even access to the teams that control them. That means relying on naming conventions alone is usually not reliable. Just the ability to look through the library and select a manual swap (for example, “Primary Color” to “primary-color”) would make a huge difference and make so many of those phantom styles and “missing library” alerts go away.