Totally agree, I’m having the same issue or misunderstanding of how the instance swap property is meant to be used. At the moment it seems pretty basic and not worth breaking up my component sets to use. Sorry to reply sans any solutions, hopefully someone else who maybe knows more will respond.
Found this thread but haven’t gotten too deep into it to see any response from devs. Why can't instance swap call on variants?