Using "Navigate to" on a non-default varient in a component

Hi, I’m having trouble using “Navigate to” overrides on component variant interactions. I’m trying to make a component where a non-default variant will navigate to a different frame (while the rest just change between the varients).

For example. I have a list that you can navigate up and down with the arrow keys (the arrow keys here are changing the list component between variants of each other. When you press up from the top of the list, it navigates to a new frame, when you press down at the bottom of the list, it navigates to a new frame.

For some reason, this only works for the variant that is set first when starting the prototype (whether the first or last item is selected by default). Is this a bug? As designed? Any workarounds other than having individual frames for each list item being focused?