Layer selection, while Prototype panel is active, is difficult

While working in Figma with the prototype panel selected, I often want to create lots of interactions quickly across screens.

The ideal workflow would be: Select layer/s > drag the connector to create the interaction > then select the next layer and repeat

Instead, this is what I’m having to do: Can’t select the layer usually while in the Prototype panel so I have to go back to the Design panel > get the layer selected > go back to Prototype panel > drag the connector to create the interaction > again can’t select the layer usually while in the Prototype panel so I have to go back to the Design panel > repeat

Has anyone found a quicker way to make lots of interactions without having to switch back to the Design panel over and over?