How to change fill image on all variant of a component instance?

I’m trying to make an image gallery with a hover effect. I thought I could create one component with a blank image in both variants (normal/hover), duplicate it multiple times on my page, change the fill image for each instance, and keep the hover effect. But the images aren’t linked. When I hover and it switches variants, it changes the image.

Is that even possible? Will I need to create multiple main components?