This seems quite an intricate setup.
Could you, by any chance, share a file with this so I can try and help you set it up, or at least see if I can replicate the same issue?
I’m not from Figma, just a moderator, but I’m heavilly invested in complex Interactive components.