Touch/scroll propagation conflicts between interactive components and screen

I’ve built an interactive component that’s basically a toggle switch with smart animation effects when changing state. You can change state by either dragging or tapping on the unselected position. I’m using multiple instances of this component on a screen. When playing the prototype or using Figma Mirror, it all works great when the screen content is not long enough to need to scroll. But once the screen scrolls vertically the drag action on the interactive component is not captured by the component and instead it scrolls the screen. I have the screen set to only scroll vertically and the component works by horizontal scrolling, but it seems even the slightest bit of vertical motion is captured as a vertical scroll.