I’ve been having similar issues and have figured out what the issue is but have a somewhat limited solution.
I think the reason this is happening is because when the modal is vertically aligned to the centre of the page, the modal may sometimes sit on a half pixel rather than a full one depending on the height of the viewable area.
So if the viewable area height is an odd number, the modal will sit on a half pixel, and if the viewable area height is an even number, the modal will sit on a full pixel.
When sitting on a half pixel, the line does not show, which is a Figma prototype rendering issue.
You can see how this will affect your prototype view as you manually resize the height of you prototype window to a smaller size than the size of your prototype. You should see the button line appear and disappear as the viewable height area changes and the centred modal position adjusts accordingly.
Unfortunately the only situation where the line will show is if the Y position of the modal and button inside, as well as the height of viewing area are all even numbers. Of course we have no control of the height of the viewing area, so no guarantees.
Really we need the FIgma prototype to automatically make sure overlay items are positioned on the nearest pixel to resolve this.
Is this something on your radar @Figma_Support ?
In the meantime, the best you can do to reduce this happening is to make sure you use as many even numbers as possible for positioning and sizing eg… for your modal height, button height and y position, and even the size of your frame.
I’ve also noticed the same thing also happens if you adjust the width too to anything smaller than prototype width. See the screenshot below where both the left and top of the button is cut off as the screen size adjusts).

I hope this is of some help.
I would have voted up if available, but hoping for a fix soon!