While the above is true. It depends if you have a Flow starting point set or not, it will behave differently for each case as noted in the linked article above (see excerpt below). Deleting my flow starting point fixed the problem for me.
Prototypes with a flow starting point
If your prototype has a starting point, Figma will create a “history” of frames you’ve visited when viewing the prototype.
• ← will go to the previous frame, if one is available.
• → will navigate to the next frame. If there is no history—like after using ←—then it will navigate to an adjacent frame you haven’t visited.
Prototypes without a flow starting point
If a prototype doesn’t have a flow staring point, Figma orders frames based on their co-ordinates in the canvas. First, by their x co-ordinate from left to right, then their y co-ordinate from top to bottom.
If the y co-ordinate is offset in any way, then frames may appear out of order. To fix, you can set horizontal alignment for each row of frames to align-top.
• ← will go to the previous frame if one is available.
• → will navigate to the next frame. First by x co-ordinate, then y-co-ordinate.