Smart animating between frames is making elements move around

Whenever I use smart animate to go from one frame to another, the elements start moving in weird ways like I’ve shown in the video. How can I fix this?