How to achieve this animation style?

You already have an answer on your image: create two screens, one in hover state and one in normal state. Add on hover trigger and transition to second screen with smart animate.

I want it to act as a button instead of a transition though, and when the picture becomes teh bigger button then the other images become smaller. Any help with that?

Set regular on click event on the hovered page to direct the user to the other page.

Make them smaller then. Just look at your screenshot and repeat.