Thanks for reaching out. To understand correctly, would you be able to share the file link? Generally, when you use “While hovering” with change to action, it should go back to the previous state. Here is my sample:
Please let me know what you’d like to achieve a little bit more.