I bet this has to do with the size of the images you imported, which appear to be gigantic, though scaled down in size. For example, I have zoom to around 3000% before I even begin to see anti-aliasing on the edges of the purple abstract shapes.
Try this: select your shapes and export at 1x (or even 2x). Then import them, replacing the images you exported.
Still, it’s disturbing that this is an issue only on an M1 mac, and not on Windows. I can confirm it’s slow on mine M1 as well.
By the way, I tried your file out on an Intel Mac (2.3 GHz 8-Core Intel i9) on Catalina with the same amount of memory as my Big Sur M1 (16GB), outputting to a lower resolution monitor than my M1 outputs to, and it was significantly faster on the Intel mac than the M1. Night-and-day faster, in the client or browser.
Sounds like there is further optimization to be done for the M1. Did you report this issue?