Long loading times for Swap Instance panel

Hey, I noticed that the Swap Instance panel in the Design tab takes a lot of time to load. No matter the internet connection (slow or fast) this take ages. Sometimes I need to restart Figma in order to make it work.

Some of the libraries, like this one: Phosphor Icons added to the Community, has more than 4,000 components

Not sure what is the technical strategy for serving components from the libraries but this is really slowing down my workflow (alongside the new, and messed up, way of not look-up for components among all the libraries).