How do I stop seeing components from unrelated libraries?

Figma allows me to insert instances of components from libraries that are not relevant to the project I’m working in. I do not understand why they show, and the libraries do not show in the libraries panel. How do I stop seeing components from unrelated libraries as I use SHIFT + i to search for components?