Frustrating experience with 'Go to main component' button for view only users

There’s a thing where the ‘Go to main component’ buttom may be there and may not. It looks like this:

At first it looked like view only users can go to main component if it’s in the same document and can’t if it’s in other document. But some people from dev and QA teams told me they can go to main component in other document while other people with same view only rights can’t. After some research I’ve discovered that this button appears for the user after accessing that other document with main component by direct link for the first time, even if document doesn’t have invite-only access. I understand concern that ‘Anyone with the link’  doesn’t mean free access, it is still require user to have a link to access the document but the UX of ‘Go to main component’ button is very frustrating. It could be simply grayed out with tooltip saying something like “You don’t have access to that document” or in this particular case, “Open this document with direct link in order to access its library”.