Limiting access to specific libraries only

Oh, you mean the search results. Yeah, that’s a known issue. You can vote for solving it here: Enable Assets search only for enabled libraries