Permissions for Users to Only See Specific Groups

It would be useful if I could specify which groups in collections I want certain users to be able to see and interact with. Currently you can set permissions on which user can see a group, but there is no ability to limit a User's view in collections to only be specific groups. This would be very useful for users that do not have any use for the ALL collection, and that should only be able to work out of specific group views.