Selecting multiple contacts

If I want to send an email to a group, in the example below they all have the same tag, and I select "All in group" that will naturally select everyone in that group. You can see in the example below that 199 people are selected.

However, if I unselect just one person in that group that I want to exclude, instead of now selecting 198 people, the system goes to 49 people (i.e. it only keeps the contacts selected that are on the page... which is a maximum of 50 people).

This is annoying and weird. It would be better if when I unselected that one contact, the number of contacts selected went to 198.

Furthermore, it'd be great to be able to show 100, 200 and even up to 500 people per page.