When hosts are selected, and then the host group is changed, the hosts stay selected. This is signified by the button at the bottom of the list that is labled as "Go (X)" where X is the number of hosts selected.

The issue is if there is a host selected in another host group, it is not obvious which host group the host is in.
If a user then needs to unselect hosts - for example only hosts in one host group need to be selected - the user then has to either go through all the host groups to find the host to unselect, or navigate away from the host selection screen which could possibly be an inconvenience if the user has selected a lot of hosts on the current host group.

A "Unselect hidden hosts" button would be beneficial in that it would allow the user to unselect hosts that are in a different host group without losing his current workflow.