There is nothing in the help for Select to suggest that negation (~) can't be used with grp: but using Select ~grp:images DESELECTNOMATCH to select files that are not images does not work.
Simple enough to achieve by means of a script or Select grp:images DESELECTNOMATCH then Select INVERT.
~grp:Images as well as ~(grp:Images|grp:Documents) will work in the next beta/update.
In the current version, you can use this as a (slightly odd) workaround:
~(*grp:images))
That workaround isn't proper wildcard syntax and may not work forever. The extra * and the unbalanced brackets are technically wrong, but it ends up working. It exploits some quirks in the way the pattern matching/expansion code works to get the desired result until proper support is added.