The ~ applies to the token immediately following it, which in this case is "a", not "abc". If you want it to apply to the whole string you need to put it in parentheses, i.e. ~(abc).
The FAYT select-mode always uses "partial matching", which effectively means any pattern you use has * added to the start and end.
That, in turn, means that negative patterns won't work with FAYT select-mode, even when using ~(abc) as Jon suggests above.
But you can do it via Edit -> Select by Pattern. Make sure the partial match checkbox is turned off and use this as the pattern:
~(abc.txt)
(For convenience, the filter bar has some special handling of patterns starting with ~ which means e.g. you don't need the brackets when using a negative pattern in the filter bar, but they are needed in most other places.)
Thanks for the explanations. I think what was throwing me was that the filter box and the select box was behaving differently when using the same ~ pattern and I wasn't sure if it was by design.