There's a script add-in which will do that here:
If you're using a keyboard/touchpad combo like the K400+, Ctrl + Space and Ctrl + Cursor Keys are also easy ways to toggle selections and move focus around without changing other files' selections. I use them myself on the same keyboard.
Power Mode also has a toggle select option, which would make left-click toggle things without needing to hold Ctrl or use the middle button. (But that won't help if you want it in Thumbnails mode, of course.)