Another thing to note is that if you hold Ctrl and move over a selected item, it'll deselect it.
If your mouse settings have the hover time set extremely fast, that could mean that a slight movement of the mouse is going to potentially select/deselect it multiple times.