"Select Only" should do what you want, as long as you're in Power Mode.
Make sure you drag down and not left, right or diagonally, to avoid initiating drag & drop.
So you haven't switched to Power Mode? The Power Mode options only affect that one mode; they have no effect on Details, Thumbnails or any other mode.
To switch the current folder tab into Power Mode, use View > Power Mode in the default toolbars. (Note that it may change back to Details Mode if you change folders. Just switch modes and see if the mouse is behaving as you want first, then worry about that.)
To make Power Mode the default all the time, see Folder Formats: Quick Guide.
Opus also behaves like that by default, but if it's not what you want then we don't need to spend time on it.