I've looked into this in detail and have realised it's not really a bug, although it is something we might change to make things easier to understand.
Essentially, the Image Converter (the thing that rotates image files) ignores EXIF rotation except in the special case that you tell it to rotate by the opposite of the EXIF rotation to fix the image.
Outside of that special case, the Image Converter always works on the actual pixel data, without any EXIF rotation added to the mix. If it rotates an image it'll clear the EXIF rotation data, which can mean that (in places where the EXIF rotation is used) the end result isn't what you'd expect as it's effectively two rotations instead of one.
There are three independent "use EXIF rotation" options for Thumbnails, the preview pane and the standalone viewer, but none affect the converter. Since the converter can be launched from all three of those places, and others, it wouldn't make sense for it to use any of their flags. It could have its own flag, but that could still be out of step with any of the others.
We might merge all the flags into a single setting, which probably makes sense anyway. But, even then, there are some complications to work out, like what happens if you convert an EXIF-rotated JPG into a BMP or some other format that doesn't contain EXIF, without asking for it to be rotated. Probably it should be rotated anyway, but the amount of cases to consider, code for, and test means it's complicated.
It's something we need to think about some more, and might change. It's definitely confusing if you see an image in one orientation in thumbnails (with EXIF rotation turned on for thumbnails) and then say "rotate 90 degrees" and you get a new thumbnail that isn't rotated 90 degrees from what you were looking at. And I don't think we need three separate EXIF rotation options, either; people probably either want it on or off everywhere. But there are a lot of interconnections to consider before we can change anything.
For your use-case, with the way things are currently, the best thing to do would be to either remove the rotation info or apply it and bake it into the image. Both can be lossless with some JPEGs, but removing the rotation info is the best option as it's always lossless. You can then rotate the image as needed.
(Of course, another option is to change the EXIF rotation value without touching the pixel data at all. That'll work as long as the image is displayed in things that use that value, and that's always lossless as well.)
The interactive image converter UI will also show you what's going to happen for any given operation, if in any doubt.