Guess that is it then. I changed from "reset zoom level for each picture to: fit to page" to "original size" and going back/forth was faster, practically instantly. Zooming still takes a while.
It would be great if multiple scaled images were prepared in advance.
Also, the caching seems to apply only to the next/previous. I use Show VIEWERCMD=goto,+10 for faster navigation and I am planning to write a script to jump to the first image of the next/previous folder, so a more aggressive caching would be great.