If you would like the Esc key in the image viewer to exit full screen mode, instead of closing the viewer, while still closing the viewer when in windowed mode:
Go to Settings > Customize Toolbars > Keys and create a new Viewer Hotkey:
Since Esc is special, you'll need to use the menu on the right of the hotkey field to specify it:
Click Advanced in the command editor to put it into multi-line mode.
Use this as the command:
@if:Show VIEWERCMD=fullscreen
Show VIEWERCMD=fullscreen
@if:else
Show VIEWERCMD=close