Provided you go Back, and don't re-read the directory, the previously calculated sizes will still be there.
Make sure the parent button/hotkey you are using runs Go UP BACK and not just Go UP. That will ensure that if the parent folder is in the history, the cached details for the folder will be used instead of re-reading the folder. (And if the folder isn't in the history, it will be re-read, so you always end up in the parent folder, unlike Go BACK on its own.)
For the file display border button, make sure Preferences / File Displays / Border / File display border Up button does "Up Back" is turned on.
Alternatively, Preferences / Miscellaneous / Advanced / go_up_always_back can be set so that anything which runs Go UP is turned into Go UP BACK.
You can also turn on automatic size calculation for some or all folders, if you wish.