If you haven't already, have a read of these FAQs in case they are relevant. If they don't help, let us know and we can think of more things to try.
Calculate Folder Sizes Automatically -- Explains how to prevent folder sizes being calculated. Note that there is a global setting in Preferences and then per-folder overrides of that setting in Folder Formats. Also note that if you have turned on columns which display the number of child folders/files then they will force the sizes to be calculated (as Opus is being forced to list all children anyway when those columns are on).
That may be all you need to speed things up. If not, try some of the suggetions in Crash, exit or high CPU when viewing a certain directory, particularly using ShellExView to see which shell extensions you have installed. Are there any shell extensions you can see which came with something you installed/changed recently?
Some shell extensions have been known to try to contact a network server every time you change directory, for example. (I think it was a source-control extension which did this in the past.)