See Calculate Folder Sizes Automatically for other things which may trigger folder size calculation.
Specifically, if you have columns turned on which recursively count total files/folders below each directory, or if you have modified Folder Formats for the type of folder in question so it overrides the global Preferences setting.