Hopefully that doesn't mean that you actually have the 'Calculate folder sizes automatically" option enabled... If you're not normally operating in details or power mode, then that's alot of the disk thrashing Leo mentioned for no reason if you're not even displaying the size column.
Why not make a button that enables details for you AND runs the get sizes function AND sorts by size so you can easily see your empty and 0 byte folders in order to delete them without having to navigate into everything?