This sounds it might be the same issue as discussed here: DO12 - Very slow to launch new lister after last one closed
As a test, can you try turning off the Preferences / Folder Tree / Appearance / Use configured file display colors for tree items option and see if it makes a difference?