If you're going to modify/hack the WIndows DLLs you've got to expect some things to break.
@leo not really I modded dlls and exe for years now and as long as you only replace the resource lets say icon and you obey by its original icon sizes and color depth it won't cause problems other than windows file protection warnings, but I disabled that so now the only negative effect it does is that files have different hashes so it won't be digitally signed anymore, because its is not an original file anymore, but that doesn't do any negative performance impact.
I tried with another vmware installation - installed my modded windows, but this time the first app after installing windows was dopus.
It worked perfectly no shell locking, just like on original windows - even better, if I click my computer very early it will still launch dopus and explorer doesn't have a chance to pop up instead! So yea it was one of few programs that I installed after windows and it was:
PC tools Firewall plus
Why? Well this firewall upon starting starts a service and that takes quite a while, so until firewall is ready, dopus will lock shell, because it is waiting. This answers why it didn't work on new installation, but because I didn't have this firewall on old installation, it wasn't the cause. So there are more programs or more specifically 3rd party services that dopus have problems with, but only when it replaces explorer.
Too bad that I will have to use a different firewall now, because I was really happy with this one, as it does its job well (prevents programs to do unwanted things such as opening links using firefox!), while not slowing down windows boot like zone alarm does.