High DOpus CPU usage during non-DOpus operations

chrisp · November 1, 2012, 5:16am

Hello,

Directory Opus is absolutely thrashing my CPU whenever i go to install just about anything on a freshly installed Windows 8. Quite literally anything that i've installed has made Directory Opus use up to 75% CPU while it has been installing. Adobe, Corel, Microsoft, anything at all from anyone. This is on an i7-2600K @ 3.8GHz too, so we're looking at up to 6 maxxed out fast cores.

I've had a look at the call stack of various threads during these CPU spikes (using "Process Hacker" which is a "Process Explorer" clone) and i can see no strange .dlls and what not. It's all shell32.dll, user32.dll, dopuslib, and dopus.exe, etc. Nothing but the core Microsoft and Directory Opus DLLs/modules/EXEs.

I used ShellExView to disable everything that was not Microsoft, even GPSoftware extensions. I also killed/restarted explorer.exe and restarted Directory Opus to make sure there was no lingering hooks or extensions etc.

Still the same ridiculous CPU usage.

The only thing that seems to make Directory Opus use less CPU is closing listers. At around only 3 listers, i get around 20% CPU maximum used by Directory Opus. But what's the point of that? Directory Opus is for power users. I want to have 10+ listers open!

Here's some call stacks during the high CPU usage:

0, ntdll.dll!NtNotifyChangeDirectoryFile+0xa 1, KernelBase.dll!ReadDirectoryChangesW+0xa9 2, kernel32.dll!ReadDirectoryChangesW+0x36 3, dopus.exe+0x550e87 4, dopus.exe+0x552345 5, user32.dll!DispatchMessageW+0x1fe 6, user32.dll!MsgWaitForMultipleObjectsEx+0x237 7, dopus.exe+0x54ffbb 8, dopuslib.dll!IsWow64+0x31 9, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60673 10, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60727 11, kernel32.dll!BaseThreadInitThunk+0x1a 12, ntdll.dll!RtlUserThreadStart+0x21

0, user32.dll!TranslateMessage+0x4a 1, user32.dll!TranslateMessage+0x90 2, user32.dll!PeekMessageW+0x85 3, dopus.exe+0x54ffab 4, dopuslib.dll!IsWow64+0x31 5, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60673 6, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60727 7, kernel32.dll!BaseThreadInitThunk+0x1a 8, ntdll.dll!RtlUserThreadStart+0x21

0, ntdll.dll!NtWaitForSingleObject+0xa 1, KernelBase.dll!WaitForSingleObjectEx+0x9a 2, dopus.exe+0x4673b9 3, dopus.exe+0x550159 4, ntdll.dll!KiUserApcDispatcher+0x2b 5, ntdll.dll!ZwWaitForMultipleObjects+0xa 6, KernelBase.dll!GetProcessHeap+0x56 7, user32.dll!WaitMessage+0x2e3 8, dopus.exe+0x550005 9, dopuslib.dll!IsWow64+0x31 10, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60673 11, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60727 12, kernel32.dll!BaseThreadInitThunk+0x1a 13, ntdll.dll!RtlUserThreadStart+0x21

0, ntdll.dll!NtWaitForSingleObject+0xa 1, KernelBase.dll!WaitForSingleObjectEx+0x9a 2, dopus.exe+0x4673b9 3, dopus.exe+0x5522e6 4, user32.dll!DispatchMessageW+0x1fe 5, user32.dll!MsgWaitForMultipleObjectsEx+0x237 6, dopus.exe+0x54ffbb 7, dopuslib.dll!IsWow64+0x31 8, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60673 9, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60727 10, kernel32.dll!BaseThreadInitThunk+0x1a 11, ntdll.dll!RtlUserThreadStart+0x21

0, ntdll.dll!ZwWaitForMultipleObjects+0xa 1, KernelBase.dll!GetProcessHeap+0x56 2, user32.dll!WaitMessage+0x2e3 3, dopus.exe+0x550005 4, dopuslib.dll!IsWow64+0x31 5, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60673 6, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60727 7, kernel32.dll!BaseThreadInitThunk+0x1a 8, ntdll.dll!RtlUserThreadStart+0x21

0, ntdll.dll!NtNotifyChangeDirectoryFile+0xa 1, KernelBase.dll!ReadDirectoryChangesW+0xa9 2, kernel32.dll!ReadDirectoryChangesW+0x36 3, dopus.exe+0x550e87 4, dopus.exe+0x552345 5, user32.dll!DispatchMessageW+0x1fe 6, user32.dll!MsgWaitForMultipleObjectsEx+0x237 7, dopus.exe+0x54ffbb 8, dopuslib.dll!IsWow64+0x31 9, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60673 10, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60727 11, kernel32.dll!BaseThreadInitThunk+0x1a 12, ntdll.dll!RtlUserThreadStart+0x21

0, ntoskrnl.exe!SeAccessCheck+0x1ef 1, ntoskrnl.exe!KeDelayExecutionThread+0xc32 2, ntoskrnl.exe!KeWaitForSingleObject+0x1cf 3, ntoskrnl.exe!_misaligned_access+0x809 4, ntoskrnl.exe!SeAccessCheck+0x280 5, ntoskrnl.exe!KeDelayExecutionThread+0xc32 6, ntoskrnl.exe!KeWaitForMultipleObjects+0x25d 7, ntoskrnl.exe!ObWaitForMultipleObjects+0x29c 8, ntoskrnl.exe!ObWaitForMultipleObjects+0x713 9, ntoskrnl.exe!KeSaveStateForHibernate+0x2a33 10, ntdll.dll!ZwWaitForMultipleObjects+0xa 11, KernelBase.dll!GetProcessHeap+0x56 12, user32.dll!WaitMessage+0x2e3 13, dopus.exe+0x550005 14, dopuslib.dll!IsWow64+0x31 15, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60673 16, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60727 17, kernel32.dll!BaseThreadInitThunk+0x1a 18, ntdll.dll!RtlUserThreadStart+0x21

0, ntoskrnl.exe!SeAccessCheck+0x1ef 1, ntoskrnl.exe!KeDelayExecutionThread+0xc32 2, ntoskrnl.exe!KeWaitForSingleObject+0x1cf 3, ntoskrnl.exe!_misaligned_access+0x809 4, ntoskrnl.exe!SeAccessCheck+0x280 5, ntoskrnl.exe!KiCheckForKernelApcDelivery+0x23 6, win32k.sys!EngCopyBits+0x1e209 7, ntoskrnl.exe!KeSaveStateForHibernate+0x2a33 8, user32.dll!Ordinal2427+0x100a 9, user32.dll!WaitMessage+0x282 10, dopus.exe+0x550005 11, dopuslib.dll!IsWow64+0x31 12, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60673 13, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60727 14, kernel32.dll!BaseThreadInitThunk+0x1a 15, ntdll.dll!RtlUserThreadStart+0x21

0, ntoskrnl.exe!SeAccessCheck+0x1ef 1, ntoskrnl.exe!KeDelayExecutionThread+0xc32 2, ntoskrnl.exe!KeWaitForSingleObject+0x1cf 3, ntoskrnl.exe!_misaligned_access+0x809 4, ntoskrnl.exe!SeAccessCheck+0x280 5, ntoskrnl.exe!KiCheckForKernelApcDelivery+0x23 6, Ntfs.sys+0xdb877 7, fltmgr.sys!FltReleaseContext+0x90e 8, fltmgr.sys+0x10b6 9, ntoskrnl.exe!ObWaitForMultipleObjects+0x8a8 10, ntoskrnl.exe!NtNotifyChangeDirectoryFile+0x2b5 11, ntoskrnl.exe!KeSaveStateForHibernate+0x2a33 12, ntdll.dll!NtNotifyChangeDirectoryFile+0xa 13, KernelBase.dll!ReadDirectoryChangesW+0xa9 14, kernel32.dll!ReadDirectoryChangesW+0x36 15, dopus.exe+0x550e87 16, dopus.exe+0x552345 17, user32.dll!DispatchMessageW+0x1fe 18, user32.dll!MsgWaitForMultipleObjectsEx+0x237 19, dopus.exe+0x54ffbb 20, dopuslib.dll!IsWow64+0x31 21, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60673 22, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60727 23, kernel32.dll!BaseThreadInitThunk+0x1a 24, ntdll.dll!RtlUserThreadStart+0x21

This is happening with Directory Opus 10.0.2.0 and 10.0.2.1. Windows 8, and Windows 7.

This forum post has actually been a few days in the working. I've gone back to Windows 7 now and although the CPU usage isn't as high, Directory Opus is still opening up about 10-15 threads that are each using 2.5% CPU. Directory Opus is unresponsive during this. (ie: I could not alt-tab to it) I wasn't even using Directory Opus at the time for anything. I was just extracting about 145 rar files with WinRAR which contain about 3.8GB of data and 17,077 files.

It seems as though Directory Opus is hooking into something or monitoring something to do with every single file operation on the computer, and taking a hell of a long time to update the GUI or its internal structures or whatever.

I'm at a loss what to do. The only thing i can think of is to go back to previous versions of Directory Opus and check if the same CPU usage occurs. This seems to be a relatively new problem. I don't recall having such high CPU usage in the past.

Some sample call stacks on high CPU usage threads for the Windows 7 WinRAR issue are:

0, ntoskrnl.exe!_misaligned_access+0x17e7 1, ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x93d 2, ntoskrnl.exe!KeWaitForMutexObject+0x19f 3, ntoskrnl.exe!_misaligned_access+0xba4 4, ntoskrnl.exe!_misaligned_access+0x1821 5, ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x93d 6, ntoskrnl.exe!KeWaitForMutexObject+0x19f 7, ntoskrnl.exe!NtWaitForSingleObject+0xde 8, ntoskrnl.exe!KeSynchronizeExecution+0x3a23 9, ntdll.dll!NtWaitForSingleObject+0xa 10, ntdll.dll!RtlDeNormalizeProcessParams+0x5d8 11, ntdll.dll!RtlDeNormalizeProcessParams+0x4cb 12, shell32.dll!SHCreateDirectoryExW+0xa3b 13, shell32.dll!SHChangeNotifyDeregister+0xb34 14, shell32.dll!Ordinal893+0x8cc2 15, shell32.dll!SHBindToParent+0xeb0 16, dopus.exe+0x34e8c7 17, dopus.exe+0x348712 18, dopus.exe+0x539f19 19, dopus.exe+0x5295c1 20, dopus.exe+0x534134 21, user32.dll!TranslateMessageEx+0x2a1 22, user32.dll!SetTimer+0x178 23, user32.dll!SendMessageW+0x5d 24, dopus.exe+0x5c31b3 25, dopus.exe+0x59b722 26, dopus.exe+0x5d46f6 27, user32.dll!TranslateMessageEx+0x2a1 28, user32.dll!CallWindowProcW+0x9c 29, user32.dll!CallWindowProcW+0x18 30, comctl32.dll!CreateUpDownControl+0x254d 31, comctl32.dll!DefSubclassProc+0x2b4 32, comctl32.dll!DefSubclassProc+0x7c 33, dopus.exe+0x3eb52 34, dopus.exe+0xc343b 35, comctl32.dll!DefSubclassProc+0x2b4 36, comctl32.dll!DefSubclassProc+0x7c 37, dopus.exe+0x393380 38, comctl32.dll!DefSubclassProc+0x2b4 39, comctl32.dll!DefSubclassProc+0x18b 40, user32.dll!TranslateMessageEx+0x2a1 41, user32.dll!TranslateMessage+0x1ea 42, dopus.exe+0x5da447 43, dopuslib.dll!IsWow64+0x31 44, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60673 45, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60727 46, kernel32.dll!BaseThreadInitThunk+0xd 47, ntdll.dll!RtlUserThreadStart+0x21

0, ntoskrnl.exe!_misaligned_access+0x17e7 1, ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x93d 2, ntoskrnl.exe!KeWaitForMutexObject+0x19f 3, ntoskrnl.exe!_misaligned_access+0xba4 4, ntoskrnl.exe!_misaligned_access+0x1821 5, ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x93d 6, ntoskrnl.exe!KeWaitForMutexObject+0x19f 7, ntoskrnl.exe!NtWaitForSingleObject+0xde 8, ntoskrnl.exe!KeSynchronizeExecution+0x3a23 9, ntdll.dll!NtWaitForSingleObject+0xa 10, ntdll.dll!RtlDeNormalizeProcessParams+0x5d8 11, ntdll.dll!RtlDeNormalizeProcessParams+0x4cb 12, shell32.dll!SHCreateDirectoryExW+0xa3b 13, shell32.dll!SHChangeNotifyDeregister+0xb34 14, shell32.dll!Ordinal874+0x5fea 15, shell32.dll!Ordinal733+0x251b1 16, shell32.dll!Ordinal99+0x4967 17, shell32.dll!Ordinal733+0x24676 18, shell32.dll!SHBindToParent+0xeb0 19, dopus.exe+0x34e8c7 20, dopus.exe+0x348712 21, dopus.exe+0x539f19 22, dopus.exe+0x5295c1 23, dopus.exe+0x534134 24, user32.dll!TranslateMessageEx+0x2a1 25, user32.dll!SetTimer+0x178 26, user32.dll!SendMessageW+0x5d 27, dopus.exe+0x5c31b3 28, dopus.exe+0x59b722 29, dopus.exe+0x5d46f6 30, user32.dll!TranslateMessageEx+0x2a1 31, user32.dll!CallWindowProcW+0x9c 32, user32.dll!CallWindowProcW+0x18 33, comctl32.dll!CreateUpDownControl+0x254d 34, comctl32.dll!DefSubclassProc+0x2b4 35, comctl32.dll!DefSubclassProc+0x7c 36, dopus.exe+0x3eb52 37, dopus.exe+0xc343b 38, comctl32.dll!DefSubclassProc+0x2b4 39, comctl32.dll!DefSubclassProc+0x7c 40, dopus.exe+0x393380 41, comctl32.dll!DefSubclassProc+0x2b4 42, comctl32.dll!DefSubclassProc+0x18b 43, user32.dll!TranslateMessageEx+0x2a1 44, user32.dll!TranslateMessage+0x1ea 45, dopus.exe+0x5da447 46, dopuslib.dll!IsWow64+0x31 47, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60673 48, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60727 49, kernel32.dll!BaseThreadInitThunk+0xd 50, ntdll.dll!RtlUserThreadStart+0x21

0, ntoskrnl.exe!_misaligned_access+0x17e7 1, ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x93d 2, ntoskrnl.exe!KeWaitForMutexObject+0x19f 3, ntoskrnl.exe!_misaligned_access+0xba4 4, ntoskrnl.exe!_misaligned_access+0x1821 5, ntoskrnl.exe!KiCheckForKernelApcDelivery+0x25 6, Ntfs.sys+0xadae3 7, FLTMGR.SYS!FltIsCallbackDataDirty+0x2df 8, FLTMGR.SYS+0x16df 9, ntoskrnl.exe!MmCreateSection+0xb32f 10, ntoskrnl.exe!NtWaitForSingleObject+0xe14 11, ntoskrnl.exe!NtWaitForSingleObject+0xbd1 12, ntoskrnl.exe!NtWaitForSingleObject+0x1194 13, ntoskrnl.exe!KeSynchronizeExecution+0x3a23 14, ntdll.dll!ZwClose+0xa 15, KernelBase.dll!CloseHandle+0x13 16, kernel32.dll!CloseHandle+0x41 17, shell32.dll!SHCLSIDFromString+0x1b09 18, shell32.dll!SHCreateShellItemArrayFromIDLists+0x1703 19, shell32.dll!SHCreateShellItemArrayFromIDLists+0x1554 20, shell32.dll!SHCLSIDFromString+0x1878 21, shell32.dll!SHCLSIDFromString+0x192e 22, shell32.dll!SHCLSIDFromString+0x192e 23, shell32.dll!SHCLSIDFromString+0x192e 24, shell32.dll!SHCLSIDFromString+0x44f 25, shell32.dll!SHGetFolderPathW+0x954 26, shell32.dll!SHParseDisplayName+0x25e 27, shell32.dll!SHGetFolderPathW+0x954 28, dopus.exe+0x485047 29, dopus.exe!DummyExeFunction2ToAvoidSymbolConfusion+0x63ab1 30, dopus.exe!DummyExeFunction2ToAvoidSymbolConfusion+0x638e6 31, dopus.exe+0x347c00 32, dopus.exe+0x539f19 33, dopus.exe+0x5295c1 34, dopus.exe+0x534134 35, user32.dll!TranslateMessageEx+0x2a1 36, user32.dll!SetTimer+0x178 37, user32.dll!SendMessageW+0x5d 38, dopus.exe+0x5c31b3 39, dopus.exe+0x59b722 40, dopus.exe+0x5d46f6 41, user32.dll!TranslateMessageEx+0x2a1 42, user32.dll!CallWindowProcW+0x9c 43, user32.dll!CallWindowProcW+0x18 44, comctl32.dll!CreateUpDownControl+0x254d 45, comctl32.dll!DefSubclassProc+0x2b4 46, comctl32.dll!DefSubclassProc+0x7c 47, dopus.exe+0x3eb52 48, dopus.exe+0xc343b 49, comctl32.dll!DefSubclassProc+0x2b4 50, comctl32.dll!DefSubclassProc+0x7c 51, dopus.exe+0x393380 52, comctl32.dll!DefSubclassProc+0x2b4 53, comctl32.dll!DefSubclassProc+0x18b 54, user32.dll!TranslateMessageEx+0x2a1 55, user32.dll!TranslateMessage+0x1ea 56, dopus.exe+0x5da447 57, dopuslib.dll!IsWow64+0x31 58, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60673 59, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60727 60, kernel32.dll!BaseThreadInitThunk+0xd 61, ntdll.dll!RtlUserThreadStart+0x21

0, ntoskrnl.exe!_misaligned_access+0x17e7 1, ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x93d 2, ntoskrnl.exe!KeWaitForMutexObject+0x19f 3, ntoskrnl.exe!_misaligned_access+0xba4 4, ntoskrnl.exe!_misaligned_access+0x1821 5, ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x93d 6, ntoskrnl.exe!KeWaitForMutexObject+0x19f 7, ntoskrnl.exe!NtWaitForSingleObject+0xde 8, ntoskrnl.exe!KeSynchronizeExecution+0x3a23 9, ntdll.dll!NtWaitForSingleObject+0xa 10, ntdll.dll!RtlDeNormalizeProcessParams+0x5d8 11, ntdll.dll!RtlDeNormalizeProcessParams+0x4cb 12, shell32.dll!SHCreateDirectoryExW+0xa3b 13, shell32.dll!SHChangeNotifyDeregister+0xb34 14, shell32.dll!Ordinal874+0x5fea 15, shell32.dll!Ordinal733+0x251b1 16, shell32.dll!Ordinal99+0x4967 17, shell32.dll!Ordinal733+0x24676 18, shell32.dll!SHBindToParent+0xeb0 19, dopus.exe+0x34e8c7 20, dopus.exe+0x348712 21, dopus.exe+0x539f19 22, dopus.exe+0x5295c1 23, dopus.exe+0x534134 24, user32.dll!TranslateMessageEx+0x2a1 25, user32.dll!SetTimer+0x178 26, user32.dll!SendMessageW+0x5d 27, dopus.exe+0x5c31b3 28, dopus.exe+0x59b722 29, dopus.exe+0x5d46f6 30, user32.dll!TranslateMessageEx+0x2a1 31, user32.dll!CallWindowProcW+0x9c 32, user32.dll!CallWindowProcW+0x18 33, comctl32.dll!CreateUpDownControl+0x254d 34, comctl32.dll!DefSubclassProc+0x2b4 35, comctl32.dll!DefSubclassProc+0x7c 36, dopus.exe+0x3eb52 37, dopus.exe+0xc343b 38, comctl32.dll!DefSubclassProc+0x2b4 39, comctl32.dll!DefSubclassProc+0x7c 40, dopus.exe+0x393380 41, comctl32.dll!DefSubclassProc+0x2b4 42, comctl32.dll!DefSubclassProc+0x18b 43, user32.dll!TranslateMessageEx+0x2a1 44, user32.dll!TranslateMessage+0x1ea 45, dopus.exe+0x5da447 46, dopuslib.dll!IsWow64+0x31 47, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60673 48, dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x60727 49, kernel32.dll!BaseThreadInitThunk+0xd 50, ntdll.dll!RtlUserThreadStart+0x21

I see a whole bunch of Shell stuff. What the heck are you guys hooking into there? Why is SHCreateDirectoryExW even happening as an example? WinRAR is making the folders, not Directory Opus!

I also have Avast Internet Security installed. Disabling it doesn't stop Directory Opus from destroying my CPU while WinRAR does its thing.

chrisp · November 1, 2012, 5:52am

Grr.. Where's the edit button? I meant to type "10.2.0.0" and "10.2.0.1" as the versions of Directory Opus that i've tried.

Right now i'm going back through each 0.0.5.0 version difference and testing them.

1 lister is on E:\Blah\ where the .rar files are. 5 listers are on E:. WinRAR extracting to U:. No other programs loaded. Fresh reboot.

~5% CPU usage on at least 5 threads. Roughly the same call stacks as before.

I've tried versions: 10.2.0.0, 10.2.0.1, 10.1.0.0, 10.0.5.0. Going to try 10.0.0.0 next. That's the lowest i have.

I've tried it with all but Microsoft shell extensions enabled. Also tried letting the Directory Opus shell extensions stay enabled too.

Nothing's working..

chrisp · November 1, 2012, 6:06am

Well, 10.0.0.0 is the best version yet. It at least can be alt-tabbed to while WinRAR is extracting. It uses ~0% CPU one second, then up to ~20% the next second, then repeats.

Directory Opus 10.0.5.0 and greater cannot even be alt-tabbed to while WinRAR is extracting and use a constant ~30% CPU. It also takes a good ~15 seconds after the WinRAR operation has completed for Directory Opus to become responsive again and allow me to alt-tab to it.

Again, no shell extensions but Microsoft and Directory Opus enabled. Fresh reboots. This is almost a fresh install of Windows 7 too.

chrisp · November 1, 2012, 6:23am

As a last resort, i tried uninstalling Avast! Internet Security completely.

Absolutely no difference.

Killed a couple of background tasks (ie: Fraps, FastStone Capture). Checked Directory Opus's module list in Process Hacker; nothing but Microsoft and GPSoftware modules. A couple of "{bunch of GUID stuff}.db" at the end of the list, though (located at "C:\ProgramData\Microsoft\Windows\Caches" apparently).

Only thing left i can think of is to uninstall Diskeeper, but i doubt that will help. There's really nothing else running in my process list. Nothing installed that would do anything to the Shell except a couple of disabled Adobe, Corel, and UltraEdit extensions.

Leo · November 1, 2012, 8:51am

Opus will monitor any folder you have open in a lister or tab (and subdirs if they are in flatview mode), and if you have the folder tree open then it will monitor the entire local filesystem. It has to in order to respond to changes to those folders, the same as any other file manager.

(Opus may do a few things that some file managers don't, e.g. check if directories are junctions so that listers showing the same location via an alternative name will be updated instead of going stale. We're thinking about making those things optional.)

Collections can also cause additional monitoring since they track changes to the files they contain. If you've got lots of large collections sitting around, deleting them may help.

The monitoring obviously will use some CPU, and more the more windows you have open (since each window has to do a bit of work to tell if the filesystem changes apply to it, and update itself if they do), but using up a huge amount of CPU shouldn't be happening unless there are a huge number of windows/tabs open.

You can use Preferences / Miscellaneous / Advanced: no_external_change_notify to disable processing of filesystem events.

Also in the same Preferences page, make sure notify_debug and shellchang_debug are both set to False. Enabling either of those will impose extra overheads on every filesystem change. (They are both False by default.)

Are you extracting them to places that Opus is showing in a lister or tab?

What type of files are being extracted? (Videos, nested archives, music, etc.?)

For example, if you extract video files to a directory Opus is looking at then, depending on the lister configuration and setup of the machine, Opus may inspect each video file to populate the Description or Dimensions columns or generate a thumbnail or similar, which in turn causes a video codec to be called. Some video codecs have bugs triggered by some video files that cause a lot of CPU usage. (I'm not saying that's what is happening. It's just one example of what can happen, and why the type of files being extracted, and whether or not Opus is viewing the place they are being written, can matter.)

That doesn't seem too unreasonable, and may suggest there was a problem with the Windows 8 install somewhere, but even that still seems fairly high, if we are only talking about extracting 17,077 files in total, which isn't that much.

How many listers are we talking about, and how many tabs are open in each one?

Opus provides lots of tools for quickly navigating to folders, or re-creating entire multi-window layouts or folder tab groups, which may be better than keeping so many windows open. Using those can make the desktop less cluttered as well as avoid the performance overheads of keeping lots of file displays up-to-date when they aren't really being used.

It probably isn't. I don't think Opus is calling SHCreateDirectoryExW in any of those examples. When you see something like this:

12, shell32.dll!SHCreateDirectoryExW+0xa3b

The offset there (0xA3B) is large enough that that memory location is unlikely to be part of the same function. It's just that SHCreateDirectoryExW is the nearest thing in the DLL that the tool you're using knows about, so it is reported as an offset from there. Also, if you look at the same callstack there is a sequence of other Shell32.dll functions which are apparently leading up to a call into SHCreateDirectoryExW, which doesn't make sense. The size of the offset plus the nonsensical sequence of calls, all made after crossing the boundary into Shell32.dll, mean it's almost certainly an unrelated function and not SHCreateDirectoryExW. If the tool is basing things just on the list of explicitly exported functions from the DLLs, its accuracy may be improved by configuring it to use the Windows debug symbols, but even they are missing things in places.

This is exactly the same reason that we added things like DummyDllFunctionToAvoidSymbolConfusion to Opus itself, since without that dummy symbol some other function was always showing up in Process Explorer and similar tools because it happened to be the last exported.

chrisp · November 1, 2012, 12:25pm

Thanks for the insightful response!

That may explain the high CPU usage then. If i turn the Folder Tree off or do the "no_external_change_notify" setting you suggested, it completely stops Directory Opus from using almost any CPU. We're talking virtually <0.01% CPU after those changes.

But of course, i need the Folder Tree and the updates happening.

Looking at just standard Windows Explorer, explorer.exe seems to cap at about 2% CPU when looking at the folder that i'm extracting to (with the folder tree up and looking there too). Directory Opus with just one lister seems to manage to get about 13% CPU usage during the same file operation even when i'm not open into the same drive in the Folder Tree or the listing on the right. I'm not entirely sure why Directory Opus is using so much more CPU than Windows Explorer. I guess it's all the nifty features!

This was interesting actually. I was doing some profiling, and in the first analysis, Directory Opus was chugging along mainly in a "TransformMD5" function that's part of Windows' bcryptprimitives.dll. I had a look at my collections in Directory Opus, and after removing the duplicate file results (which was set to check MD5 hashes), i couldn't get Directory Opus to do any more hashing. I haven't actually done any duplicate searches for a couple of weeks, so i have no idea why Directory Opus was doing any hashing at all.

The duplicate file results was also referencing a "lib://" that strangely doesn't exist anymore. It completely clicked with me that my collections may be the culprit, but actually the collections i created are all gone. I could've sworn that i saved them along with my settings backup when i installed Windows 8, and then reinstalled Windows 7. Regardless of that anyway, i'm pretty sure that the drive/folder that i was extracting to is not in any collections that i had created.

Yeah i had a tonne of listers open. About 10 of them, i suppose.

These were already set to false.

[quote="leo"]Are you extracting them to places that Opus is showing in a lister or tab?

What type of files are being extracted? (Videos, nested archives, music, etc.?)

For example, if you extract video files to a directory Opus is looking at then, depending on the lister configuration and setup of the machine, Opus may inspect each video file to populate the Description or Dimensions columns or generate a thumbnail or similar, which in turn causes a video codec to be called. Some video codecs have bugs triggered by some video files that cause a lot of CPU usage. (I'm not saying that's what is happening. It's just one example of what can happen, and why the type of files being extracted, and whether or not Opus is viewing the place they are being written, can matter.)[/quote]
I'm not sure any more if Opus was technically "showing" the place that i was extracting to. If you mean it had a Tree Folder open with that drive/folder eventually nested somewhere in "Computer", then yes it was "showing". But i mentioned in my later posts that i was sitting there with 1 lister in "E:\blah", and 5 listers in "E:", and i was extracting to "U:\blah\blah\blah". But if the Tree Folder being enabled means that it's tracking the entire filesystem, then i suppose my entire computer is being monitored.

The files i'm testing extracting are a mix of ".url" and ".jpg".

If there was anything wrong with the Windows 8 install, then Microsoft have a really embarrassing situation on their hands, because i installed whatever came hot off their servers on the 26th of October from a bootable USB stick (that the Microsoft 8 Upgrade tool or whatever it is, created for me). I didn't really change much at all. It was just a default Windows 8 install with almost no customisation (since i couldn't quite get the hang of the new "Metro" UI). Directory Opus is almost the first program i install too.

[quote="leo"]How many listers are we talking about, and how many tabs are open in each one?
Opus provides lots of tools for quickly navigating to folders, or re-creating entire multi-window layouts or folder tab groups, which may be better than keeping so many windows open. Using those can make the desktop less cluttered as well as avoid the performance overheads of keeping lots of file displays up-to-date when they aren't really being used.[/quote]
I guess 5 or 6 listers with only 1 tab per lister in the WinRAR tests i was doing, but before with Windows 8, it was about 10 listers.

I guess i'm not very good at using Directory Opus. I tend to just have a lot of listers open in the folders that i'm frequenting. Windows does a nice job of stacking them into one taskbar button, so it doesn't really take up that much room. I would've thought that since there's only one dopus.exe, that all those listers were linked and relying on the same data underneath. I guess they're not after all.

It probably isn't. I don't think Opus is calling SHCreateDirectoryExW in any of those examples. When you see something like this:

12, shell32.dll!SHCreateDirectoryExW+0xa3b

The offset there (0xA3B) is large enough that that memory location is unlikely to be part of the same function. It's just that SHCreateDirectoryExW is the nearest thing in the DLL that the tool you're using knows about, so it is reported as an offset from there. Also, if you look at the same callstack there is a sequence of other Shell32.dll functions which are apparently leading up to a call into SHCreateDirectoryExW, which doesn't make sense. The size of the offset plus the nonsensical sequence of calls, all made after crossing the boundary into Shell32.dll, mean it's almost certainly an unrelated function and not SHCreateDirectoryExW. If the tool is basing things just on the list of explicitly exported functions from the DLLs, its accuracy may be improved by configuring it to use the Windows debug symbols, but even they are missing things in places.

This is exactly the same reason that we added things like DummyDllFunctionToAvoidSymbolConfusion to Opus itself, since without that dummy symbol some other function was always showing up in Process Explorer and similar tools because it happened to be the last exported.[/quote]
Very interesting! Sorry to have misunderstood this whole thing. I guess i'm not very good at debugging. A little information is dangerous, huh? Here i was thinking that i was doing complicated analysis of the problem but it turns out i had it all wrong. Utterly embarrassing.

Well, i'm not sure where to go from here. My high CPU problem is not really solved. If it makes any difference, almost all of the 8 HDDs i have installed or plugged in, have around a million (or more) files and/or symbolic links on them. Positively massive amounts of files. I'm not sure that you'd see this kind of real world usage that often. Perhaps Directory Opus just needs a bit more performance tweaking for large amounts of files? Maybe there's something going on with my drivers. I've got the latest WHQL Intel RST (Rapid Storage Technology) v1.6.2.1002 drivers installed, and my HDDs + 1 SSD are in AHCI mode. I tried briefly in normal IDE mode using the default Windows drivers, but that didn't solve anything.

The only thing i can think of doing is just have less listers open, or learn how to use tabs and stuff if that'll help. Can you suggest anything else that'll help handle millions of files? Bear in mind that i don't have to actually access large nested folders to get this CPU problem to come up. It happens wherever i am on the drives. Perhaps my MFTs are overflowing like crazy and take too long to access? I don't know..

In the end.. I'm sitting here with 1 lister on "E:" with a Folder Tree up (with nothing except "Computer" expanded). I'm extracting jpgs from ~150 rar files in "F:\blah" to "U:\blah\blah\blah" with WinRAR now, and one of Directory Opus' threads is eating my CPU anywhere up to around 13%. That would actually make a lot of sense, since i have an i7-2600K, and 12.5% is one "hyperthread" or logical "core", however you want to call it. So a single-threaded operation (that has something to do with updating the lister from monitoring the filesystem) in Directory Opus is just getting stuck and overwhelmed with information. I just don't know why.

chrisp · November 1, 2012, 12:52pm

Oh.. One last thing.

I noticed that dopus.exe is doing about 1-2MB/s of I/O during the WinRAR tests, according to Process Hacker.

That seems like a lot of I/O, doesn't it?

chrisp · November 1, 2012, 12:56pm

Makes me wonder, actually.. Looking at Directory Opus' CPU time.. 8m17s total for this session. 6m9s of that is kernel time. Also, seems like a lot, right?

chrisp · November 1, 2012, 1:36pm

Here's some more performance information for you to chew on. I loaded up a fresh explorer.exe (in its own process), and a fresh single lister Directory Opus. Both were pointed to the destination of a WinRAR extraction of ~150 .rar files containing ~3.8GB of jpegs in ~17,000 files. The folder trees were both fully expanded to show updates as the sub-directories were created.

Explorer Total CPU time: 5.974s.
Explorer Kernel time: 3.478s.
Explorer "Other" I/O: 7.23MB.

Directory Opus Total CPU time: 50.965s.
Directory Opus Kernel time: 35.849s.
Directory Opus "Other" I/O: 75.92MB.

"Other" I/O and CPU time only increased during the WinRAR extraction.

Windows Explorer is in its default mode, which adjusts the listing on the right according to the content. In this case it was a normal list of details with "Filename", "Date Modified", and "Type".

Directory Opus is set to "Details" mode for the right-side content, with "Name", "Size", "Type", "Modified", and "Attr" columns.

There's a tad discrepancy there between the two processes. They should be doing approximately the same thing, right?

Leo · November 2, 2012, 11:26am

That isn't something which Opus itself calls, at least not directly. (Maybe something Opus calls then goes on to call TransformMD5 for some reason. Or maybe it's the symbols/offsets playing tricks again.)

Opus has its own internal MD5 routines which it would use to generate MD5s when it needs them (e.g. for the MD5 Checksum column or the Duplicate Files tool).

lib:// is the Libraries folder. Those are separate to Collections, which are under coll://

If you have a huge number of collections (including nested collections) then it can cause performance problems.

The drivers may vary with different hardware combinations. I've run into a couple driver problems on Windows 8, and have heard of other problems from friends. It's a bit surprising, since the desktop side of things doesn't look like it has changed much, but there do seem to be some teething problems. Whether they are related to this is unknown, but it's one possible explanation for the huge CPU usage difference when doing the same thing.

Ah okay, that's not a huge number of windows/tabs. You should be able to have that many open without running into problems.

They are in some ways, but each file display has to do some processing of its own as well.

Lots of symbolic links could be a factor. When there is a filesystem event, Windows (usually) gives the path that was used to make that change. Opus will try to work out if that event also applies to other paths it is interested in, by checking if links/junctions map to each other. I'd have to do some research to work out exactly when/how this comes into play, but it could add additional overheads to the event processing.

[quote]I noticed that dopus.exe is doing about 1-2MB/s of I/O during the WinRAR tests, according to Process Hacker.

That seems like a lot of I/O, doesn't it?[/quote]

Possibly, although 2MB/s isn't a huge amount on a HDD.

It suggests Opus (or something loaded within the dopus.exe process, at least) is reading parts of either the archives themselves or the files which are being extracted.

It might be worth using Process Monitor, filtered on ReadFile events within dopus.exe, to see what Opus is reading.

If the files being extracted are visible, it's normal for Opus to open & read a small part of the files to test their filetypes, in some cases. (Obviously, with thumbnails or columns that show metadata that comes from file contents, more of the files will be read.) And if the folder tree is set to show archives, Opus may open and read parts of archive files to confirm they really are archives.

Yes, that does seem like a lot of time churning in the kernel vs out of the kernel.

Drivers and other low-level things such as anti-virus scanners (remember there's Windows Defender as well as any extra ones) are usually attributed to kernel time, so if there is a slow-down there then that may be a symptom.

(e.g. If the Opus folder tree wants to test if one of the .rar files is an archive, and then opens it, opening it may cause a virus scanner to re-scan the archive, causing I/O to read its contents and CPU usage if something causes the scanner to take a long time. There has also been the odd case of disk-imaging/backup/repair tools installing low-level drivers which cause odd problems like that, although they typically get fixed pretty quickly in my experience.)

Sorry that it's still so much guesswork here.

chrisp · November 2, 2012, 2:25pm

[quote="leo"]That isn't something which Opus itself calls, at least not directly. (Maybe something Opus calls then goes on to call TransformMD5 for some reason. Or maybe it's the symbols/offsets playing tricks again.)

Opus has its own internal MD5 routines which it would use to generate MD5s when it needs them (e.g. for the MD5 Checksum column or the Duplicate Files tool).[/quote]
Gah.. I got tricked again! This time i was relying on something a little more reliable than Process Hacker.. Visual Studio 2012, with all the debug symbols correctly configured and downloaded.

[quote="leo"]lib:// is the Libraries folder. Those are separate to Collections, which are under coll://

If you have a huge number of collections (including nested collections) then it can cause performance problems.[/quote]
Yeah i mixed that up a bit. At the moment i don't have any Libraries except for the default "Documents", "Music", "Pictures", "Videos". I remember creating my own library at one point that included folders from many drives. But it's gone now. The "Duplicate" results collection was referring to the files i did have in those libraries. Also, i have nothing except "Find Results" and "Find Results (1)" in File Collections now.

[quote="leo"]Possibly, although 2MB/s isn't a huge amount on a HDD.

It suggests Opus (or something loaded within the dopus.exe process, at least) is reading parts of either the archives themselves or the files which are being extracted.

It might be worth using Process Monitor, filtered on ReadFile events within dopus.exe, to see what Opus is reading.

If the files being extracted are visible, it's normal for Opus to open & read a small part of the files to test their filetypes, in some cases. (Obviously, with thumbnails or columns that show metadata that comes from file contents, more of the files will be read.) And if the folder tree is set to show archives, Opus may open and read parts of archive files to confirm they really are archives.[/quote]
Ok this is where it gets really interesting. The thing about that I/O is that it's not normal Read or Write. It's "Other", (which i mentioned in my later post about Explorer using 1/10th of the CPU of Directory Opus), which i guess means that Directory Opus is not actually doing normal ReadFile calls, and it might point to something kernel-wise like i mentioned later.

I've checked this with Process Monitor and have found startling results. I filtered out everything but "dopus.exe". Directory Opus is calling a whole bunch of RegOpenKey, RegQueryKey, RegQueryValue, and RegCloseKey on, for example:

"HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\MountPoints2\CPC\Volume{5aae07ef-2193-11e2-a8a3-806e6f6e6963}",
"HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\MountPoints2\CPC\Volume{5aae07ef-2193-11e2-a8a3-806e6f6e6963}\Generation", and
"HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\MountPoints2{5aae07ef-2193-11e2-a8a3-806e6f6e6963}_LabelFromReg"

But there's a few different GUIDs involved. I'd say about 5 or so of them. Roughly the number of drives in my system, i'd say.

There's about 175,000 of each RegOpenKey, RegQueryKey, RegQueryValue, and RegCloseKey in the span of just monitoring for 20 seconds. That's a total of 911,620 "Reg*" operations.

Along with those, there's 26,859 NotifyChangeDirectory operations. 55,060 QueryDirectory operations. 55,100 CreateFile.

CreateFile is opening "u:" then "u:\blah", then "u:\blah\extracted subdir", then "u:\blah\extracted subdir\nested subdir" for seemingly every file operation.

I think i can see now where all the overhead and CPU usage is coming from. Over only 20 seconds i extracted ~6000 files of the 17,000 in the rar files. (I cancelled the operation because it was just lagging up horribly). 911,620 registry operations, 55,000 x 3 (CreateFile, CloseFile, QueryDirectory) filesystem operations. All for just ~6000 files that Directory Opus wasn't even working with.

There were no ReadFile events at all, which is not surprising.

There was only a single lister open in Directory Opus with the Tree Folder enabled. "U:" was not expanded in the tree at all, and was not showing in the details on the right. I was on a completely different drive with that lister.

[quote="leo"]Yes, that does seem like a lot of time churning in the kernel vs out of the kernel.

Drivers and other low-level things such as anti-virus scanners (remember there's Windows Defender as well as any extra ones) are usually attributed to kernel time, so if there is a slow-down there then that may be a symptom.

(e.g. If the Opus folder tree wants to test if one of the .rar files is an archive, and then opens it, opening it may cause a virus scanner to re-scan the archive, causing I/O to read its contents and CPU usage if something causes the scanner to take a long time. There has also been the odd case of disk-imaging/backup/repair tools installing low-level drivers which cause odd problems like that, although they typically get fixed pretty quickly in my experience.)

Sorry that it's still so much guesswork here.[/quote]
No worries. This is taking up a lot of my time fiddling around, but if it gets worked out in the end, i'll be happy with the time spent.

In regards to the kernel time. I don't know if i should bother with the debugging any more, since i seem to be doing it wrong.. But, Visual Studio profiling has told me to monitor SysCall to figure out what's being done in the kernel. Once i can figure out how to do that, i'll check it out.

Unless that would be a waste of time? The boatloads of registry and file operations going on for just 6000 extracted files from WinRAR seems like a lot of overhead for something that isn't being displayed at all.

Keep in mind one thing that i haven't mentioned yet.. And that's that Directory Opus actually has a bug that stops stuff in the Folder Tree from updating occasionally. Sometimes i'll go to the tree and have a look for something i just extracted, only to not find it. I have to hit F5 in the tree or collapse and expand the parent to get it to actually show. So if Directory Opus is spending so much CPU updating this stuff, you'd think that it should at least display it right, right?

chrisp · November 3, 2012, 4:05am

Well, i've been thinking about this. Without knowing exactly what Directory Opus does during external filesystem updates, which of course would be trade/commercial secrets, i can't really do much work over here to determine where all the CPU is going.

I've done a few simulations with my own code. Directory Opus takes 1m42s total CPU time of which 1m34s is kernel time, to process the filesystem updates that happen when i extract those ~17,000 files with WinRAR. Using Process Monitor to see what Directory Opus is doing, i've managed to find a few things out.

During ~576,000 iterations (which is what Directory Opus does during the whole extraction) of registry operations which consist of opening HKCU/Software/Microsoft/Windows/CurrentVersion/Explorer/MountPoints2, opening a GUID subkey of that, getting the DWORD value of "Generation", then closing the 2 keys opened.. My simulation manages to sustain ~540KB/s I/O and use 16.629s of CPU time, of which 12.776s is kernel time. I can't replicate the I/O during subsequent runs.. Seems to be some sort of registry caching going on.

Following that, and this is a completely wild guess because of a) It's hard to filter and find with Process Monitor exactly what functions Directory Opus is calling, and b) I mentioned before that what Directory Opus does is a commercial secret.. ~220,000 iterations (the number of "CreateFile" operations that Process Monitor tells me Directory Opus does) of FindFirstFileW on a subdir of the extracted folders then a FindClose.. We get a sustained ~2.2MB/s of I/O and 20.18MB of total "Other" I/O, which may ring a bell because it's what i mentioned before that Directory Opus is peaking at. Also, 9.360s CPU time, of which 8.424s is kernel time.

Getting there, but still there seems to be a lot more that Directory Opus is doing. We're looking at only 25% of the CPU time accounted for.

I'm not sure where exactly the inefficiencies are.. In the Windows kernel? If so, then why is Windows Explorer taking 1/10th of the CPU time to display exactly the same thing that Directory Opus is?

I think you need to look at how much Directory Opus does during every filesystem update. Do some profiling etc. I'm not sure if you have a testbed there that could simulate my HDD/system conditions though.

The only thing i can think of as a suggestion is to somehow coalesce reactions to filesystem updates. Perhaps don't do something every single time it updates.. Wait a little bit (~100ms?) and see if there's more happening in the same folder, then update it in one go.

I'm still thinking about this. It is literally giving me a headache though, and i feel as though it's work that i shouldn't need to do. You guys need to get all over this and work it out.

chrisp · November 3, 2012, 7:07am

Ok.. Just noticed something ridiculous here. Directory Opus takes 1m42s CPU (1m34s kernel) to update its internal whatever when WinRAR is extracting, right? Nothing else is happening on the computer at the time. Nothing else to process.

WinRAR uses 44s of CPU time (18s kernel) to actually extract those files, doing a total of 4.08GB write I/O and 3.86GB read I/O.

Also, Process Monitor registers 5,389,594 events during the extraction for Directory Opus, while WinRAR only has 820,443 events.

Something is really, really wrong here with Directory Opus. How can Directory Opus take 230% of the CPU time to just look at the names of what was extracted, that WinRAR takes itself to actually extract (and decompress) the files? Decompress! Arguably the most CPU intensive operation on any computer!

I was actually thinking that something in my install or system might possibly be responsible for Directory Opus chewing up my CPU, but after this.. Barely a chance. I'm open to suggestions still, but i'm really doubtful now. This needs to be looked at ASAP.

ashmid · November 5, 2012, 1:06am

Hi chrisp,

I've experienced similar difficulties with Directory Opus - with a disproportionate amount of time spent in the background DOpus process vis-a-vis time spent in other file-intensive applications. I've written about some of my experiences in this thread here (although not to the level of detail that you have explored!):

The solution that I've found, for the meantime, is that I always run Dopus in "no-external-change-notify" mode, and I use an AHK script which automatically sends a "Refresh All" command to Dopus whenever I move a lister to the foreground. (Yes, DOpus is that good - unparalleled in so many ways - that I stick with Dopus despite this workaround!) I included the source for the AHK script in my post referenced above.

Overall, I find this to be an excellent workaround. When Dopus is in the background, it doesn't interfere and doesn't clock any context switches. And when I do bring it to the foreground, it's up-to-date within milliseconds.

The one problem with this setup is that the "Refresh All" command also resets a few other things: it resets the file selection; the file filter; and the GO command line (that is, any half-typed commands are closed up).

I've thus asked the developers whether they could add a command to perform a "Soft Refresh": a command which would cause Opus to reread the list of files within the directory, but which would (file-list permitting) still leave intact the selection, the filter, and the Go bar.

I believe that if a Soft Refresh were available, then the ideal way of running Dopus really would be the "no-external-change-notify" mode, with automatic Soft Refreshes.

In any case, chrisp, I found your messages here very informative and insightful (and I found your spelunking stamina to be inspirational!) Thanks!

chrisp · November 5, 2012, 7:15am

ashmid,

This is positively the best news i have received in quite a while! Thank you for taking the time to notice and read my thread, and respond. I thought i was completely alone on this, constantly doubting myself and wondering if i had done something embarrassingly wrong somewhere.

I just read your entire thread that you linked to, and it is also quite insightful and inspiring!

After setting "no_external_change_notify" to True, the AutoHotKey script you wrote does the trick just nicely!

Of course, it's positively a kludge (but a good one!) that could have been unnecessary if Directory Opus just performed well in the first place.

On a side note, related to your thread, i'd like to report that my Context Switch 1s delta gets up to around 7000 max. Currently i have 4 listers open and a custom-programmed 14 process/thread file and network operation going on in the background, and as you can imagine, Directory Opus is both sluggish itself, and making my system sluggish.

In my WinRAR test scenario (which i won't bother detailing again because i believe i've repeated it enough in the thread), i'm looking at up to 25,000 (!!) context switches per second. This takes a mighty toll on my system.

So, anyway. The awesome AHK fix you have created works well! Although i'm wondering what the impact would be on folders with even a few thousand files/folders in them. These folders on my computer take about 10 seconds to read with Directory Opus. Although the AHK thing appears to refresh the folder near instantaneously, i'm wondering if that's because the folder's content's details are in the filesystem cache. If i left that folder alone for, say, an hour, and it had been emptied from the cache, would Directory Opus take another 10 seconds to re-read the entire folder when i alt-tabbed to it? That would be horribly annoying.

Of course, that may make me look like an idiot for having a few (or more) thousand files/folders in the same folder, but i'm already one step ahead there, and almost every operation that i do involving up to millions of files, neatly segregates them into 1000-2000 files per folder. Reading folders is a lengthy process sometimes. Perhaps NTFS is just no good for this? Maybe Linux people and their EXT3 (or whatever they use now) really do have a reason to laugh at Windows users?

On the other hand, as Leo suggested in the other thread and which i had already done before reading it, i've run Windows Explorer (explorer.exe) alongside Directory Opus. And, yes, as i mentioned before, Explorer uses 1/10th of the CPU to display the exact same information. You seemed to have similar findings. Something that i wasn't going to mention was that competing file managers that i have tested also use approximately only 10%-25% of the CPU of Directory Opus.

I know i post a lot and i may seem impatient to get this fixed. I realise that it may take a long time to fix. But, just some acknowledgement or a promise of eventual progress would be nice. These days i tire of the usual dismissive nature of support responses, usually making it seem like the user has misconfigured something, or at least trying above all else to see if we "broke" it somehow. I think everyone forgets that we really are just users. We are using the software. We didn't make any of it - Windows, Directory Opus, drivers, actual hardware, whatever. At most we usually just click "Next" a few times. If it's not performing right then how much of the blame can really lay on us?

chrisp · November 5, 2012, 7:52am

Just posting a confirmation of what i feared..

The AHK script, when alt-tabbing to a Directory Opus lister displaying a folder of 3,929 folders, just made Directory Opus re-scan the whole folder and it took about 30 seconds.. Sure enough, the folder was obviously emptied from my filesystem cache. The script works great on small folders though! Thanks again, ashmid!

Makes me remember something i've always longed for in Windows: an MFT cache (or whatever it would be called). I've given FancyCache a go, but it doesn't let you specify single files/folders (ie: "$MFT"). Grrr i hate scanning drives with millions of files on them!!

BetterRed · November 5, 2012, 8:57am

chrisp - I have been following your thread, I think we have some things in common, but there's not been anything worthwhile I could add. So this is more by way of a "You are Not Alone" message - like ashmid I am in awe of your tenacity.

On one of my two major projects I have about 8,000 end node folders, each of which contains

[ul]3-4 folder shortcuts (which are always hardlinks) some shortcuts have 1023 hardlinks so I've had to create a 2nd shortcut for those folders.
1-6 text files some of which may be hardlinks
1-3 weblinks, 1-2 ftp links
1-5 image files
1-2 archives - normally zips, occasionally multi-gig, multi0part rars - but mainly 100-500MB zips [/ul]
Each end node is targeted by 3-4 symlinks in 4 other hierarchies - so there are about 28,000 symlinks.
Remember all those shortcut hardlinks - they target the parent folders that hold the 28,000 symlinks (I call them network navigators )

The other project is a about 30% larger, but its not as dynamic in terms of adding new entities.

I am not experiencing the sort of issues you are facing - presumably because I am not running massive unarchive operations, I only need to do that when TPTB lose the main database, and I'd drive that process via existing bespoke scripts and programs. I'd have Opus shut down along with just about everything else - including the phone

One other difference - I only run one Lister - a static tree, two side by side browser panels each with 10 identical tabs - in details mode

So... if there is anything more you'd like know, or you'd like me to try, feel free to ask either here or in a PM. I'm on Sydney time.

Cheers BR

BetterRed · November 5, 2012, 9:05am

Sorry

"each with 10 identical tabs" is ambigious. The 10 tabs target 10 different drives and folders. The same 10 ten tabs appear in the Left & Right panes.

Oh, and I use WinRAR as my production Archiver.

BR

Leo · November 5, 2012, 9:42am

We plan to do some profiling and improvements or new options (where needed) to the file change code in the future, but it's not going to change over night and is not causing problems for most people.

I think it's best to just wait for that to happen, as a we don't have time to respond to the volume of information in this thread, and a lot of it is not actually relevant to be honest.

(For example, registry lookups cost virtually nothing as the registry is designed to cache things very well. And the CreateFile etc. API calls that Process Monitor and similar tools log are often make it look like a file is being read/modified/created when it isn't really, because various high-level OS functions do that internally just to check a file's metadata, and Process Monitor (etc.) are logging the low-level API calls, not the high-level ones.)

We will look into this, but I don't think there's anything to be gained from continuing here as it is taking too long to respond to all the conjecture and what we really need to do is just profile it ourselves and work out if any of the things Opus is doing can be avoided or made optional for those who find it causes problems on their systems, for whatever reason(s).

chrisp · November 5, 2012, 11:36am

Conjecture? I'm a little insulted, after all of the time and effort that i have put in to this.

As in my example above, i have coded and compiled (since i am a software developer) an exact simulation of Directory Opus' registry calls during the WinRAR extraction process. Unless you're saying that Process Monitor is lying and that those 576,467 x 4 registry operations are not happening, then the 16 seconds of CPU time that it takes out of Directory Opus' 40 seconds is most definitely not "virtually nothing". It's as much as 40% of the CPU time. We're talking about the exact same sequence of registry actions repeated 576,467 times. It is also most definitely cached by Windows since it is repeated in rapid succession.

Of course, there's the semantics argument where one could maintain that this is all conjecture because the simple truth of it is that only GPSoft has the actual source code to Directory Opus. That doesn't make the research any less scientific in nature, however, and therefore not "conjecture".

But i won't argue the point more because this will clearly devolve into a "No, it's your fault!" thread. Thank you for hearing our pleas and for responding, Leo. I trust that you will follow through on your word and there will be some profiling and investigation done behind the scenes.

Thank you, also, to both ashmid and BetterRed for your contributions! Really, it sounds like though we have much in common in regards to Directory Opus' poor performance, we're actually somewhat diverse in our filesystem applications. This diversity should be proof that it's perhaps not all that true that this is "not causing problems for most people". In fact it's probably more true, for example, that "most people don't have the means to notice or measure performance issues, let alone post in a public forum about it".

Just a quick clarification here, by the way. Process Monitor imparts hefty penalties for monitoring the system, and the initial measurement of Directory Opus' CPU time at 1m42s is actually closer to about 40 seconds with no Process Monitor running. Still, that's roughly a 1:1 ratio of CPU time for WinRAR and Directory Opus. This would actually lend further truth to the simulations i have conducted, since the simulated CPU time is very near the actual measured CPU time of Directory Opus.

Finally, please remember that this is for the good of Directory Opus and all of its users. In no way is this a campaign of malice.