[quote="leo"]That isn't something which Opus itself calls, at least not directly. (Maybe something Opus calls then goes on to call TransformMD5 for some reason. Or maybe it's the symbols/offsets playing tricks again.)
Opus has its own internal MD5 routines which it would use to generate MD5s when it needs them (e.g. for the MD5 Checksum column or the Duplicate Files tool).[/quote]
Gah.. I got tricked again! This time i was relying on something a little more reliable than Process Hacker.. Visual Studio 2012, with all the debug symbols correctly configured and downloaded.
[quote="leo"]lib:// is the Libraries folder. Those are separate to Collections, which are under coll://
If you have a huge number of collections (including nested collections) then it can cause performance problems.[/quote]
Yeah i mixed that up a bit. At the moment i don't have any Libraries except for the default "Documents", "Music", "Pictures", "Videos". I remember creating my own library at one point that included folders from many drives. But it's gone now. The "Duplicate" results collection was referring to the files i did have in those libraries. Also, i have nothing except "Find Results" and "Find Results (1)" in File Collections now.
[quote="leo"]Possibly, although 2MB/s isn't a huge amount on a HDD.
It suggests Opus (or something loaded within the dopus.exe process, at least) is reading parts of either the archives themselves or the files which are being extracted.
It might be worth using Process Monitor, filtered on ReadFile events within dopus.exe, to see what Opus is reading.
If the files being extracted are visible, it's normal for Opus to open & read a small part of the files to test their filetypes, in some cases. (Obviously, with thumbnails or columns that show metadata that comes from file contents, more of the files will be read.) And if the folder tree is set to show archives, Opus may open and read parts of archive files to confirm they really are archives.[/quote]
Ok this is where it gets really interesting. The thing about that I/O is that it's not normal Read or Write. It's "Other", (which i mentioned in my later post about Explorer using 1/10th of the CPU of Directory Opus), which i guess means that Directory Opus is not actually doing normal ReadFile calls, and it might point to something kernel-wise like i mentioned later.
I've checked this with Process Monitor and have found startling results. I filtered out everything but "dopus.exe". Directory Opus is calling a whole bunch of RegOpenKey, RegQueryKey, RegQueryValue, and RegCloseKey on, for example:
"HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\MountPoints2\CPC\Volume{5aae07ef-2193-11e2-a8a3-806e6f6e6963}",
"HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\MountPoints2\CPC\Volume{5aae07ef-2193-11e2-a8a3-806e6f6e6963}\Generation", and
"HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\MountPoints2{5aae07ef-2193-11e2-a8a3-806e6f6e6963}_LabelFromReg"
But there's a few different GUIDs involved. I'd say about 5 or so of them. Roughly the number of drives in my system, i'd say.
There's about 175,000 of each RegOpenKey, RegQueryKey, RegQueryValue, and RegCloseKey in the span of just monitoring for 20 seconds. That's a total of 911,620 "Reg*" operations.
Along with those, there's 26,859 NotifyChangeDirectory operations. 55,060 QueryDirectory operations. 55,100 CreateFile.
CreateFile is opening "u:" then "u:\blah", then "u:\blah\extracted subdir", then "u:\blah\extracted subdir\nested subdir" for seemingly every file operation.
I think i can see now where all the overhead and CPU usage is coming from. Over only 20 seconds i extracted ~6000 files of the 17,000 in the rar files. (I cancelled the operation because it was just lagging up horribly). 911,620 registry operations, 55,000 x 3 (CreateFile, CloseFile, QueryDirectory) filesystem operations. All for just ~6000 files that Directory Opus wasn't even working with.
There were no ReadFile events at all, which is not surprising.
There was only a single lister open in Directory Opus with the Tree Folder enabled. "U:" was not expanded in the tree at all, and was not showing in the details on the right. I was on a completely different drive with that lister.
[quote="leo"]Yes, that does seem like a lot of time churning in the kernel vs out of the kernel.
Drivers and other low-level things such as anti-virus scanners (remember there's Windows Defender as well as any extra ones) are usually attributed to kernel time, so if there is a slow-down there then that may be a symptom.
(e.g. If the Opus folder tree wants to test if one of the .rar files is an archive, and then opens it, opening it may cause a virus scanner to re-scan the archive, causing I/O to read its contents and CPU usage if something causes the scanner to take a long time. There has also been the odd case of disk-imaging/backup/repair tools installing low-level drivers which cause odd problems like that, although they typically get fixed pretty quickly in my experience.)
Sorry that it's still so much guesswork here.[/quote]
No worries. This is taking up a lot of my time fiddling around, but if it gets worked out in the end, i'll be happy with the time spent.
In regards to the kernel time. I don't know if i should bother with the debugging any more, since i seem to be doing it wrong.. But, Visual Studio profiling has told me to monitor SysCall to figure out what's being done in the kernel. Once i can figure out how to do that, i'll check it out.
Unless that would be a waste of time? The boatloads of registry and file operations going on for just 6000 extracted files from WinRAR seems like a lot of overhead for something that isn't being displayed at all.
Keep in mind one thing that i haven't mentioned yet.. And that's that Directory Opus actually has a bug that stops stuff in the Folder Tree from updating occasionally. Sometimes i'll go to the tree and have a look for something i just extracted, only to not find it. I have to hit F5 in the tree or collapse and expand the parent to get it to actually show. So if Directory Opus is spending so much CPU updating this stuff, you'd think that it should at least display it right, right?