GP SoftwareTwitter
Opus FAQsManualCommandsObjects

Thread count


#3

Hi Leo,
1] XTM Task Manager reports that the overwhelming majority of the threads are stuck in dopuslib.dll!DoesExeFileContainZip+0x60808, in a wait state.
2] HOWEVER, XTM also reports that each of these aforementioned threads has accumulated tens of millions of context switches, and the CS count is climbing by the thousands even as I watch them (while opus is only running in the background!). Thus, although these threads are not actually doing anything, they all seem to be waking up many times every second, only to go right back into their wait states. Now, if you take a single thread and have it wake up again and again there won't be much impact, but when you have 100 threads, and they are each requiring thousands of context switches every second, the performance impact is considerable. So something is seriously wrong here, I believe. (Overall, XTM reports that dopus.exe has hit one billion context switches on my machine, after running for 6 days - a number rivaled on my machine only by photoshop.)
3] I don't think that 16 instances is "crazy". I prefer instances of the program rather than tabs, and each one of my instances has a single tab (or two tabs, one on each side). I realize that "the opus way" is to use lots of tabs rather than lots of instances, but at the end of the day I only have 20 or so listers open between those instances, and, believe me, I use all of them regularly.

  • Avi

#4

dopuslib.dll!DoesExeFileContainZip+0x60808 <-- That is a red herring to some degree. The offset is a long way off that function, and Process Monitor is only reporting it as DoesExeFileContainZip because it doesn't have symbols for anything nearer.

If they're waking up and going back to sleep instantly, they are probably either UI threads (i.e. threads for windows) or threads to handle file change notifications. (Opus monitors the file system and passes changes that may interest each window to that window's thread. The thread will then quickly work out if it really is interested in the event and either process it or throw it away. On a multi-core system, it's probably more efficient than using fewer threads.)

"The Opus way" is to use Opus however you want to use it. :slight_smile: I don't personally use tabs very often, but I'd soon fine 16 windows too much clutter as well. I tend to close folders when I'm no longer using them, and re-open them when I need to, with some buttons and hotkeys (and the recent and favorites lists) to quickly take me to things I use a lot. Much easier than keepign windows around that aren't being used, IMO. But, of course, it depends what you're doing and how you like to work.

Whether Opus is running or not, and regardless of how many Opus windows I have open, PerfMon shows the total system context switches as about 2000 per second when the machine is fairly idle, and it goes up to 5000 per second when I just start moving the mouse around, and 20,000 per second if I start resizing the PerfMon window. Therefore, accumulating a billion context switches over almost a week does not seem excessive, even for a single process, if that process is being used (or even if it just has the mouse gliding over its windows at lot, or similar UI events, which trigger lots of context switches).

Are you seeing slow performance on your machine? If not then I honestly think you're looking for problems where none actually exists, which seems like a waste of time. Any program that you use for a sustained period, and do not exit and restart between uses, will easily accumulate a large number of context switches over a week, as Photoshop did in your case as well.


#5

Leo,
First of all, I did detect a sluggishness in my machine, and then I went to task manager to see what might be taking up the processor. Although I didn't have any processes using large percentages of CPU power, I did have the high context switch count here, which seemed a likely culprit (as Mark Russinovich has famously shown, and as you clearly also know, threads that have high numbers of context switches, but which generally return immediately, will show a 0 in the CPU usage column, but will nevertheless impact upon performance, because the actual context switching takes many [thousands] of processor cycles, even though these cycles are not attributed to the CPU usage count of the particular process).
As you correctly note, it's hard to learn anything from accumulated context switches, so let's look at the CS delta. Without any opus windows in the foreground, and without moving the mouse or touching the keyboard, dopus.exe is reporting an average of about 15,000 context switches every second on my machine. This is far, far more than any other process on my system.
Now, there is quite a bit of file activity going on on my system in the background, and it's possible that this is the culprit, since you note that the dopus threads are monitoring all file activity. So here I'd like to ask:
1] Is it possible to test this? Is there a debug switch or setting that would allow me to turn off the dopus hook into the file activity monitor in order to examine the impact?
2] Overall, it seems to me that the constant file monitoring is not necessarily a feature that will be desirable all the time, especially since it seems that each one of the dopus windows is doing its own monitoring. In my case, I would like to be able to perform heavy file activity on my machine without incurring the penalty of having all my dopus windows hook that activity. In fact, in the usual case, all I want dopus to do is to refresh any given lister when that lister comes into the foreground. Regarding listers that are in the background, I'd much rather have them wait around without hooking anything, until such time as I bring them to the foreground.
So, I wonder: would it be possible please to provide an option for users to turn off the constant file monitoring?


#6

15,000 context switches per second when idle is certainly worth looking at, I agree. I get much less than that on my machine (more in the region of 15 rather than 15,000, although there isn't much disk activity going on in the background).

You can turn on Preferences / Miscellaneous / Advanced: no_external_change_notify to disable processing of filesystem events to see if that changes things.

Why not just close the windows if you aren't using them? :slight_smile: You can use layouts to enable you to quickly re-open sets of windows.

(By the way, the folder tree causes more processing as it has to monitor the entire filesystem for changes, not just the drive or directory that is open.)


#7

Hi Leo,
Thanks for your suggestion. Setting "no external change notify" drastically lowered the CS delta. Now, instead of 15,000 context switches per second, I am down to under 100 context switches per second - and that's with 16 windows open!
At this point, then, I think I'm going to make the "no external change notify = true" to be my permanent setting. Is there anything I should be aware of if I am going to be using this setting on a permanent basis?
One thing I've noticed is that with this setting, the lister never updates automatically, even when I switch the lister to the foreground. That is, if I use the command prompt to copy a file into a directory, and then I go back to the lister with that directory, the new file won't show up until I hit F5.
This situation is still better than what was happening previously, but I'd like to put in a feature request for Dopus to handle this situation better. Specifically:
1] With "no external change notify" set to true, I believe it would make sense for dopus to do an automatic refresh of the directory when the lister is brought to the foreground and becomes the active window.
2] Further, if possible, I would suggest that even with "no external change notify=true", dopus should still monitor filesystem events so long as dopus is the active foreground application, releasing the hook when it goes into the background. When I am actively using dopus I do want to use my full processor power to be dedicated to the lister. It's just when it's in the background that I would like to avoid the impact on the performance that the file system hook entails.
(I understand that the no external change option may be viewed as just a debug setting, and not worthy of further tweaking; however, given the drastic impact of the file system hook for people who have constant file activity going on in the background [I imagine, for instance, that anyone running a server would have a similar issue], I believe that it would be reasonable for dopus to treat the "no external change notify=true" option as a normative option for users.)
By the way, I would like to follow up your suggestion to "use layouts to enable you to quickly re-open sets of windows". As a fairly new convert to directory opus (I used xplorer2 up until a few weeks ago, when I became enamored with the speed and slickness of Dopus), I'm not quite sure what that means. Can you please direct me to a documentation page or tutorial that would give me a better sense of how to do what you are suggesting?
Thanks,
Avi


#8

That's what the option does. It is intended more for diagnostic purposes (as we have used it here today) than to be left on all the time.

(There's another option which disables notifications only on network drives, which some people need to work around things like bad NAS devices which constantly send changes for no reason.)

That wouldn't be great on large and/or slow directories. Every time you flicked from Opus to another window and then back again, Opus would re-read the entire directory, adding a delay each time you switched back and losing your file selections and scroll position. It'd be like pushing F5 every time you clicked on Opus.

We do have some improvements to the change notification system planned for a future version, but whether or not they help in your situation is hard to guess. One or two people have had a problem where there is so much filesystem activity from other processes, and on devices like network drives where it can take a long to process each event, that new events happen faster than the old ones can be processed. We plan to detect that happening and have the lister stop processing events accordingly (with some UI to alert you and allow you to turn processing back on when it suits you).

After seeing the the context switch delta reduced, did you also see your actual performance problem go away?

What is causing all that background filesystem activity? It's possible that whatever is accessing all those files non-stop is causing the problem, rather than (or as well as) Opus responding to the changes. Be careful not to confuse a symptom with the cause.

If you want to see which files are being accessed to trigger all that activity -- which may in turn point to what is accessing them -- you can make Opus output a list of the events it is responding to. Details are in the 2nd last section of this guide.

Layouts are described here: gpsoft.com.au/help/opus10/defaul ... youts1.htm

The video about working with multiple windows, in the tutorial section here at the forum, goes into them (and other things) in some detail as well. (The video is a little old but the concepts haven't changed.)

(BTW, you can use blank lines in forum posts without any problems. It won't submit the form or anything like that, and you don't have to remember to use Shift-Return like on Facebook and similar sites. Adding them between paragraphs will turn a wall of text into something much easier to read.)


#9

Hi Leo,
Thanks for your quick response.

1] Yes, the sluggishness problem has gone away. It seems clear to me that the very intense file system hooking was impacting negatively and palpably upon my system.

2] I intentionally run a background process that does lots of text and image processing on large batches of files; this process runs often, and around the clock, on my machine. With a multi-core machine, I'm happy to dedicate one core to this long and constant task.
You've talked about symptom and cause, but in this case it is clear to me that the problem is Opus, not my background file processing. It would obviously be ludicrous to suggest that in order to have Opus run smoothly I should avoid running important background tasks. Rather, Opus should be able to run smoothly on machines even when there is a lot of file activity going on in the background, because virtually all of the file activity is irrelevant to what my specific Opus listers are showing. I am sure you would agree that one shouldn't have to worry about incurring 15,000 context switches per second from a file manager when running a background processing activity.

3] In terms of the refresh issue: I understand that my suggestion is the equivalent of having an automatic "F5" keypress whenever shifting to opus. Yet, I truly believe that that is what is called for in this situation. If I am going to turn off the file system hook, I will want to press F5 whenever switching a lister to the foreground, in order to make sure that things are up to date. The refresh is extremely quick, I find, on my machine. And it would be much much more convenient if Dopus would do it automatically.

I agree that when used as a debug setting, the "no external change notify=true" setting does not need to enable an automatic F5. However, from the foregoing discussion, I believe there is a very strong argument for adding this as a feature, as follows:

a] People who run background processes with lots of file activity will not want have the file system hook, because the performance impact can be significant.
b] For such people, an automatic F5 when switching a Lister to the foreground is a major usability and productivity feature.
c] Such people are likely running on high-end machines in the first place (otherwise they wouldn't want to run intense background processes), and thus for these people the extra F5 won't incur a major delay. And I'm quite sure that in such situations the delay incurred by the extra F5 will be far less of a performance penalty than having all file system activity hooked all the time, with all the context switches that that incurs.

Thus, I humbly submit my request for an added feature to optionally allow users to have an automatic refresh performed whenever switching to dopus.

Thanks for the links about layouts and the tip about the blank lines.

Sincerely,
Avi


#10

If you close all but one of your 16 windows, are things still sluggish?

How many files/folders are in the folders that the 16 windows are monitoring; are they huge folders or fairly small ones?

Do you have the Folder Tree turned on in the windows?

Any of the windows in Flat View mode or anything else unusual?

Are any of them monitoring the dirs where the background process outputs its results?


#11

Another thing worth trying, in addition to what I just posted above:

  • Open Windows Explorer instances pointing at the same directories and see if that has similar behaviour, since Explorer has to monitor the folders as well.

#12

Hi Leo,
The comparison to Windows Explorer is particularly instructive. I opened up 16 explorer windows, including one pointed at the directory with all the file activity. Each of these explorer instances is in its own process, and each one gets between 10 and 20 context switches per second. Overall, only 240 context switches per second for all 16 windows. That's how I expect my background processes to act.
In contrast, DOpus, as we saw, takes 15,000 context switches per second for the 16 windows (whether or not I have a window pointed at the folder with all of the file activity).

  • Avi

#13

You missed all my other questions. :slight_smile:

If I open 16 Opus windows I don't see anywhere near what you're seeing, so there is still more to understand of this issue before we can be sure what's causing it or why, or do anything about it.


#14

Hi Leo,
Here are the answers to the rest: I'm not using flat view; I do have the trees open; the context switch count is just as high whether or not they are open to the directory that is generating the files; the folders have moderate amounts of files (most of them under 100; some in the 500-1000 range; and one or two of them have a couple of thousand files). If I close all but one of the windows the count goes way down, but it's not an effective way to work for me.
Leo, at the end of the day, I'm very happy working with the noExternalNotify mode. I've been doing that the last couple of days, and its been a pleasure. The system is no longer sluggish. I get all the wonderful benefits of Opus without the drain that I was experiencing. But the one very frustrating part is that I often forget to hit F5 when switching back to an old window, and then the file list is puzzling until I remember that.
Hence, I'd like to persist with this hopeful and humble request for this addition to DOpus: the option to have a given window refresh automatically whenever it is brought to the foreground.
If you'd like to persist debugging the issue, to get to the bottom of the high context switch, I'll be happy to comply with providing any debug logs or running special debug versions or the like. But, as you note, you are not reproducing the problem on your system; and, at the end of the day, noExternalNotify is just fine for my purposes, and I'd be most happy continuing with this mode, with the proviso that the automatic refresh were provided as an option.
A month or two back, there was a debate on the BitsDuJour website about Dopus versus other programs. One of the proponents of a different application wrote in noting the very high number of updates and new features that Dopus provides per year, claiming that that was a sign of instability. I saw it as precisely the opposite - the high number of updates indicates that Dopus is responsive to user requests, and does not simply hand over a closed feature set. From the day I installed Dopus I've had the pleasure of the top notch answers and responsiveness that you provide on this forum - completely unparalleled in my experience with other file managers from other competitors - and now, for the first time, I am in need of a specific small modification to make Dopus usable given the condition of my system. And, I strongly believe, this will provide an attractive option for many other power users as well. Even if Dopus is not using 20,000 context switches on your system, it is still generating thousands per second, and I am sure that there are many other users out there who were prefer to lighten the load on their systems, just refreshing the Dopus windows upon activation.
I do hope that you will consider including this feature soon on one of those many updates that Dopus prides itself upon.


#15

Hi Leo,
Any progress on this item? Like I said, if you'd like me to perform any further tests to debug the thread count issue, I'm happy to comply; ideally, however, I'd like to just have the ability for the auto-refresh whenever a dopus window takes the foreground (preserving the selection, if possible).

  • Avi

#16

It's not something that we can fix that quickly, especially when we still can't reproduce it and it doesn't seem to be widespread.

We plan to refactor some of the change notification code to improve some things in the future, but I cannot say when that will happen.


#17

OK, in that case, what about the auto-refresh option? Is that a possibility? As I argue above, I think it's a capability that would be useful not just in my specific situation; rather, such an option would be useful for anybody who has limited processing power and wants to reduce the load imposed by opus upon the system, by turning the no-change-notify mode from a debug mode to a really practical and useful and efficient mode of operation.


#18

No, we won't be adding that, for reasons explained above.


#19

Leo, I don't follow the logic. Why won't you consider adding this option? Correct me if I am wrong, but from a programming perspective it's a relatively minor item - that is, it doesn't require refactoring major segments of code, but rather it is simply invoking the refresh routine when the window gets focus.
It would solve the problem that I have reported in a satisfactory manner, without requiring a refactoring of the change notification code.
And even if you cannot duplicate tens of thousands of context switches, you did duplicate thousands of context switches - and that is something that does impose a real load on a system. Indeed, any sort of change-notify system will necessarily impose a certain amount of load on a system. And it makes a lot of sense that users who wish to avoid that extra load - say, users with netbooks - would want to be able to work in a no-change-notify mode. This minor addition would make that wish a reality.


#20

From what I understand, Opus installs a system-wide hook for all file system events. The big advantage that we gain from this is that we can be sure that all listers are completely up-to-date at all times. However, this also has a certain impact of performance. The majority of filesystem events that occur are not directly relevant for Opus, yet each filesystem event will nevertheless incur a context switch to Opus's process, and these context switches can add up: each a context switch adds not just a few extra instructions, but rather many hundreds of instructions (and a user/kernel transition) to each filesystem event.

Now, as Leo has shown in the past (Thread count), on normal machines, the overall impact of the system-wide hook is barely noticeable. However, there are a couple of cases - which are becoming more and more common - in which I believe the impact is significant:

(1) Netbook users are generally pressed for processor cycles to begin with - netbooks are not built for performance, and processor cycles are at a premium. For these users, the extra cycles may well be an issue. Consider the following: if a netbook user runs Opus and then minimizes it, spending an hour in another application completely, over the course of that hour every single file transaction will be hooked and an extra context switch to Opus will be incurred. That is: just having Opus running in the background, without even switching to it once, will end up incurring a penalty of many extra processor cycles during the course of normal computer usage.

(2) Conversely, many machines these days come with upwards of 10 cores; but this also poses a problem, because users are now likely to run many file-intensive processes simultaneously. That is: although with single and dual-core machines users tended to do one thing at a time, leaving intense background tasks for overnight processing, multi-core machines have changed this situation. As an example, in one situation I needed to run an OCR program over some 50,000 documents. Previously I would have waited, but now, on our new 12-core machine, I simply ran it in the background, allowing it to use a couple of the cores, while I continued my usual work. The problem is, though, that Opus' system-wide hook now hooks into all of this activity simultaneously - all filesystem events from all 12 cores - and the cost is now palpable. As I noted in the aforementioned thread, under these conditions Opus' context switch count regularly hits 15,000 switches per second on my machine. I believe that anyone running multiple file-intensive processes on multiple cores will incur such an issue.

Fortunately, the good folks at Opus have provided us with the "no_external_change_notify" option (Prefs / Misc / Advanced), which turns off the system-wide hook completely. With this option in effect, Opus takes virtually no toll whatsoever on the system when it is not in the foreground. And, indeed, this is how I use Opus most of the time. The problem is that when this option is in effect, open listers don't refresh themselves ever, not even when they come into the foreground; they only refresh when the user performs a manual refresh. So, for instance, if I'm running my OCR program in the background, and from time to time I switch to an open OPUS lister to see any new files that have been output from the OCR, I won't see anything at all until I press "F5". From my experience, this is the solitary annoyance while running in the otherwise blissful "no_external_change_notify" mode.

In order to solve this, I've written an AutoHotKey script that takes care of specifically this issue. It installs a hook (not system-wide, but rather only for the dopus process), which is triggered anytime a new Opus window is brought to the foreground, at which point it performs a full refresh (GO refresh=all). As noted, the hook is limited to the specific Opus process, so when Opus is not in use, this hook won't ever be invoked, either. Here's the code:

[code]#Persistent
SetBatchLines,-1
HookProcAdr := RegisterCallback( "HookProc", "F" )
WinGet, OpusProcessID, PID, ahk_class dopus.lister
if OpusProcessID=
{
MsgBox Error: Opus is not Running
ExitApp
}
hWinEventHook := SetWinEventHook( 0x3, 0x3, 0, HookProcAdr, OpusProcessID, 0, 0 )
Return

HookProc( hWinEventHook, Event, hWnd, idObject, idChild, dwEventThread, dwmsEventTime )
{
Run "C:\Program Files\GPSoftware\Directory Opus\dopusrt.exe" /cmd Go REFRESH=all
}

SetWinEventHook(eventMin, eventMax, hmodWinEventProc, lpfnWinEventProc, idProcess, idThread, dwFlags)
{
DllCall("CoInitialize", Uint, 0)
return DllCall("SetWinEventHook", Uint,eventMin, Uint,eventMax, Uint,hmodWinEventProc, Uint,lpfnWinEventProc, Uint,idProcess , Uint,idThread, Uint,dwFlags)
}
[/code]

Note that the script must be run after Opus is already running (because it needs to first identify the process ID of the Opus process).


#21

I still dispute your reasoning, sorry, but I don't see the need to argue any further. If you've found something that seems to work for you then great.

I've moved these posts on to the main thread to keep everything together.


#22

Leo, what is it exactly that you "dispute"?

1] I think you agree that having the system-wide hook installed necessarily incurs a penalty of extra processor cycles throughout the user's usage of the computer, while Opus is in the background, and that these cycles won't be incurred at all in "no-external-notify" mode.

2] I think that where we disagree is that you don't believe that these extra cycles can add up to anything measurably significant, while I see a very problematic 15,000 context switches per second on my machine. Am I correct that this is our disagreement?

3] I posted the Autohotkey script above so that other users who experience the same problem will be able to experience the performance boost of "no-external-notify" mode without the refresh problem. If anyone else experiences the problem and finds the script to be a
useful workaround, please do let us know.

4] Leo, I'd like to work with you, please, to attempt to reproduce the problem on your end. If I write a program that simulates the type of intense multi-thread file activity that I regularly have running on my machine, would you be willing to run it to see whether you can reproduce the situation?