DOpus stops responding when using npm or build

I noticed many times that if I perform an operation that affects huge number of files, DOpus stops responding (or responds very slow) util the operation is finished. E.g. I open command line (using a shortcut in DOpus) in a project folder and run npm install (which uses npm packet manager to download all dependencies). This creates node_modules folder with about 25k files. DOpus responds very slowly until the operation is finished. Similar thing happens if I run e.g. a Java build.

I tried to disable all label assignments but no luck.

By "not responding" I mean that I can't move any DOpus windows or event bring them to front by clicking them on the task bar. Clicking anything in a lister does't have any effect. But DOpus windows are not marked with "not responding" title. Other applications work fine (maybe a little slower because CPU is used extensively but in general are responsive).

Is there anything you guys could do to avoid long response time?

Screenshot of Process Explorer:

image

Most likely the change notification system cannot keep up with the number of events being generated. That usually means either something unusual on the Opus side, or something is generating a really extreme amount of events.

On the Opus side:

  • How many windows/tabs do you have open?
  • Are any showing large/complex Flat View, collections, libraries, etc. folders?
  • Do you have a large number of Collections?

Setting Preferences / Miscellaneous / Advanced: no_external_change_notify = True can be used to test if ignoring change notifications stops the problem from happening. But it should not happen unless something quite unusual is going on.

If the problem is change notifications, it will also affect other software. (I think Explorer will start to drop/ignore change notifications if there is a flood of them, so you may start to see it not notice changes to files/folders. OTOH Opus will not drop them but if they are generated faster than it can process them then it can cause the problems you describe, as well as high memory usage.)

I have 3 empty collections (I'm not viewing them at the moment, just have these collections created).
2 DOpus windows, each with 1 tab.
They'are showing normal List view, no flat view
Node_modules folder (which content is changed intensively) is visible in one of the listers but its content is not displayed and it's collapsed in the folder tree.

I tried setting no_external_change_notify = True (and restarted DOpus)
Now it looks like DOpus behaves normally and uses almost no CPU.

When I set it to back to false, the problem occurs again.

I guess no_external_change_notify set to true has many side-effects and should be set to false normally. Is there any other solution to this issue?

If you want, I can send you example package.json file that is used to configure project dependencies. If you then install node.js and run "npm install", you can see what exactly happens.

That command must be generating hundreds of thousands of events over a very short time.

We should make some changes in Opus to make it drop change notifications when there is a flood, but that may take a while. This has come up about 3 or 4 times before (in the past ~15 years) so it's uncommon but if you can't change the thing that is generating all the events then it can definitely be a pain. Disabling change notifications (or exiting Opus) while it runs is the best thing to do for now.

Thanks for looking into the matter.

Does it change anything if DOpus wasn't the parent of the process that is flooding?
As I mentioned earlier, this relates not only to node.js but frequently also to building Java apps (when a lot of files are copied to target directory).

It shouldn't matter what the parent process is.

Copying a few thousand files should not cause a problem, but if something is constantly opening and closing them, or creating millions of temp files, that might cause a flood.

If it's using all the CPU (especially if its priority is set higher) then that might also contribute to the events becoming backlogged, since some CPU is needed to process the change events. (It must be quite extreme, though, and not just the CPU usage on its own, since it isn't triggered by things like compiling C++ code, which will use all 8 of my CPU threads 100%.)

My bet is that while it is happening, if you have an Explorer window open and change some files in the folder using another program, it will not always see those changes.

You can also use the change notification debugging mode in Opus to see the events it is processing, which may indicate the amount of events and files they apply to. You should notice the amount is really extreme even compared to other activities which modify a lot of files, if it is what I'm thinking it is.

(There are other possibilities, of course, like something causing event processing to take a really long time even when there aren't that many events, but they are much less likely based on what we know so far.)

I have just tried on my home computer and it was much better. Weird, maybe it's more related to some antivirus (we use Sysmantec) or other software at work. Thanks for all the tips I may use for debugging.

I'm still having the same issue: DOpus windows hang for some time when performing operations on huge amounts of files. This applies not only to using npm but happens also when deleting files with DOpus (without recycle bin).

Now figured out that it only happens if I use Set TREEROOT {sourcepath}
If I reset TREEROOT then DOpus does not hang.

So e.g. if I have two windows: one with specific treeroot set and another with default treeroot and perform operation on huge amount of files (like running npm install @angular/cli or deleting nod_modules after it's finished), then only the window with specific treeroot hangs.

Previously I reported that it doesn't happen on my home computer. It was probably because on my home computer I had files on disk D: rather than deep inside c:...\Documents so I didn't have the need to use specific treeroot.

Could you please verify and confirm this issue?

We're in the process of refactoring how the folder tree works so this will likely change in the future, although whether the treeroot ability remains is up in the air, as it complicates things a lot. If treeroot is causing problems, avoid using it for now.