Today I had to transfer multiple terabytes of data to refresh my mirrored drives and observed gradual slowdown of dopus as I browsed many folders both local and remote.
I've had this before after prolonged operation and usually a restart of dopus fixes it but today I have to wait for the entire queue to be done or I risk having to queue all operations again.
It happens gradually. At first I lost right click dragging. Left click dragging was working fine all the time.
Attempting to drag with the right mouse didn't seem to do anything. Right clicking a file and the lister background was still working though.
After a while I could no longer double click folders to enter them. Right click and going Explore with Dopus was still working though so dopus just didn't execute the default verb when double left-clicking anymore.
Eventually dopus became extremely sluggish. Renames take up to a second. There lag on folder listings and navigation between folders on both local and remote drives including SSDs. The slowdown when focusing a different tab or lister is evident in the few hundred milliseconds required to change the color of the breadcrumb bar to indicate the change of source / destination. Same lag is felt when focusing / unfocusing any of the opus listers.
=== Occurrence
I think I first noticed this behavior somewhere after version 12.6
It could also somehow coincide with the Fall Creators update of Windows 10.
However I doubt it's related to my hardware or windows installation as it persisted after my recent upgrade.
So it happened on both:
Windows 10 Pro x64 1709 Fall Creators update installed via windows update
4 core I7 2600K
16 GB DDR3
Samsung 840 Pro SSD
and then after that on
Windows 10 Pro x64 1709 - a fresh install of a fully upgraded Windows (including Fall Creators upd.) from Microsoft's website
6 core i7 8700K
32 GB DDR4
Samsung 960 Pro series NVME
== Analysis
My observations are as follows:
The slowdown has nothing to do with slow media or storage as it's exhibited when working with the SSD as well.
There are no resource or memory leaks. Even after extreme usage dopus still uses around 150mb of system RAM.
CPU usage is negligible at all times. Even saturating a gigabit network with a file transfer takes < 5% on 1 core out of 12.
All AV products are disabled so they don't impact performance. Defender disabled via group policy.
Nothing in my workstation is overclocked and it's operation is completely solid at all time - for days.
The following represents all the shell extensions installed on my system:
Using Process Monitor when things become slow may reveal something that either Opus or a shell extension is doing which is slowing things down. If you save a log in PML format, zip it (they compress really well) and email it to crashdumps@gpsoft.com.au we can see if anything stands out.
Thanks for the tip Leo.
When using process monitor I found out dopus was constantly trying to access non-existent directories.
When I checked the paths - it hit me that I had some very old stored queries (in collections) which were set to search in those paths - long gone now.
Is it possible that stored queries constantly refresh or do something by just being there and without being accessed?
They could be the reason for a notification overload. I'm removing them for now but I'm not sure if they're the actual culprit in my case. We'll see if the problem comes back.
I'm back with some more info.
I managed to slow down dopus to a crawl with mere 40 tabs.
Test #1:
Saved all listers to a layout
Closed all listers and tabs except for one
Suddenly DOpus became fast and responsive.
Reopened saved lister layout
Slow and lagging again.
Test #2:
Closed all tabs pointing to network paths
Still slow.
Saved all remaining tabs pointing to local disks as a lister layout.
Closed all remaining tabs except one.
DOpus is now fast and responsive.
Reopened saved local tabs layout again.
DOpus starts lagging.
Closing all tabs until the following remain:
2 Split listers - 2 local tabs each.
1 lister - 1 tab.
All pointing to folders on C drive (SSD).
Lagging.
Disabled all Script Addins
Still lagging.
Restarting dopus does not seem to fix anything.
Seems to be some notification overload or a threading bottleneck.
Tab count is killing it.
------
ProcMon is showing dopus accessing filesystems and folders that are NOT OPEN as tabs.
CreateFile, CloseFile, QueryFile.
Even when restarted with minimal tabs.
Those files are not being changed so there's no change notification going on them triggerring dopus.
------
Full system reboot fixes it for a while. Everything is going extremely fast.
The only thing left that comes to mind is remote desktop hooking into apps to make them slower for some reason and not unloading those hooks properly when back in front of the PC.
Eventually I'll try downgrading to 12.6 and 12.5x next to see if there's any difference.
That might be normal if it's reacting to filesystem change events and looking at the folders to see if any junctions point to them (which could mean folders you do have open need updating). Or it could be something else.
Is it possible to send us a PML log file from Process Monitor, and tell us which events to look at? The stack trace information it includes will let us see which code is causing the access (and also if it is part of Opus or something else).
After using the computer for a few days away from work I think this thing is related to remote desktop somehow. I suspect that RDP does not unload the hooks(?) it uses properly when reconnecting to a dropped remote session and/or resuming the session locally. Thus the behavior deteriorates more and more with each session reconnect until the next reboot. Somehow dopus seems to be more affected by this compared to other programs. I'm using RDP again today so we'll see how it goes.
This may be unrelated but I've seen lower directory opus responsiveness ever since the Intel Spectre/Meltdown patches. Nothing that can be done about that though, it's just something we have to live with.
I reverted back to 12.7, it is still going fast even though it has been on for a day and 21 hours. When I was on the beta (12.7.1), I could see signs of slow down after a day and a half. I will keep monitoring it.
I haven't experienced the slowdown recently. I suppose it's because of lower dopus usage on my part these days and as such I can understand how hard it would be to reproduce the heavy duty stuff I've put it through before that.
The only things that changed since then are:
I've upgrading my nVidia video driver - seems an unlikely cause considering I only skipped one driver version so far.
I keep only one lister with 10-15 tabs on each side, occasionally opening another for some temporary work
This seems to be enough for preventing dopus from going haywire.
About the slowdown:
The slowdown was occurring in between moving around hundreds of gigabytes of data among both local and remote disks and using dopus heavily for days, sometimes weeks.
I must also stress that the slowdown wasn't caused by a shortage of resources during file operations and persisted even after all of those were finished.
Listers and tabs were accumulating and eventually I was having more than 3 listers open with more than 20 tabs in total for each on both sides.
Procmon didn't show anything out of the ordinary
They happened both in 12.7 and 12.7.1 and as such could be unrelated to GQJ17's problem.
Just as other users reported, I've noticed severe slowdowns manifesting visually on the UI since the Intel Spectre/Meltdown patches, especially when SVN operations are involved. I've noticed that heavy disk usage causes mouse freeze/lag which was previously really rare.
If the mouse pointer is lagging, that's a very low-level issue, outside of anything that could involve Opus (other than that perhaps Opus is using the disk or drawing on the screen, which is tipping the system over the limit; but to be so close to the limit would mean something is wrong at a low level, i.e. drivers/hardware).
Windows 10 has a habit of replacing good drivers with old or incorrect ones, which is worth checking. Motherboard/chipset/storage drivers would be my guess here, as well as the usual GPU and mouse ones.
Yup, I don't think it's Opus. It's more like Windows handles heavy disk usage less gracefully than before. Perhaps I should have posted this in that other thread
Guess I spoke too soon but at least I have some definitive results this time. The problem looks to be your plain old file-system notification flood. It seems dopus can't handle huge quantities of those efficiently and as such everything slows down while it's being hammered.
Debugging them doesn't show any dropped events and it looks like dopus is slowing itself down by giving too high priority to them and catching them all.
While browsing within a folder or even a drive - which has many events generated for it - is expected to become at least somewhat slower, this does not excuse all other tabs and listers, representing other drives, as well as the entire GUI to become sluggish and less responsive because of them.
Shutting down all programs doing disk access makes dopus go back to normal even without restarting it but the UI animations still feel a bit choppy.
This seems to be an especially severe problem for Windows 10 as it constantly does some stupid thing in the background generating notifications left and right adding to all those triggered by the user. And thats not even taking in the flood from those sloppy programmed crapware drivers and helper programs from all the hardware manufacturers.
I also noticed dopus is acting on events triggered for paths for all open tabs in all listers all of the time and when those tabs become too many it fails acting on those notifications either randomly or sometimes all notifications stop working completely. Only handling notifications for both active tabs (left and right) for every lister at the same time and only hooking up other tabs when they become the active ones seems like a more efficient approach in this situation. It's either that or background tab hibernation like some web browsers do it.
Which version of Opus are you using? Recent updates have some improvements to the change notification code for that kind of thing. I think we have some more on the way as well.
At the same time, there will always be a non-zero cost to processing filesystem changes, and the cost will always increase the more tabs you have open, so having 60+ tabs sitting around will always have some performance impact if there's a flood of change events. But it'll be better with recent updates.
We can't really "hibernate" tabs, unless you want a situation where either they completely miss changes that happened while they hibernated, or they start using huge amounts of memory buffering up all the changes while they hibernate and then use lots of CPU and delay you the moment you activate them. Neither seems that good.
If you aren't using a tab for a long time, it's best to close it rather than let them pile up. If you're using tabs as bookmarks, you might want to consider making some toolbar buttons to go to commonly used folders and use those instead of tabs (or as well as tabs).
Edit: I guess a hibernated tab could re-read the whole folder when you activate it. That might be worth doing.
