Opus using 7GB of RAM part 3

Latest observations.

Its getting interesting.

I have now seen the issue on a second machine, an almost vanilla W8.1 system. On each machine I have seen the memory usage increase incrementally each time I log on via RDP, with subsequent rises approximately double the size of the previous one. The latest incident went like this - after 4 or 5 smaller rises I recorded these values for dopus.exe 182MB, 357MB, 609MB, 1176MB, 2304MB. The last was when I logged on to the machine locally. The second machine's incident followed much the same pattern. Above about the 200MB mark the 100% CPU spikes become noticeable and coincide with the memory usage rises. In between these RDP triggered rises the memory usage remains quite stable. I have dopus.exe sitting at 2304MB quite happily now, with no listers open (not under VMMap which crashed unfortunately).

Before crashing VMMap shows that it is Heap (Private Data) memory that expands. Doing a trace and looking at the stack of the high memory items (either RtlAllocateHeap or RtlReAllocateHeap) only shows windows system32 dlls and dopus.exe/dopuslib.dll entries. Looking at the threads tab in the Process Explorer properties window during the CPU spikes shows many instances of dopuslib.dll!DummyDllFunctionToAvoidSymbolConfusion+0x4a4e8 consuming CPU cycles.

I immediately tried to replicate all of this on the second machine but was unable to. Interestingly though, dopus.exe still gives a little CPU spike every time RDP connects, except without a corresponding rise in memory usage.

The only other variable I can think of is that, in all cases of this issue that I have seen, it has been following an extended period, usually overnight, leaving a dopus lister open and RDP either connected or previously connected then disconnected.

That's exactly what I see. We have the same issue. Usually after leaving Opus running for an extended period and using RDP. When connecting to RDP there is a CPU spike from Opus, and then sometimes it allocates more and more memory, approximately double the amount each time once the problem starts. I can't pin down what starts it either, but I'm glad you have been able to get some data from VMMap.

I wonder if WinDbg might be able to help... But Opus being closed source, there probably isn't much we can do with it. I'll try to get some VMs set up.

beatbox20, what other stuff are you running? Like you I have a pretty much virgin Windows 8.1 installation. Aside from qBittorrent and TrueCrypt the only other non-standard things installed are 7zip and PaleMoon. No codecs or anything like that.

On the vanilla machine just dbPowerAmp, Classic Start Menu, Norton Internet Security and SQL Server. I don't think it's any of those, unless you also have them.

The only connection I can see between dopus and RDP is a dopus.exe handle (in Process Explorer lower pane Ctrl-L, Ctrl-H to show handles)

Type: Event
Name: \BaseNamedObjects\TermSrvReadyEvent

Relevant or irrelevant?

Oh yeah, Classic Start Menu. Forgot that one. But that really is it.

I suppose the next step is to dig in to what happens when an RDP session connects.

I should say that when installing dopus I always choose not to replace explorer or any other function when asked. Don't know if that would make a difference?

mojo-chan - can you still repeat this issue reliably with 0.1MB increase from the start as previously stated? How are you measuring the 0.1MB?

I have not had any problems with VMMap crashing due to RDP logons or missing dll's. I have found with VMMap paused it will run for much longer than usual -- Options - Trace Snapshot Interval - Paused.

Next time I catch this happening I will try closing the TermSrvReadyEvent handle to see if that stops it.

I install as an Explorer replacement. My measurement is just using the Task Manager, nothing special.

At the moment I can't reproduce anything totally reliably, at least on demand. Given long enough it happens though, so the plan was to make a couple of VMs with one running Opus and the other opening and closing a RDP client session endlessly. I'm away at the moment so I can't test anything, but I'll try VMMap again when I get back.

Latest observation - the plot thickens...

I found out how to prevent idle RDP sessions from disconnecting here sevenforums.com/tutorials/11 ... sions.html thinking great! this will reduce the number of times I have to keep re-connecting RDP and hence minimize the dopus memory growth issue. Wrong!

Next morning I was pleased to see the RDP session still connected, however both firefox and dopus had crashed, leaving these messages

This is the first time I have seen the compatibility message. This compatibility entry was found in the registry

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers]
"C:\Program Files\GPSoftware\Directory Opus\dopus.exe"="DISABLEUSERCALLBACKEXCEPTION"

Event Viewer entry showing dopus at > 8GB
Warning 23/05/2015 10:41:00 AM Resource-Exhaustion-Detector 2004 Resource Exhaustion Diagnosis Events
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: dopus.exe (4896) consumed 8882839552 bytes, firefox.exe (3288) consumed 537026560 bytes, and mstsc.exe (7440) consumed 240885760 bytes.

So this was from normal memory usage to maximum possible with no connecting or disconnecting of RDP. This is what I have usually seen in the past. The going up in steps behaviour triggered by RDP logons I have only seen recently.

On top of that the idle RDP disconnect fix I applied turns out to not be always effective, as I have had several disconnects throughout the day today.

So, just to clarify, you left your RDP session connected all night?

That's an interesting development. When I get back I'll try to repeat your experiment. If it works it would be extremely helpful because I can easily leave VMs with RDP sessions connected indefinitely.

Yes that's right mojo-chan.

I can leave my work LAN RDP session connected indefinitely too. That's to the W8.1 machine. But I don't see the same dopus behaviour there. I have only seen the stepped increases there. If I connect to my work W7 machine from home it disconnects frequently when idle. I have tried many "fixes" to keep it connected but so far none have worked. Occasionally it does stay connected all night. Last night was one such night.

It seems like it is worse for you, like it was for me back when I was on Windows 7. There must be some variable, maybe a setting or something.

Latest observations

Win8.1 machine - after 1 week dopus still behaving after many RDP connects, VMMap (paused) very slowly increasing now at 2.5GB (on Win7 it increases much faster)

Win7 machine - Rebooted Friday. Got the stepped increase today (Monday) when logging in locally following a weekend of remote connects/disconnects. It was doing a 1GB -> 2GB increase. I had not noticed any previous increases while logged in remotely.

I managed to capture the dopus.exe thread stack from Process Explorer during the CPU spike/memory increase and after the increase. The after increase stack is quite stable over time. Maybe the differences can shed light on things? leo, jon?

During CPU spike/memory increase


After increase

Interesting, it looks like some kind of bad pointer issue. The kernel is stepping in to catch faults and generate exceptions by the look of it, so perhaps those exceptions could be caught and analyzed.

I hope this can finally be debugged now. Windows 10 is due soon, and presumably there will be a new version of Opus to go with it, so it would be nice to have it resolved in time for an upgrade.

We're not expecting any major changes to Opus for Windows 10. The existing version works fine with the preview builds and we don't expect Windows 10 to be a factor in when we put out Opus 12, which is not planned for any time soon.

Re this issue, I think I may have found something this morning, but I still need to do some more investigation. It's promising, though; I think I have reproduced it once or twice, and have a theory to test now.

That's good news indeed leo, thanks for attending to it.

This may fix the leak, if you want to try the change before the next beta.

Opus 11.13.1 beta needs to be installed first (link in my signature, if needed). Then fully exit Opus (via File / Exit Directory Opus) and, using Explorer or something that isn't Opus, copy the dopus.exe from this zip over your existing one in C:\Program Files\GPSoftware\Directory Opus.

(Test version removed, as the change is now part of 11.13.2 beta.)

Fingers crossed, and maybe we owe you that free upgrade now. :slight_smile:

I'll try to try it out, I'm still running 10 so I need to upgrade first and don't have much time at the moment. Thanks for the quick response.

If anyone deserves credit here, it's beatbox20.

OK I am running test build 5630 now. Fingers crossed.

Happy for mojo-chan to get the free upgrade :slight_smile:

I have upgraded my problematic machine and am installing the update now.

Are we right to assume this is fixed now?

The fix that was in the test version is also part of 11.13.2 beta.

Well, it hasn't failed yet, but it can take a while sometimes... Touch wood, fingers crossed it seems okay so far. Out of interest, can you say what it was? I'm just interested to understand how an RDP session could interact with other software in this way.