Opus using 7GB of RAM part 3

Okay, I am trying to get VMMap to make a dump, but it is broken. This is what happens when I try to dump Opus it with VMMap, as per the instructions.

VMMap itself goes nuts and crashes with excessive RAM usage. How do I fix this? Is it related to the problem with Opus? Can you provide any other debugging tools since this one doesn't work?

Did you give it a chance to finish? It doesnt look like it's crashed, just that it's busy working.

I left it for half an hour and then it crashed.

The folks in the SysInternals forums may also have some ideas.

I'm not sure VMMap is designed to be being left running in trace mode for a long time, but it is the best debugging tool for this that we know of.

It's great if you know a reliable way to quickly trigger a leak, but it's going to run out of memory if you leave it running.

If, once it gets into this state, you really can make it leak memory by simply connecting a new RDP session, then that would be the time to log what's happening.

No, ï crashes if I open up opus immediately try to dump it. Opus is only using 20k in task manager and maybe 100mb total in VMMap. Maybe this is related to the problem. Some kind of circular reference that causes excessive RAM use.

Can you confirm that under normal circumstances if you open Opus and immediately dump it on a 64bit system VMMap doesn't crash or use gigabytes of memory? If so I'm going to try a fresh VM and opus with no config, in in case it is an option that causes the problem, and work from there.

What are the exact steps I should follow to "try to dump it"?

Nothing in our memory leak guide says to create a dump file (the word "dump" isn't on the page at all). The steps that are in the guide definitely work, or we could not have made the guide and the screenshots of it working. I just checked and they still work today.

I've been trying to save the state of VMMap since it inevitably crashes before Opus starts to goes nuts with RAM. Since it's the only debug tool I was hoping to get something out of it at least... The fact that it won't save its state doesn't inspire confidence.

Anyway, the output from your instructions seems to be useless:

I can't see why Opus would load any additional DLLs if it starts eating RAM simply by running for some time (not actually being used). As I said, I can't wait for it to happen because VMMap will crash. What other options are there?

One more thing:

Even though Task Manager doesn't show it, VMMap does show memory increasing every single time I start an RDP session. It's only a small increase, but it happens. Presumably over time that increase grows until it starts becoming noticeable, so the screenshot I posted is of the fault in progress. Is there anything else you need to debug it?

The top line in your screenshot shows 2 allocations of 1.47 MB. If those are the largest allocations in the process then the problem can't have happened yet when you made the screenshot (or the problem happened before tracing was turned on, in which case the guide wasn't followed properly), so it's not surprising that the results do not indicate the cause of the problem.

I'm not sure you understand what the stack view in VMMap is showing, if you say that. It does not list every DLL in the process. It lists the DLLs involved with a particular memory allocation.

DLLs are loaded and unloaded all the time, anyway, in reaction to filesystem events triggering file detail updates, which trigger shell extensions, which are loaded and unloaded on-demand..

There are no other debugging tool options that I know of. Re-opening the same thread again and again is not going to change that, I am afraid.

The instance of Opus in that screenshot did start to spiral out of control after I ran my RDP start/stop script. Unfortunately VMMap crashed before I could get more details, but what you are seeing is Opus at the start of the fault. Chances are the memory allocation that eventually used gigabytes of RAM wasn't very big at the time it was taken, which is one of the reasons why I wanted to save the entire state with VMMap.

So there are two problems. Firstly I can't reliably start the fault off... This time I was just (un)lucky. Secondly when it does happen VMMap dies too quickly to give us any useful information. I'll keep working on the first, but the second seems hopeless.

Please keep this thread open, I'm still working on the problem. If I can just find a way to make it happen much faster there is a chance I can catch it.

A little update.

Closing and re-opening an RDP session crashes VMMap. Unfortunately closing and re-opening an RDP session is also what triggers the problem with DOpus, so I don't think even just leaving a logged in session permanently running will work. I am going to try it anyway, using VNC instead of RDP.

I also switched to using QBittorrent. The issue has not been seen since I made the change, but I know it isn't dependent on uTorrent because I see it on other machines that don't run that software as well. Considering uT and QBt do basically the same thing in slightly different ways it makes me wonder if something about uTorrent makes it happen more often. The caching system in uTorrent is more efficient than the QBt one, so perhaps the coalescing of sequential writes that uT does is a factor.

Did I mention that the bug is confirmed on Windows 8 as well?

And right on cue it happens again with QBittorrent, so that rules that out. I'm starting to wonder if it has anything to do with Bittorrent at all, and is just an RDP problem.

So no new information at all then, from the past two threads on this.

RDP is fairly common (and something we use ourselves daily). Torrents are incredibly common. If either were the cause on their own then we would have hundreds of reports, given that 7GB is more RAM than most peoples computers even have.

If the issue only happens when using RDP and torrents and Opus together at once... that seems far fetched, unless the issue is in the network/firewall layer (since both RDP and torrents involve the network), but that would be unlikely to only cause Opus to allocate a large amount of memory and not anything else.

We still don't have anything to go on, and none of this brings us any closer to knowing what's causing what you're seeing.

There has to be something more to this. A component on the systems you're seeing it on, or something unusual you are doing with Opus that most people wouldn't do. We have no way to guess what that could be, and we've been over this many times before so I'll stop typing now.

If we didn't ever try disabling the Movie and AudioTags plugins, that might be worth a try. That will stop most of the things which cause Opus to talk to video codecs about the movie files that are being downloaded, more or less leaving only the ways which Explorer would also use. (Opening an Explorer window in thumbnails mode for the same download folders, and seeing if it leaks, might be worth a try, along the same lines.) Video codecs have been known to go crazy when asked to inspect incomplete files, so the partially downloaded torrents could be triggering that kind of problem.

Make sure uTorrent's Append .!ut to incomplete files option is turned on as well, along the same lines.

Let us know if you find something to go on (but please don't bump the thread otherwise, as we're going round in circles).

I am making progress. Drivers have been ruled out. I don't think it is bittorrent related. I think it is all down to RDP.

I have noticed something else odd. Once it has happened you can re-start Opus but it remains very slow to respond to double clicks on the desktop. Once a window is opened it is fine, but the initial opening is very slow.

I think the issue happens when RDP sessions are unexpected disconnected, so I'm trying to narrow down the circumstances or develop a way to simulate it. Sometimes I get RDP session disconnects while connecting to a VPN or due to wifi issues, and I have noticed that re-connecting takes longer than usual afterwards. I suppose it has to close the old session down before the new one can start, but the delay is much longer than it normally takes to shut down which suggests something else is happening. My guess is that Windows waits for applications to respond and after some time-out period just forces handles closed, like it does when shutting the machine down.

I bet this is what is causing the problem. What I am working on now is a way to simulate this kind of fault from a script so I can make it happen on-demand and fairly quickly.

Okay, I wasn't asking for help, merely documenting my process. You don't have to keep responding until I figure it out.

I have come up with a solution, and would like to share it and ask for some help testing it.

I made a little program called memlimit, that sits in the task tray and monitors the target app's memory use. If it goes over a certain amount, the target app is terminated. I use it to monitor dopus.exe, with a limit of 100MB memory. Opus is killed when it goes nuts, even if I'm not there to see it.

The app is x64 and takes two command line argument: and . Process name should be dopus.exe, or you can try notepad.exe for testing. The limit can be whatever you like.

When started the app appears in the system tray. A red "no entry" sign means the target is not running. A green tick means it is running but not using too much memory. Hover the mouse over to see the current memory use and limit. If the target uses too much memory the icon changes to an exploding bomb and the target is terminated. You might also see an exclamation mark if something went wrong. Make sure you launch the app with sufficient privileges to terminate other apps, e.g. as an administrator.

Note that the app only polls once every five seconds, so the icon can take a few moments to catch up when you open or close the target app. I might increase the polling time, or make it configurable.

I have only done light testing myself but it seems to work. No warranty, use at your own risk etc. PM me for a copy, if it proves reliable I'll make it publicly available.

I have also found a way to reliable re-create the problem.

Fresh install of Windows 7 SP1 and all updates inside a VM. Last version of Opus 10. No other software. Connect to the VM with Remote Desktop.

Download this file:
break_opus.7z (1.12 MB)

Extract the archive and run all six batch scripts. They will pull a lot of bandwidth. Make sure you keep a lister window with the wget directory and the downloaded ISO files in it visible.

Periodically disconnect and re-connect RDP. After a few days you will find Opus using massive amounts of RAM. If I do this I can re-create the problem consistently within a week.

Note that you can't run VMMap as it crashes when you disconnect/reconnect RDP. I have no idea how you can debug Opus in this instance. RDP seems to be essential to cause it.

Just FYI, VMMap 3.12 doesn't seem to crash for me when disconnecting/reconnecting RDP. (This is on a real Windows 7 machine, maybe a VM is different.)

Thanks Jon, but that's the version I have been trying. It consistently crashes every time I re-open an RDP session on a real machine running Windows 7 x64.

Oh, and here is my little memlimit app. x64 only, only tested on a few machines but it seems to work okay.

It needs to be started from the command line or a shortcut. The syntax is: memlimit

The process can be found with Task Manager. The limit in MB is obvious, however note that the amount it watches is not the same as the one in Task Manager which is calculated slightly differently. It's in the same ballpark and catches Opus though. I suggest a limit of 100MB for dopus.exe. Mouse over the tray icon to see the current memory use.
memlimit.zip (38.6 KB)

Have you tried turning off Clipboard Sharing in your Remote Desktop client?