Spoke too soon, it's still boned.
Has this been fixed in V10? I have the latest V9 and although it seems to happen much less often now it still happens.
I don't think there is an issue in Opus to be solved here. Memory leaks which only one person reports are almost always due to shell extensions (or similar 3rd party tools/components which hook into other software).
There are no shortcuts here, you need to use the guides already linked to if you want to track down what's causing the problem.
It's a fresh install of Windows 7 SP1. Only Opus, TrueCrypt and Firefox are installed. TrueCrypt and Firefox don't have any shell extensions. The machine just serves files over standard Windows shares all day, nothing more. Also, if it were a shell extension wouldn't it affect Explorer as well? I have tried having just Explorer open and it doesn't seem to happen.
Oh, and my mistake, I meant I am on V10 and was wondering if V11 had fixed this flaw.
Since someone revived this thread, I'll post and update to my experience here. Since I stopped using Microsoft's Remote Desktop Connection (RDP client on a Mac), the memory exhaustion problem has not recurred (that's 9 months now, where it used to bloat several times a week).
It only seems to affect machines I RDP into as well. It must be some issue with Opus and RDP. Nothing to do with shell extensions.
So is this bug even open? Is there any chance of it being fixed? I presume Opus 11 still has the same flaw, so there is no point upgrading. If there is a fix will it be available for users of V10 or will I have to upgrade?
Come on guys. I'm very happy with Opus for the most part but the lack of fixes for ancient bugs is very frustrating.
Tying it to RDP seems pure speculation at this stage, especially if we're talking about modern versions of Windows. (RDP on Windows XP had some issues, although mainly to do with CPU usage from the clipboard components.) I use RDP extensively. For the last month I've used RDP all day, every day because I've damaged my back, can't sit down, and have to use a laptop on the floor while RDPing into my workstation. I've literally never seen this problem.
We can't fix a bug when there's no proof the bug is in Opus, and when we have nothing go on, and when only one or two people are seeing it (we don't know if both of you are seeing the same problem; lots of components can cause memory leaks and you may each have leaks from different things).
If you want us to believe there is a bug or have a chance of fixing it, you have got to use the guides and tools we pointed you to. it is the only way. There's no magic wand we can wave over the source code to magically fix something when we have no indication or where the problem is or if it is even a problem in Opus itself. We can't do anything from our end, unless you want to ship the machine to us.
No machine is really a "fresh Windows install", either. You've got a unique combination of drivers, which may cause leaks themselves. TrueCrypt installs a filesystem driver which could cause problems (although that is unlikely as we use TrueCrypt as well and would have seen it, unless it's tied to particular settings or usage).
Heck, some drivers install shell extensions (e.g. NVidia's drivers do this, and their extensions have had several bugs). Have you looked with ShellExView or are you just assuming there are none? Most machines have hundreds of shell extensions that you would not even know were there unless you looked.
Not necessarily. Opus and Explorer are not exactly the same. A shell extension may leak memory if things are done in a certain order but not if they are done in a slightly different order, for example. There have been lots of shell extension problems that affect one program and not the other, in both directions. (But things are more likely not to have problems in Explorer because that is the one and only program most people test their extensions against.)
One thing to try:
Set Preferences / Miscellaneous / Advanced: no_external_change_notify = True and see if the problem still occurs.
If that fixes it, it's possible that the file server is generating file change events faster than Opus can process them, so the events build up into a bigger and bigger queue. If the changes stop for long enough for the events to be processed then the memory will be returned, so it's not a leak, but it can still cause problems if Opus never has a chance to process the backlog of events. It's very rare but (literally) a couple of people have run into this in unusual situations, so it is worth ruling out.
I'm just responding for my own situation...
Of course tying my memory bloats to RDP is pure speculation, and my post was not trying to generalize the problem beyond my circumstance. Given that the problem is so very difficult to diagnose, and the tools insufficient, what else do you expect users to do?
So we're left with creating and testing theories, and trying to find some cause and effect. Your advice is to remove things, and see if the problem does not reoccur. It is good advice. I'm pretty confident I found the trigger in my situation, and that's good enough for me, since at the end of the day its about avoiding the problem.
I mentioned in my old post - you and I use RDP with different clients, yours stable, mine crashing, aborting, failing to quit, and failing to reconnect w/Windows multiple times per day. Mine is considered "unsupported" for any Mac OS X version in the past couple of years (Microsoft has essentially abandon the tool - they haven't touched it since 5/5/2011).
I agree that the guide and tools are useful. But I hope also that you can agree, they are also severely limited, and in some situations, where the recurrence takes a long time, not possible to use as they do not support long-term use - they crash.
I think we'll all agree, it can be a very difficult problem to resolve.
[quote="leo"]Set Preferences / Miscellaneous / Advanced: no_external_change_notify = True and see if the problem still occurs.
If that fixes it, it's possible that the file server is generating file change events faster than Opus can process them, so the events build up into a bigger and bigger queue. If the changes stop for long enough for the events to be processed then the memory will be returned, so it's not a leak, but it can still cause problems if Opus never has a chance to process the backlog of events. It's very rare but (literally) a couple of people have run into this in unusual situations, so it is worth ruling out.[/quote]
I will try it, but it seems unlikely because the machine is idle most of the time and the memory never clears. Having said that I did think it could be related to file change notifications, but surely the rate would have to be incredibly high for Opus not to process them in a timely manner.
I appreciate it is a hard problem to resolve, but the thing is it has been getting better over the years. It happens less now than it used to, so clearly something is changing. Windows and drivers have been updated, Opus has been updated. If it is a driver issue it would surely affect more than just Opus, or if not then Opus must have some kind of issue that causes the problematic interaction.
I have been doing more testing and this bug isn't related to plug-ins or Explorer add-ins. I did a fresh install of Windows 7 Pro, with only Opus 10, Firefox and uTorrent installed in it. Nothing else other than all available updates from Microsoft. It still has the same problem. Neither Firefox nor uTorrent install any Explorer add-ins. The bug must be in Opus, and presumably still exists in V11.
The only solution I have found is to use Kiwi Application Monitor to kill Opus when its memory usage gets silly. The machine I used for testing is a VM, so other than the standard Microsoft drivers for basic hard it has a different set of drivers for things like video.
Just to confirm, I tried out Opus 11 in a VM and the bug is still there. I can reproduce it somewhat reliably by running uTorrent in the same VM. It seems like you have to have opened the uTorrent download directory. Even if you close it the bug can still happen later.
Interestingly network drives don't seem to be affected. My guess is that it is related to some kind of filesystem change monitoring that isn't supported for network drives, only for local drives. The fact that uTorrent keeps writing to partially downloaded files, or perhaps just the partially downloaded nature of some of the files, triggers the bug in Opus. Maybe something like trying to read an archive or get image dimensions on a partially downloaded file, or just getting endless update notifications as the file keeps being written to.
Any news on fixing this problem? I can help the developers recreate it if necessary.
If the issue is a conflict with dopus and utorrent may I suggest an alternative qbittorrent. Its Open source and has no adds.
I don't think it is a conflict with uTorrent specifically. I was just about to post that I tried it out with Perfect Dark and had the same result. Took a couple of days to happen.
If you show us screenshots of how the Opus window is configured (please make sure they include all file display columns) and give us examples of torrents to download that re-create the problem, we will try to reproduce it.
If video files are involved, we also need to know if any video-related software or codecs/splitters have been installed. (Which may be none if it's literally just "Windows 7 Pro, with only Opus 10, Firefox and uTorrent".)
For testing I used the latest uTorrent and the following torrents:
thepiratebay.se/torrent/9852366 ... FOLK_MUSIC
thepiratebay.se/torrent/8492209 ... rostWire_F
releases.ubuntu.com/12.04/ubuntu ... so.torrent
releases.ubuntu.com/14.04/ubuntu ... so.torrent
legittorrents.info/index.php ... 6eae8a6f7a
legittorrents.info/index.php ... be716f5d23
legittorrents.info/index.php ... eca8cfa54e
legittorrents.info/index.php ... b0ded36ef0
legittorrents.info/index.php ... 7344db606a
The OS is WIndows 7 x64 with all current updates. On both the real machine and the test VM I had Opus 10 and uTorrent installed. On the real machine I also have Firefox and a couple of other bits, and a basic set of drivers. On the VM the only other thing was the VMWare integration tools/drivers. No extra codecs or anything like that, I don't even have a monitor attached to the real machine and don't use it to actually view the files. I use shared drives to access everything from other machines.
I've tried reproducing this using those torrents.
The video is here: TTest.7z (64 MB) It's at 3x speed so it's only 9 minutes instead of 27. Nothing much happens, but you can see what I did in case anything seems wrong or different.
Memory usage increased slightly, but only by about 7MB. Nothing out of line with what I'd expect with components being loaded/cached etc. (This was a debug build which had just been launched, too, so it would use more RAM than normal and not yet have loaded much at the start.) Nothing like 7GB was used up.
I suspect the issue here is due to video codecs/splitters. The files being downloaded include a few different video files, and certain video codecs/splitters have been known to crash or do other things when asked to inspect partially-written video files. None of the ones on my machine seem to do this, however. Unless I just got lucky. (For it to happen it could require the file to be inspected at just the right time.)
The only thing I saw go wrong was that the spinning blue circle indicating file information is being queried would get stuck sometimes until I refreshed the folder. (When the folder flashes at some points in the video, it's because I am pushing F5. Sometimes a lot to see if forcing more frequent refreshes of the metadata might trigger the problem, although it didn't.) That may indicate that the video codecs/splitters on my system are getting stuck or failing when asked to inspect the partially-written files at certain stages, but I'm not sure. It's something more minor we might look at later, although there may not be much we can do about buggy video codecs except move the queries into another process.
Let me know if you have any suggestions on what I might want to try doing differently to trigger the problem, as I'm still yet to see it happen.
Leo, I gave you call stack that should clearly show the routine that cleans up memory mess that DOpus made (it's not memory leak, but memory accumulation). Both mojo-chan and me have the problem in similar circumstances - folders that change frequently.
I know it's easier to blame everybody else, but, you are not correct here.
That call stack was not useful, unfortunately.
It doesn't prove that you're both seeing the same issue, or that the large allocations were due to Opus itself. It only showed that one thread in Opus was freeing some memory at one point during shutdown, which is not surprising. (The call stack was of the main shutdown routine.)
You might be right, but we still have very little to go on, and it's still strange that this problem is so noticeable when it happens and yet has not had more reports. That implies there is either an unusual activity or configuration causing it and/or an interaction with other components. I mentioned video codecs/splitters because they have caused problems like this in the past, specifically when asked to inspect half-written files like the ones being downloaded in mojo-chan's scenario.
If you want to help, give us ideas on ways to reproduce the problem, or capture the allocations happening with VMMap.
[quote="leo"]That call stack was not useful, unfortunately.
It doesn't prove that you're both seeing the same issue, or that the large allocations were due to Opus itself. It only showed that one thread in Opus was freeing some memory at one point during shutdown, which is not surprising. (The call stack was of the main shutdown routine.)[/quote]
no, you haven't read carefully. at memory level of 10-20Gb, shutdown lasted for few minutes. only one thread was active during that period given callstack was actual for the whole time. that is typical to freeing accumulations of million of small objects in a loop.
btw, are you sure you cannot see on the call stack which collection's destructor is being called? collections are often templated, so you might be able to get the name of the class that leaks from function signature.
ok, i can understand that. the trouble is, this will be answered after bug is found 
i have no idea how to reproduce. many times i've seen it, network paths containing active logs were involved. if you remember, we previously talked about this - i would say that some queue that tracks changes or holds update requests gets filled quicker than it's emptied. however, i've tried many times by going to the same folders and 'leak' hasn't happened. there must be something additional...
unfortunately, vmmap won't help here. it always took days for this to happen (if you remember, i don't restart my computer )
since mojo-chan seems to be able to reproduce this problem more often, maybe he could make you full dump of dopus process (using process explorer, for example) - if you're lucky, on some thread you'll catch some collection with, I'd say, millions of elements.
I agree with gmit, this is an Opus issue. Leo, you keep mentioning codecs but I don't have any installed, and it seems unlikely it would be one of the ones installed with Windows. Your test was far too short, you need to keep it running for a few days or even weeks. It helps if you keep rotating torrents too; as gmit says it seems to be file churn that does it. I think it is probably not seen more often because it is unusual to have so much file churn and because most people don't run their machines 24/7 for days at a time.
I see the same behaviour as gmit if I shut down. Opus hangs for minutes, freeing memory. I can see the memory being freed slowly. It really seems like a death of a thousand cuts, and sure enough when it is going up it happens fairly slowly too.
Maybe I can write a little app that simulates torrent downloads in an accelerated manner to try cause the bug faster. I'll try running VMMap too. I have a dump of Opus with the problem evident, but it is from my actual machine so I need some time to go through it and check it for privacy issues. If/when I get it to happen in a VM I'll dump that.