Yesterday I had added to Opus Collection my external hard drive contents as sub-collection for offline browsing. Collection folder get size is app.27 MB.
Today's morning I had confused where dopus.exe eats ~600 MB memory.
At the first onset I suspect Collection. Yes, I'm right. Removing collection fix problem with this memory leak.
It's normal behavior with big sub-collections or bug, and I must report to support?
How many files are in the collection(s)?
The list of files in collections are stored in memory. (I think so that Opus knows if it has to update them when file-change events happen.) So if there is a very large number of file it could use up a lot of memory, I guess. Could be something else going on, though.
Approximately 3300 files. On hdd this collection take up only 27 MB. I mean /dopusdata/Collection folder.
Not sure about your theory, unless things have changed significantly in the pre-release version I'm testing with (which is possible).
I just added 5031 files to collections, both into lots of sub-collections and then into a single collection, but the collections folder is using just under 1MB and the memory usage increase was nowhere near 600MB.
Normally I'd assume high memory usage like that was probably due to a shell extension, video codec or plugin going haywire over some of the files (more info here), but the huge collections folder suggests something else might be going on.
Is this something you can reproduce? If so, could you zip your collections folder up so it can be looked at to see why it's using so much space?
Have you tried to restart opus after adding 5031 files to collection as subcollection?
I found memory eating only after restarting.
Restarting makes no difference here, except that if I have the Description column on and go into the folder Opus uses a lot of CPU (but not memory) generating descriptions for all the files (which makes sense, and stops once it's done).
I think when Opus starts it verifies that the items in collections are still there and checks that their cached date/time and size information is correct. Are you running a real-time anti-virus scanner or similar which might trigger high memory usage if lots of files are examined like that?
Does the high memory usage happen only if you enter the collection after restarting, or does it happen if you restart and don't open any Opus windows (or only open ones pointing at normal folders and no collections)?
I have Kaspersky AntiVirus 2011, but it disabled now. I see memory leak in default windows TaskManager:
first I have only dopusrt.exe running, then doubleclick on desktop and dopus.exe appears in task manager then it grow ups to 300MB memory usage. No dopus window appears. Then I doubleclick again and see the dopus lister.
Now I will make another test and report here for results.
Tests failed. Anyway memory usage is too high. I sent to you my Collections folder for testing.
I see now that when you said 3300 files were in the collections you meant the collections settings folder. i.e. There are 3300 collections, not 3300 files listed by the collections...
I don't think collections are well-suited for what you're using them for. If you want to organise that many files and folders, and can't do so by moving the files themselves directly, you're better off using NTFS hardlinks/softlinks.
The collections system in Opus isn't designed for having thousands of collections with hundreds of thousands of files within them. (It'll work but it'll use a lot of memory, and possibly also a lot of CPU checking each file-change event to see if it applies to any of items in any of the collections.)
As I told in first private letter to you I was deleted some big folders because there not 3300 files already, but even this small Collection example eat memory, so this is not so important how much files there.
You said that added ~5000 files and it worked well? So I suspect that main problem is unsupported Unicode in Collections.
AFAIK, Unicode works fine in collections. What gives you the idea that it doesn't? (The problem in the PM thread was that the names got mangled by zipping them up.)
Even in the set that you sent me via PM, that is not a "small collection example." You've got about 1200 collections in there! Collections really aren't designed for that kind of usage, at least not right now.
Collections were envisaged as something you could use for temporary file lists, like as the result of searching or while building a list of things you wanted to copy as a batch. They're not suited for using as your main way to organise and view all your files. (At least, not the way they work right now. Perhaps that could change.)
This Collection in 1200 files eat 150 MB memory by dopus.exe, I think this is enough too.
Is there any reason why you use Collections to organise your files?
What sort of files are these?
I ask because I have squillions of files to organise and find that the best solution is to deploy index and search software that will home in on a file depending on the keywords that I throw at it. But this works only with certain file types, mostly documents.
You refer to "external hard drive contents as sub-collection for offline browsing", I may have misinterpreted your message but to my untutored mind this is asking for trouble.
A collection isn't a copy of the file, but a set of connections to the source files. If the files are not attached to the PC, isn't DOPus is going to beat itself to death trying to find them?
So, rather than feeling frustrated at problems that you have with your way of using DOpus, think about better ways of managing your files that do not cause these problems.
My collections are small sets of files that are relevant to a particular project, or that DOpus creates when searching.
There is any files (archives, music, video, etc). I know that collection is only links to files, but "Member" type will disappeared after disconnecting external hdd, because I try to use subcolletion for offline browsing folders and see what files I have.
I suggest using disk cataloging software for that purpose, like WhereIsIt (I use myself).
In other words, as I suggested, analyse your requirement and find an appropriate tool.
In this case, the suggestion is that you try Where Is It? which describes itself as software for "Catalog and organize your disks collection".
I wonder, though, if this is the sort of feature that the next version of DOpus might incorporate. It is, after all, a file management issue.
Would another option be for the user to create Libraries on the fly? In other words, don't keep one there hogging your memory, but build one using search as required. As Leo pointed out earlier, the original idea behind a Collection was to capture search output, among other things.
We're considering changing the collections system to not keep collections in memory unless they're being used; the advantage is obviously a lot less memory used (in these extreme cases anyway), the disadvantage is they would no longer be able to track changes to files they contain. Any comments?
Well, introduce static collections maybe, where all items need to be manually assigned/unassigned and no change tracing is done. Such collections would require running "synchronization" with source disk manually.
Or (specifically for browsing offline resources)
Implement filesystem caching, where all filestructure with appropriate attributes can be cached and browsed offline.