I appreciate the effort you put into optimising Dopus for large file operations done in other programs a few versions ago, but unfortunately it seems some more work needs to be done.
I just created (not with Dopus) 7,462 symbolic links in a folder on an SSD while that folder was open (with a folder tree) in Dopus. These links were created in 256 subfolders with a roughly equal distribution of the links per folder (~30 per folder). These folders and links are a couple of subfolders deep on C:\ (an SSD), and the destination of these symbolic links is a bunch of files in a subfolder at E:\Blah\Blah**, where both * and * can be any hexadecimal representations of a byte (ie: 00 to FF). So it's linking to a subfolder of a subfolder that contains 256 subfolers, each with 256 subfolders of their own.
This was about 5 minutes ago , and the lister that was viewing the folder containing the folders with ~7000 links is still using 12.5% CPU (one full core), and is unresponsive.
I managed to browse to the folder with another lister, but double-clicking the folders in the detail/list view doesn't open them. Clicking on them in the folder tree opens them however. Trying to open any of the symbolic links created doesn't seem to do anything initially, but then about 30 seconds later, they load.
I just killed dopus.exe, loaded it up again, and everything is fully responsive. The folders on c:\ with the symbolic links browse to almost immediately, and double-clicking any of the symlinks opens them immediately too.
Seems like Dopus is getting bogged down updating some sort of internal structure during, directly after, and for at least 5 minutes after an external program performs many file operations.
Let me know if you need any clarification or information.
Is this a real-world situation or are you trying to find pathological cases in the change-notification code?
Either way, can you give us an easy way to do the same thing to see if we can reproduce what you're seeing?
Addition: I just remembered your old thread, which was locked for a reason. Are you only seeing this problem if you have 10 listers with 18 tabs in each and create 7,500 symbolic links? If so, as we already said, we plan to improve the change notification code in the future but this kind of pathological case is not a priority and is somewhat self-inflicted. You can disable change notification if you really need to do those things.
If you can give us a way to easily re-create what you are seeing, then that may help us improve the code, but anything else seems like ground we have already covered in the other thread.
I was not going to dignify this with a response, my wife even told me not to, but what can i say? I need this product that i paid for to perform well.
The sheer distrust exhibited on this forum is unbelievable. Yes it's a real world situation. Why would i spend time screwing around trying to get Dopus to break? The ego... Breaking Dopus, nor arguing with its developers, nor its developers, are a priority in my life. I happen to mirror websites and frequently deal with manipulating millions of files. Filesystems are actually quite useful "databases", a feature that i employ richly. I find them more efficient and easier to use than traditional database offerings.
Steps to reproduce:
Open Dopus.
Have 1 lister with 1 tab (surprised?) open with a folder tree, browse to c:\ and expand it in the folder tree.
Have a fast drive (ie: SSD) that can actually handle the I/O, as c:.
Compile and run the following code (with appropriate headers), or create a script to do the same function, since obviously i can't upload terabytes of information on my computer to you to prove this poor performance:
[code]void DopusAndItsHorriblePerformance()
{
int i;
wchar_t temp[100];
for (i = 0; i < 500; i++) {
wsprintf(&temp[0], L"c:\%d", i);
CreateDirectoryW(&temp[0], NULL);
}
}[/code]
Enjoy that example code creating 10 seconds of 12.5% single-threaded CPU usage and complete unresponsiveness of the single lister, on a 4.2GHz i7-2600K, after the directories are created.
Enjoy ~5 seconds of the same unresponsiveness when selecting and deleting those 500 folders with Dopus.
That is just a simple proof of concept, NOT what i am doing in my spare time. I actually deal with folders and files as quickly as that code can generate those folders. It's real world usage. Not common usage, but real world, yes. Dopus' performance is unacceptable.
This is not an old problem. It barely has anything to do with my old thread. You fixed that performance issue; CPU usage dropped significantly. This new performance problem seems related to the folder tree and started happening during one of the latest 10.5.0.* betas i think. That's why i started a new thread.
Your passive-aggressive threat to lock this thread has been noted, by the way, and taken as an insult. I won't respond further unless there is a direct need. I'm not going to bump this or push the issue. I'm not going to start profiling or debugging myself. It's been months since my last thread, so i hardly would call this spamming a "non-issue" as you would almost call it, especially since this is a new issue.
a) It takes a tiny fraction of a second to create those 500 folders (in the example) in an external program, which causes Dopus to choke solid for 10 seconds on a 4.2GHz core. A non-default CPU speed that most people do not even approach, so they would have to wait longer than 10 seconds.
b) It's not a "change-notification code" problem that is "self-inflicted", because Dopus chokes again trying to delete those 500 folders for a further 5 seconds. What, Dopus is having trouble "notifying" itself of things it did itself?
c) No i am not turning off the absolutely-essential-to-any-file-manager option that would disable updating the lister for any external change to any file or folder. What sort of ridiculous notion is this? A file manager that can't handle processing external changes to the filesystem?
I would say this is a pretty serious issue. I'm sure a lot of people out there would like to know just how much CPU usage, thus power, thus money, Dopus is costing them. They don't see it because they unzip their little 3 folder zips or whatever it is that most people do. It just so happens that i work with filesystems extensively, so i can see the performance issues bunch up into a horrible mess.
Oh and just so we're super clear, i know i said that i wasn't going to post again. But, i'm not going to be intimidated by threats of thread-lockage, into not posting about an issue i'm having with software i've bought. Since you have no support system in place whatsoever, save for this forum, then this forum post is technically my "support ticket", and i will update it as i see fit. Locking this thread or posting inflammatory dismissive responses would simply show how pathetic the attitude you have towards supporting your own software.
Yeah... everyone is entitled to their right to react however they want, but I'd rate this a fairly gross overreaction.
Amidst your own inflammatory remarks you mentioned something that caught my attention - that you thought the problem might be "related to the folder tree". Have you found this to be a non-issue or otherwise notably mitigated if you turn off the folder tree?
I ask because I reported a severe folder tree specific lag in performance during far earlier betas (like 10.0.x) as well as again somewhat recently... In my case, it has to do with a slower external USB drive - though it might not be a relevant factor - it's just where I have LARGE numbers of files and folders that invites the type of workflow that contributes to me experiencing the issue. What I've seen is when I'm moving several hundred folders from one folder to another on the same drive (often just a dir level or two UP in the same folder path) that there is AT LEAST a 10x lag in time to complete the operations compared to doing the same exact thing with the folder tree turned off. It's VERY conspicuous as the refresh of the folder tree to show the movement of each folder is atrociously slow. It's been so much of a pain - that I would end up cursing in frustration approaching your apparent level of disgust each time I ~forgot to turn off the folder tree before such moves (which I had otherwise settled on doing since there was no fix in Opus for this).
I don't come across it as often anymore because there was an extended period of time where I was making mass changes to large groups of a VERY large media collection. I've more or less completed that project now, so it's rare that I have similar scenarios where the same thing occurs, though it does still happen from time to time to some extent depending on the work.
Turning off the folder tree does indeed not only mitigate, but completely eliminate, the lag. It's a PITA though to have to do that. I have actually been completely closing Dopus while doing large file operations in other programs, just to avoid the horrible CPU usage and the virtual destruction of any usefulness of Dopus listers (since they freeze and don't recover for a long time). Not really an ideal solution though, is it? Regardless, thanks for the input! Turning off the folder tree should do much nicer than closing Dopus completely. Thanks a lot!
As for the gross overreaction bit, you really need to know my history with these GPsoft guys to not see it as "gross". They know what i'm talking about. And as for inflammatory, i'm a customer, i'm allowed to be. I'm always right, you know? What's frustrating is that nobody around here has any supervisors and such to answer to. A distinctive "con" when dealing with tiny companies.
By the way, i just want to say that i positively cannot stop laughing at your words: "cursing in frustration approaching your apparent level of disgust". Something about it is just hilarious. I can't quite put my finger on it, but it's just that something can be so difficult to deal with that it becomes "disgusting". That's gold aaahahaha!
Aaaahaha, you guys! Crackin' me up all the time. Rollercoaster of a ride, i tell ye. You're right though, i do drivel on a bit, even after i've already apparently finished and am done. I always forget stuff. Hindsight, you know? The "Quick Reply" button is to blame. So easy to just add a bit here and there.
Off-topic for this forum, but on-topic for this thread: I write pages and pages of stuff (with bits inserted saying "Ok i'm shutting up now so you can read back" here and there) to friends on MSN/Yahoo/Skype/etc while they're not there, and they come back and tell me to hold on while they read my "essay". It's just who i am.
Sorry about getting inflammatory before. I do work myself up sometimes and it's a bit unreasonable sometimes. We can all be friends here. I'm not out to persecute anyone. I just want my stuff to work right, 'tis all. Trying to get things to work right with blood pressure (medical) issues is not a nice combination. But it's all good. I'm off to bed soon, so i'm currently hallucinating on sleeping pills, so i'ma leave this party for the night. Probably.. More than likely.. 85% chance that you won't see another post from me tonight. Jon what the hell are you doing up anyway? It's almost 1 AM on Australia East Coast.
Okay, there might be an issue here since it's happening when creating lots of directories. That's very different to the original post about creating 7,000 symbolic links. (Symbolic links massively slow down change notification since they add extra checks for every branch of the path and creating that many links pointing all over the place seems unwise for a lot of reasons, and definitely not normal.)
I've done some testing and the delay happens if you have the folder tree expanded to show the branch where the hundreds of new folders will appear. If that branch isn't expanded then there's no delay. We'll look into that and see if we can improve things, maybe by batching up the tree insertions.
I also tried with Explorer and found that it completely ignores the folders being created, as far as the tree is concerned. They don't even appear with an F5, which seems even worse than a few seconds of CPU usage.
Video shows:
[ul][li]Opus: Creating 500 folders with the tree open and the place they'll appear in shown in the file display but not expanded into the tree. (No delay, minimal CPU usage.)[/li]
[li]Opus: Doing the same, but with the branch expanded. (Some seconds of delay while the tree sorts itself out.)[/li]
[li]Explorer: Doing the same as above. (Its tree doesn't even notice the folders were created.)[/li][/ul]
(The first time I tried to delete the folders using Explorer it took about 30 seconds, too, although that seemed to be a one-off.)
LOL... lots of passion - thats for sure . To each their own!
In any case, it appears to be the same or similar issue I've reported in the past - and I rather imagine it goes back to implementing your own custom folder tree in 10.x? I don't recall ever seeing this in v9 when you were using Explorer for the folder tree, and I had certainly been performing the same kind of mass-folder-move operations back then. It's probably something of an outer edge situation to be a ~common occurrence for most people, but hardly a pathological invention of the mind either . Also the sort of thing that's sure to cause a fair build-up of frustration over time if such a workflow IS part of ones' daily routine...
Indeed, creating 500 directories is unusual but not pathological. I was talking about creating 7,000 symbolic links.
It's interesting that one of you thinks this is a brand new problem while the other thinks it's been there a couple of years, but it doesn't really matter where it came from, only that it is. We're looking into it.
I don't think there's anything to be gained from talking about this further, we just need to investigate and see if we can improve the performance in this scenario. For now, it's easy to avoid the problem by closing the branch of the folder tree, if you create hundreds of folders a lot and don't want to wait a few seconds.
I'm flopping all over the keyboard but i'll do my best to say this:
Thanks a lot for your thorough tests and video presentation to boot!! You put a lot of work into that and i appreciate it. It shows me that you're taking my report seriously and are actively trying to get to the bottom of it. I seriously, honestly, appreciate it a lot. Thank you very much!
In the video i see the lag that happens when the 500 folders are created is roughly equivalent to what happens to me. It's great that you've narrowed it down even more than i could.
I didn't seem to catch the delay when deleting those 500 folders. Did you try that? That should be quite a few seconds deleting those. I think that's quite important, because that shouldn't be dealing with any notification change code or whatever it is. Though i may just be too "inebriated" on sleeping meds, i might have missed you deleting them.
Anyway, it's all good. Thanks for your effort. I hope you can narrow it down further and get some stuff fixed up. I see that you mentioned you're looking into it, so we can leave it at that.
As for Explorer.. Why do you think i use Directory Opus? Because it's bloody brilliant in comparison! Doesn't surprise me that Explorer messed it up.
And steje, thanks for the input and advice and humor. I couldn't have gotten by without them. But, for sure, you're right that, in the past few days i've been dealing with RARing and UnRARing millions upon millions of files and folders, processing them, deleting them, etc etc. It has been a daily routine for me for the past week, and is mighty frustrating.
Ok i really, really have to sleep. Good night everyone! Thanks for everything, despite my insolence!
Not sure if related, but I see slowdowns regularly when working with (a lot of) files. I have a collection of files in folders which are divided into subfolders, which are divided, etc. For example, one such collections consists of roughly 500k pictures in 18k folders, totalling some 450-ish GB. The biggest number of subfolders in one folder is actually around 500, though they are not created new all the time, the number of folders just grows over time. Other file-collections of mine are not pictures but I see the same behaviour, which is why I'm not saying "pictures" but "files".
Every so often, I download new files and sift throught them to see in which folder (in my collection) I think they belong, renaming the folders where I downloaded them into, moving the files or folders to the right place, etc. When doing so, I have the tree and the viewer pane opened in the lister. Now, I don't know what ecactly triggers this, I just know that, when working on this, Opus will eventually become unresponsive when moving a lot of files at once. This does not always mean big updates to the tree, often I'm just moving files from one folder to another. It's a gradual process, Opus starts becoming more unresponsive the longer I keep working.
Eventually, I have to exit Opus completely (not just closing the lister) and re-open it, just me make it responsive again. doing that frees up some memory, usually a couple hundred MB.
That sounds unrelated, and is most like a shell extension going wrong. (Easy to check: If the slowdown remains when the tree is turned off, then it's definitely unrelated.)
The Other Troubleshooting section of the FAQs has some guides which may help, depending on exactly what's causing the slowdown.
We've made some changes for the next release which should greatly improve the speed that the tree processes directory creations when there are lots of other folders in the same branch of the tree.
Thanks man! I'll give it a go against my use-case and see how things look. I'll repeat a few operations pre and post upgrade to the next beta to get a sense of the level of improvement and let you know. Thanks for getting to it...
, and the lister that was viewing the folder containing the folders with ~7000 links is still using 12.5% CPU (one full core), and is unresponsive.
I'm 
What's frustrating is that nobody around here has any supervisors and such to answer to. A distinctive "con" when dealing with tiny companies.
Crackin' me up all the time. Rollercoaster of a ride, i tell ye. 
You're right though, i do drivel on a bit, even after i've already apparently finished and am done. I always forget stuff. Hindsight, you know? The "Quick Reply" button is to blame. So easy to just add a bit here and there.
Probably.. More than likely.. 85% chance that you won't see another post from me tonight. Jon what the hell are you doing up anyway? It's almost 1 AM on Australia East Coast.
