I did before my 1st post with a 1,7GB file - only at 100% there's a little waittime before dialog closes (cache), but w/o freezing.
Did you try a another Stick or a ext. HD? I saw these issues on both type as mentioned before.
Ok, so you even can reproduce the problem, but it depends on the kind of target device. I tried several devices and indeed the intensity of the non-linear progress wasn't the same in all cases. I assume it depends on the write performance of the target device because I can even reproduce the problem using two different HDDs (no USB at all).
I found out that the freeze time (when the dialog disappears) can be avoided by just closing the destination tab immediately after the copy operation is started. I think in this case DOpus will not try to read the destination concurrently with the flushing caches.
Each time when copying a large amount of filedata (30-40 GB) with DOpus the progress bar on my system doesn't increase linearly. Instead it starts quickly, than it slows down (e.g. 10%) and stops for 30 seconds. Than it continues and runs quickly from 10% to 20% and stops again. This goes on up to 100%.
Each time when the progress bar is stopped the LED of the target device is blinking and if I try to browse the target device using DOpus itself it instantly freezes (shows the busy cursor) until the next step begins. If I do not access the destination device I can use DOpus without any freezes during the whole time (the copy takes).
I tried the same with TotalCommander and it looks very similar (progress bar jumps and everything lags when browsing target device simultaneously). Than I check the preferences of TC and I change the copy mode from "default" to "big file mode" and I restart the copy operation. This time everything keeps responsible and the progress increases very linearly from left to right in 1% steps. I even can use DOpus to browse the target device while TC is copying in the background. By the way: TC looks like a seemingly dead dinosaur aside of DOpus on the same screen. 
Maybe this will help fixing the problem!?
I do not know how TC exactly read/write the files in "big file mode" but it performs very nice even on systems having large memory available for file system caching. In addition I noticed that the "big file mode" appears to be slower for many thousands of small files than the default mode.
So if DOpus will provide an uncached copy operation too (hopefully) I suggest to introduce a configurable threshold. Files smaller than the threshold are copied cached and files larger than the threshold are copied uncached. I think this will provide the best of both worlds and even will outpace the current solutions provided by the competitors.
Dylan42, are you using the same hardware as AKA-Mythos? (Are you both members of the same team which was mentioned? I got that impression but not sure.)
It seems like the target device (or its drivers/firmware, or perhaps the OS itself) is being overwhelmed by the filesystem flushing its write-cache, and allowing that to starve concurrent read requests to the drive, even for tiny amounts of data, which is a poor design.
As already said, we'll see what we can do to work around this problem. I'm just interested in knowing if it only happens with particular hardware.
(FWIW, a few years ago when SSDs were still quite new, I had a rubbish one which would stall the entire OS every few minutes because it could not cope with the throughput of even normal Windows usage. Some storage hardware isn't very well made, it seems. Newer SSDs tend to be good, however.)
I got that impression too. 
But at least my team members are all around here busy in working.
I'm at the same company but in a different division like Mythos. Some weeks ago he made a public recommendation to other teams to evaluate DOpus to find out how it meets our use-cases. I'm pretty sure, that it's not the same hardware because our systems are much older and we changed from Dell to HP in the meantime.
Okay, thanks!
I really like the idea of having a threshold for two different copy strategies. But I propose to let the default value (if not overriden by the user) depends on the physically installed system memory.
I like the idea too, but I would prefer a pre-defined threshold of e.g. 128 MB that can be overriden by the user.
I do not like these magic self-adjusting settings.
Would buffering make any difference on small files, either way?
[ul][li]If it does:
(e.g. Is allow several new small file to be written into the write-cache before the old one has finished being written to disk.)
It may make sense to disable it for small files as well, if the aim is to make the progress bar accurate on slow USB drives with system large buffers. Of course, this may slow things down, but you may not be able to have maximum speed and unbuffered writes with completely accurate progress bars.
And what happens if you have a mixture of large files and lots of small files? If 10,000 small files hit the systems write buffer, is that any different to one large file hitting it? Will performance be degraded if those 10,000 small files are in the write buffer while an unbuffered large file is being copied to the drive at the same time, competing for the resource?
[/li]
[li]If it doesn't:
Why bother with the extra logic at all? Just do everything unbuffered if a slow device is detected.[/li][/ul]
Are there any good benchmarks to prove the behaviour with small files, one way or the other?
Or to prove that the amount of system memory plays that big a factor in this scenario? (I have 16GB in this machine, but have also used ones with 1GB and 2GB recently.)
I think there is a lot of guessing going on in this thread (including my own guessing), and these theories need testing properly before we implement any changes.
What I have observed is that writing thousands of small files unbuffered leads to an additional effort because the master-file-table (MFT) has to be updated for each file. On the other hand, if these files are written buffered there are bulk-updates made to the MFT which performs much better but the system cache may block processes due to the long flush duration.
I have a solution in my mind that is based on a configurable threshold that plays a double-role:
-
The threshold specifies for a single file which copy mode should be used (buffered for smaller or unbuffered for larger files).
-
If only smaller files are copies (but many of them) the file sizes are summed and if the threshold is reached DOpus will forcly flushes the write-cache.
This should even work very well for mixed-sizes filesets.
Going this way provides a linear progress when copying large files without filling the system cache with unwanted data and on the other hand thousands of small files are copied very well since the MFT can be updated in a bulk. Additionally the extra flush (when the threshold is reached) prevents the system caches from growing to its maximum.
Do you have any benchmarks of documentation to back up those theories? They sound plausible, but I've found this stuff often doesn't work the way you might assume.
(For example, I once spent a weekend writing a "fast copy" proof-of-concept which copied lots of small files in parallel instead of in series, assuming the per-file overhead was causing delays. It turned out to be no faster than doing things normally, because the filesystem was serialising the file creations anyway. Until ideas like that are tested, they're just ideas. Sometimes ideas to speed things up actually slow things down, too.)
Sure, I made several tests on my own machine as well as implemented such stuff for our projects.
I performed a quick test on my own machine for you by copying files in the console and access the destination device in parallel with DOpus (only in the first run). You can easily perform these tests on your own machine by using the Power-Shell. I have repeated each test 20 times and dumped the fastest run.
MANY SMALL FILES TEST (BUFFERED + UNBUFFERED)
Source:
D:\Prj\ (contains 12.840 files, total-size of 987 MB)
2x Intel SSD (intern / RAID-0)
Target:
T:\Test
1x WD HDD (intern)
PS D:> Measure-Command {xcopy.exe D:\Prj*.* T:\Test\ /e}
BUFFERED: System lags a little and Perfmon shows caches growing. DOpus lags heavily, browsing the destination in parallel is only possible with short freezes (2-3 seconds).
Days : 0
Hours : 0
Minutes : 0
Seconds : 16
Milliseconds : 485
Ticks : 164852579
TotalDays : 0,000190801596064815
TotalHours : 0,00457923830555556
TotalMinutes : 0,274754298333333
TotalSeconds : 16,4852579
TotalMilliseconds : 16485,2579
PS D:> Measure-Command {xcopy.exe D:\Prj*.* T:\Test\ /e /j}
UNBUFFERED: System runs smooth and Perform shows constant caches. DOpus lags a little bit but browsing the destination in parallel is possible (near instant).
Days : 0
Hours : 0
Minutes : 3
Seconds : 28
Milliseconds : 757
Ticks : 2087572198
TotalDays : 0,00241617152546296
TotalHours : 0,0579881166111111
TotalMinutes : 3,47928699666667
TotalSeconds : 208,7572198
TotalMilliseconds : 208757,2198
SINGLE LARGE FILE TEST (BUFFERED + UNBUFFERED)
Source:
D:\Backup\System-0.tib (single file; 5.2 GB)
2x Intel SSD (intern / RAID-0)
Target:
T:\Test
1x WD HDD (intern)
PS D:> Measure-Command {xcopy.exe D:\Backup\System-0.tib T:\Test\ /e}
BUFFERED: System lags a little and Perfmon shows caches growing. DOpus lags heavily, browsing the destination in parallel is only possible with long freezes (10-20 seconds).
Days : 0
Hours : 0
Minutes : 1
Seconds : 8
Milliseconds : 830
Ticks : 688309249
TotalDays : 0,00079665422337963
TotalHours : 0,0191197013611111
TotalMinutes : 1,14718208166667
TotalSeconds : 68,8309249
TotalMilliseconds : 68830,9249
PS D:> Measure-Command {xcopy.exe D:\Backup\System-0.tib T:\Test\ /e /j}
UNBUFFERED: System runs smooth and Perform shows constant caches. DOpus lags a little bit but browsing the destination in parallel is possible (near instant).
Days : 0
Hours : 0
Minutes : 1
Seconds : 10
Milliseconds : 9
Ticks : 700095699
TotalDays : 0,000810295947916667
TotalHours : 0,01944710275
TotalMinutes : 1,166826165
TotalSeconds : 70,0095699
TotalMilliseconds : 70009,5699
We implemented different copy operations in the past handling a lot of files in parallel (multithreaded) and in fact we could measure a performance gain but only if we did the copy operations manually using unbuffered I/O operations using SSD or flash-memory based devices only.
Please have a look at the 4k single-threaded benchmarks in comparison to the 4k-64 multi-threaded benchmarks done by AS SSD. Multithreaded access patterns provide a significant performance boost for modern SSDs. This benchmark reflects truthfully what you can expect from your SSD when copying multithreaded (certainly you have to forget all these parallel stuff for old-fashion HDDs).
The results of my tests show clearly that (at least on my machine) two things performs really bad:
-
Copying thousands small files unbuffered -> takes minutes (unbuffered) vs seconds (buffered)
-
Copying single large file buffered -> no throughput gain, inaccessible target, intermediate freezes and messed caches
From this point of view by double-role threshold idea makes sense to me.
Thank you, that looks very helpful.
It's a really big problem for me.
When i copy files in (very) slow drives (some usb key, nas server or my mounted phone), lister freezes and i need to open an other lister.
I don't remember if it was the same thing with DO9 but it's really borring. 
I can't seem to reproduce this with any of the USB drives I have here, so I'm wondering if it is a problem that's specific to a certain brand of USB drive. If you can tell me which actual brand/model lets you reproduce it I'll see if I can obtain one.
I even can reproduce the freezes by using xcopy for copying from internal SSD to HDD (no USB at all) and DOpus only for browsing the target device (passed to xcopy). It's the same problem (except the missing progress dialog) that can be easily fixed by just passing /j argument to xcopy. In this case DOpus runs smooth and can browse the target device all the time (during xcopy performs in the background). At least for me it doesn't depend on USB instead I just need to fill the system cache faster than the destination can write to let the user interface stuttering of any application that depends on the target device.
xcopy [...] /j Copies files without buffering. Recommended for very large files.
Unfortunately /j makes xcopy performing very bad for thousands of small files, have a short look to my measurements here. Certainly I can perform another video clip showing the xcopy/DOpus setup if it is helpful for you (because it's easier to see the effect than to read about it).
I alreay have success reproducing this behaviour on around 15 machines (using DOpus in traveller mode) having very mixed hardware configurations (different CPU, HDD/SDD, video, network, but all of them with large memory configuration and using Win7 or Vista in 64bit).