I tried that and indeed it works very well. Everything is responsible during xcopy is doing the job in the background. If I remove the /j the freezes are back and it's impossible to smoothly browse the target device.
Leo, is there a control code to show a simple progress dialog for external commands be applied to multi-file-selections (some kind of "4/12 files processed")?
I'm not sure if the xcopy /j suggestion is for me or other people, but it was already mentioned as a workaround much earlier in the thread.
If it was for me, I can't reproduce the problem in first place, so trying something that doesn't cause the problem won't help me. 
You should get that automatically if you use {filepath$} in the command, as long as you don't set it to run as an MS-DOS Batch function. But then you'll get a new command prompt flash on the screen for each file.
That is off-topic, though, and this is already a monster thread.
You have to prefix the xcopy command with @sync to let the progress dialog appear. Otherwise all xcopy tasks will be forked in parallel.
Try this:
@runmode:hide
@sync:xcopy.exe {filepath$} {destpath$} /e /j
[quote="AKA-Mythos"]
You have to prefix the xcopy command with @sync to let the progress dialog appear. Otherwise all xcopy tasks will be forked in parallel.[/quote]
Thanks again for the quick reply. This works very well as workaround.
Unfortunately it does not. Your solution performs very bad for a lot of small files to be copied as already discussed in this thread.
Have a look here ...
Hi,
This sounds like an issue similar to my copy problem, 40 Gig total, of which several up to 2 Gig data bases. [url]Use of TeraCopy]
I think it would be a great plus for DOPUS to add TeraCopy copy/move characteristics. DOPUS is super, and I can't stop raving about it, but why leave this single gap, which is a serious one?
Please guys, put this on the high priority list.
It is already, don't worry.
It's just difficult for us to really know what will work as we don't have any problems on our own machines. So it might take some experimentation in beta versions, to find what works best without slowing other things down. Once we put out a version for testing, feedback will be very useful to know how well it worked.
(For example, TeraCopy is faster for some situations and slower for others, and the same speed most of the time. There isn't one true way to copy files, unfortunately.)
Personally it would make me happy to see an uncached copy operation to let DOpus perform nicer in multitasking environments.
In other words, the current implementation how DOpus copies large files really hurts other processes.
In practice there are other processes aside DOpus that highly depends on the data in the system cache. But after copying large files using DOpus all the caches are filled with useless data and the other processes have a real handicap.
I have a complex maven build that is run on my machine 40-50 times per day during common software development. The build depends on about 8.000 files placed on a HDD that all are read during the build execution (compilation). Fortunately most of the files are maintained in the system cache, thus the first build takes very long (minutes) but the subsequent builds are much faster (seconds) because the harddisk is rarely accessed. After copying a large file with DOpus the system cache is cleared and the next build will take minutes again (similar to the first run).
Test Run:
System power on
Start the build (mvn install)
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3 minutes 15 seconds
[INFO] Finished at: Thu May 03 09:45:38 CEST 2012
[INFO] Final Memory: 235M/538M
[INFO] ------------------------------------------------------------------------
Start the build (mvn install)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 46 seconds
[INFO] Finished at: Thu May 03 09:48:14 CEST 2012
[INFO] Final Memory: 385M/532M
[INFO] ------------------------------------------------------------------------
Start the build (mvn install)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 31 seconds
[INFO] Finished at: Thu May 03 09:51:51 CEST 2012
[INFO] Final Memory: 365M/525M
[INFO] ------------------------------------------------------------------------
5. Copy 5 GB using DOpus
Duration: 83s
Start the build (mvn install)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3 minutes 11 seconds
[INFO] Finished at: Thu May 03 09:59:09 CEST 2012
[INFO] Final Memory: 261M/478M
[INFO] ------------------------------------------------------------------------
Start the build (mvn install)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 38 seconds
[INFO] Finished at: Thu May 03 10:01:21 CEST 2012
[INFO] Final Memory: 297M/568M
[INFO] ------------------------------------------------------------------------
8. Copy 5 GB using xcopy /j (unbuffered)
Duration: 87s
10. Copy 5 GB using xcopy (buffered)
Duration: 81s
Good Point!! 
I have never realized or even thought about that, but I'm definitely affected by this disadvantage because copying large (database files) and compiling/testing is my daily business.
Thanks for drawing these concequences. 
We know.
We've already said we'll do it.
As this thread keeps repeating itself, I'm going to lock it for now.
Once we have a test version with the unbuffered changes, please try it and send us feedback on how well it works.
A beta is now available with non-buffered copy functionality.
Currently you need to enable this by setting the file size threshold (in megabytes) above which files will be copied non-buffered. This is configured from Preferences / Miscellaneous / Advanced / copy_nonbuffer_threshold.
Please report back your findings!
Sorry to say that, but there is no difference between 10.0.5 and 10.0.5.1 on my sytem. I set the threshold to 65536 KB and tried to copy a 1 GB file and the progress bar is filled instantly to 70% and DOpus freezes.
How did you manage to configure KB if the field expects a value specified in MB?
If you set the threshold value to 65536 you'll need to copy a file greater than 64 gigabytes to see any difference.
A first quick test on two systems here give very nice results:
I even perform a re-test of my initial scenario (as shown in the video of the first post), there are no freezes and the progress increases smoothly. Looks and feels fantastic.
The only freezes I can reproduce appears when copying a lot of 10 MB files and a threshold set to a larger value (e.g. 40 MB). In this case after copying some files DOpus freezes again.
I suppose that the threshold isn't used to intermediately flush the caches if only small files are copied.
Thus I propose to forcly flush the write cache if the threshold is set and the amount of copied files exceed the threshold.
If flushing the caches doesn't work reliable you may try to copy each n-th file unbuffered if the threshold is reached by copying may small files buffered.
In other words when copying 200 files of 1 MB size and the threshold is set to 40 MB, the 40, 80, 120, 160 and 200 file will be copied unbuffered (and the caches are flushed in addition).
I quote myself, since I already wrote a posting related to the idea of force-flushing when copying small files:
[quote="AKA-Mythos"]What I have observed is that writing thousands of small files unbuffered leads to an additional effort because the master-file-table (MFT) has to be updated for each file. On the other hand, if these files are written buffered there are bulk-updates made to the MFT which performs much better but the system cache may block processes due to the long flush duration.
I have a solution in my mind that is based on a configurable threshold that plays a double-role:
The threshold specifies for a single file which copy mode should be used (buffered for smaller or unbuffered for larger files).
If only smaller files are copies (but many of them) the file sizes are summed and if the threshold is reached DOpus will forcly flushes the write-cache.
This should even work very well for mixed-sizes filesets.
Going this way provides a linear progress when copying large files without filling the system cache with unwanted data and on the other hand thousands of small files are copied very well since the MFT can be updated in a bulk. Additionally the extra flush (when the threshold is reached) prevents the system caches from growing to its maximum.[/quote]
If a file is smaller than the threshold then it will be copied using buffering, exactly the same as before.
Try setting the threshold to 1 so that only tiny files still use buffering and everything else is non-buffered. That's what I have done and it seems to work well.
Yes, but a threshold of 1 provides a very bad mass-data copying performance on my system because there are thousands of MFT writes (head seekings) when copying to a HDD.
Anyway the problem of freezing when copying a single large file seems to be fully fixed now. Thank you very much.
But the problem of freezing when copying a lot of small files (e.g. 500 KB) is still there. It looks similar to my initial video. But I assume that this can be fixed easily by introducing an extra-flush each n-th file.
Is that based on observations or just an assumption? Microsoft use an even smaller threshold of 256KB in Windows Explorer, for example.
I don't think whether or not the data content of a file is buffered has any effect on when the MFT metadata is updated. They are separate things, as far as I can tell (and the MSDN API documentation indicates that in places, too).
Ups! 
I should take a coffee before performing test-runs.
It works like a charm now, I can copy 1 GB, 2 GB, 5 GB as well as 10x 4 GB files smoothly.
Anyway copying 15.000 source files around 2-10 KB size each is still freezing here (but the freezes are only a few seconds).