Unfortunately I'm not able to reproduce these symptoms. As an example, I copied a 3.3GB file from an SSD to an HDD using xcopy (buffered, as in your example), and the destination was fully browseable in Opus the whole time with no perceived lags at all.
What kind of files are you copying? Are they large video files or archives, for example?
It definitely is not normal for a file copy between two fast devices to cause the second device to be unbrowsable. There must be something specific about the type of files, or the Opus configuration, or something extra installed on your system which is key to seeing the same delay.
For example, in the past I've seen issues where partially-copied video files would cause some video codecs to go crazy if Opus asked them for information about the files, when they appeared in a folder it was looking at, and that folder had columns which displayed info about video files (e.g. description, dimensions, bitrate, etc.).
Or maybe the target drive has volume shadow copy (system restore points / previous versions) enabled, and copying a large amount of data to it is forcing the OS to make room by clearing some of that data, which is slowing down access to the drive while it happens.
Or something low-level is choking on the amount of I/O. (Could be storage controllers -- both drivers and physical hardware -- or anti-virus etc. And with VMs that could include the host OS and real hardware/drivers as well as the guest OS and virtual hardware/drivers, and the VM software itself.)
There has to be something unusual like that going on as it just is not normal to see this on a HDD target. If it was, we all would be seeing it all the time when copying large files around. It's completely normal for the source device to read faster than the target device can write. Even without the SSD->HDD combo (which we have ourselves and exercise regularly without problems), that's true of most device combinations. Even the same brand of HDD as both source and target meets the criteria, since writing is almost always slower than reading for the same class of device.
Here's a video of me copying several large files from SSD to HDD while still being able to browse the target HDD, on a machine with 16GB RAM:
leo.dopus.com/temp/Copy_SSD_to_HDD.mp4 (23MB)
You can see the progress jumps straight to 2GB copied right at the start (I think the source files were probably still cached in memory after copying them to the SSD before the test), so a large write buffer is definitely in effect here, and the target drive is definitely writing slower than the source is providing the data.
Sorry, I too can't reproduce any of these problems. Going from internal SATA2 SSD to internal SATA2 7200RPM, External USB2.0 7200 & 5400 RPM and network 5900 RPM on an unRAID server which has to calculate and do the unRAID math at the same time. Large or lots of small files, never lags and I'm able to keep browsing/using Dopus. Running the latest Dopus and Win7 x64 Pro.
The progress bar lags on my system (Vista 64 / 8 GB) in a smimilar way as shown in Leo's video. It jumps stepwise from left to right and the throughput shows values in the range of 100 kb/s up to 80 mb/s. Thus the system's write cache seems to be filled and flushed alternately when copying large files, which doesn't make sense at all for me? Is there any advantage in copying large files pushing them through the write cache? Even if it does not freeze, the caches are filled with useless data and the progress shows artifical values and disappears much too early!?
I played around a little bit more and even if copying a single 8 GB files doesn't lead to any delays here (but jumping progress bar), copying of 16x 500 MB files to an USB stick provides 5 seconds freezes when accessing the stick in parallel with DOpus. The busy cursor is shown and I can't do anything, after 5 seconds the device is refreshed and everything is quick and fast again. This happens multiple times before the overall progress reaches the end and the dialog disappears. The progress increases stepwise in 500 MB steps.
My system is an old Core2Duo with 64 Bit WinVista and 8 GB memory, 1 TB HDD and only USB 2.
How can I create such cool videos to show the effect?
[Note: Most of what I say below talks is about write buffering in the context of individual files, not the setting that affects the entire device, which is different.]
Turning off write caching is complicated. It will also slow down some operations, so it doesn't make sense to simply turn it off.
We might turn it off for some other operations, if we can reproduce the problem it is said to be causing and be confident that nothing will go wrong, but only then.
It's not that we have intentionally turned write caching on. It is on by default. Turning it off is an extra step and requires extra logic in the copy code which may make it fail on some devices.
It isn't a simple case of flicking a switch in the code. It is much more complicated than that. Not just because a cut-off point has to be chosen about which files should use the buffer and which should bypass it. Windows is designed to assume everything will use write buffering, except for highly specialised tasks where you want to manage the buffering yourself (e.g. databases). As soon as you turn off write buffering you have to deal with low-level details of the filesystem, which vary from device to device, and which are normally abstracted away by the OS. When a program writes files with buffering off, it has to be careful how it writes the data, or the writes may fail.
(IMO, if the write buffer really is causing problems for some people, then Microsoft should really provide a "small write buffer" mode which still gives you the basic abstractions -- so you can just write data into the file without worrying about sector sizes, data alignment and file sizes -- but this is Microsoft we're talking about and they probably already know this, don't care, and will never do anything to help. 
In an age where multi-gig files are completely normal, it seems off if the operating system's default modes of writing data cause such problems, and no alternatives have been provided, but perhaps that is just how it is. I can't help feel that something is being overlooked, though.)
Some of that is just the result of how/when the speed timings are made, combined with how the buffering works. Don't read too much into the "current" speed information; it is just for guidance. The average speed is more useful. The current speed is how fast Opus can write data into the write cache. If the write cache fills up then Opus will think it is no longer writing data, so the speed will be low for a moment, but the write cache is probably being written to disk (by the OS) at a fairly constant rate. There is no way to actually measure that rate, though. That information is not provided to applications. Apps can only know how much they have given to the operating system, not how much the OS has actually written to disk. (Unless buffering is turned off, which is not always what you want, as discussed above. And even with buffering turned off on the file, the HDD hardware itself still has its own write buffer...)
If you can reproduce the problem then please provide details for some of the things I asked about to help us try to reproduce what you're seeing.
May be it would help, what I recently found out:
Even if I can reproduce the issue on my workstation I can not reproduce it on my notebook (only 2 GB RAM). But recently I connected an external drive that was encrypted with TrueCrypt. And the same problem as seen on my workstation happened on my notebook too. I tried the encrypted drive on my workstation too and the symptoms were increased (longer freezes).
Maybe the encryption driver of TrueCrypt provides additional buffering in a way, that the symptoms are intensified.
Leo, please try the following (as I did):
- Format an external USB drive using TrueCrypt in partition mode (no file container) using NTFS filesystem (default sector size for large files).
- Mount the encrypted partition as removable drive (under mount options) to your system.
- Copy a 2 GB file to the external drive using DOpus.
This lags heavily on my notebook with only 2 GB memory in the same intensity as it lags on my workstation without TrueCrypt involved.
Good Luck! 
If I try to browse the target device immediately after the copy dialog disappears DOpus freezes completly showing a busy cursor for a long time. If I use TeraCopy for the copy operation DOpus can access the target device during as well as after the transfer.
Hope this will help.
I think bringing TrueCrypt into this just muddies the waters. Things like TrueCrypt and BestCrypt add another layer of complexity and possible delays/bottlenecks and unusual behaviours. That is especially if their virtual drives are mounted via files on top of a second filesystem (where if there is a delay, it could be either filesystem at play). They can also bring a CPU dependency into things which will confuse the issue further (every piece of data read or written to their devices has to be decrypted or encrypted by the CPU). They are great tools, but I think they are best left out of this.
I don't care much about progress bar lag, either. I care much more about freezes. Progress bar lag is a natural result of caching, and caching is usually a good thing which speeds up the operation. Freezing is, of course, never a good thing.
Nobody seems to want to answer some of the questions I asked above...
I just tried with BestCrypt anyway, and I don't see any problems with it.
Sorry to say that, but TrueCrypt doesn't make any differences on my systems. We use TrueCrypt to encrypt the external devices intended for mobile usage. Thus it was simple to perform a re-test according to your step-by-step tutorial at least on my own workstation and my test system sitting aside of me.
So you're not seeing this all the time on your systems?
Or when you say TrueCrypt didn't make a difference do you mean the problem is there with and without TrueCrypt, rather than not there?
So you're not seeing this all the time on your systems?
Or when you say TrueCrypt didn't make a difference do you mean the problem is there with and without TrueCrypt, rather than not there?[/quote]
TrueCrypt doesn't raise the issue if the system isn't affected and TrueCrypt doesn't fix the issue if the system is affected. 
Leo, I haven't been in front of any computer here for a couple of days, because there was a public holiday here in germany bringing a long weekend to us.
All my tests were done with large binary files of different type (e.g. *.tib, *.iso, *.tar, *.vmz). I even tried a 2 GB dumpfile created with random data and renamed to *.bin.
I tested on my own workstation with virus scanner enabled / disabled as well as on some workstations in our team as well as on my test workstation with a fresh OS install. Even on the test workstation with fresh Win7 Ultimate 64 Bit installation (no virus scanner, no video codecs, etc.) I can reproduce the problem. But to be honest we have a Vista 64 Bit workstation having 16 GB memory and it's impossible to reproduce the problem on this machine.
I'm pretty sure that there is "something" (let us call it a driver) on these systems that is installed directly by Windows setup. This driver seems to enforces to fully flush the write caches before any read access can made. This perfectly matches to the fact that my system doesn't freeze if I close the target tab instantly after initiating the copy operation. The freezes only occurs if DOpus tries to access the target device while the buffers are flushing.
Leo, instead of investing many hours trying to reproduce the problem, may be we should go the opposite way:
Is it possible to provide a test-build of DOpus providing a quick-and-dirty copy command (argument) that uses a trivial unbuffered copy operation? This test-build can be provided to people having that problem to see if it can be fixed by using unbuffered writing. Actually the command doesn't need to copy real data, I would assume that it is enough to juse write zeros unbuffered or something else what might be easy to implement.
Maybe this would save some time?
Yep, we're thinking about doing that.
I would appreciate that too!
Maybe you can try it yourself by creating a copy command for large files
using xcopy.exe in unbuffered mode.
xcopy.exe {filepath$} {destpath$} /e /j
I tried that and the same files DOpus can not handle on my system (without
any freezes) can be smoothly copied using the external xcopy command
initiated by DOpus. It works even fine if I access the target device in
parallel using DOpus.