DOpus currently uses buffered IO operations to perform large copy tasks. This may work well for most cases but if the amount of files to be copied is much larger than the physically installed system memory this doesn't make sense at all and this leads to very strange lagging effects.
I took a short video clip showing how DOpus is lagging/freezing on a system when copying a large file (2 GB) from a fast source (a solid state disk) to a slow destination (a USB Stick).
You can see that the dialog disappears much too early because the 2 GB are fully buffered in the system's write cache. Unfortunately this write cache needs to be flushed before the destination (USB-Stick) will become accessible to DOpus again.
This means that the copy dialog is just displayed for a few seconds that are needed to read the whole file into the system cache. Than the dialog (showing fantastic 1000 MB/s throughput) disappears and in the background the cache is written to the disk (which may take a few minutes). In the meanwhile the user interface is totally blocked because DOpus is waiting for the destination to become accessible again.
As a side effect the actual user data in the system cache are purged and replaced by the incoming data of the copy operation (that doesn't need to be cached at all). In other words, after a large copy operation the system cache is filled with useless data and the actual user data has to be read again.
I highly recommend to introduced an "unbuffered" option for the RAW copy command that is automatically used by DOpus for large file transfers. This will fix all these annoying hick-ups during large file operations and it will also assure that the progress dialog is kept open all the time showing reliable values to the user. May be an "unbuf_r", "unbuf_w" and "unbuf_rw" as arguments for the copy command would be even nicer to control wheter the read, the write or both operations should be unbuffered.
PS: Certainly an asynchronuous (multithreaded) unbuffered IO operation would be the best in performance to match the access patterns of modern SSDs. In other words: reading/writing a SSD in parallel does multiply the data throughput by the number of threads involved.
It's up to the operating system how data is buffered. I doubt the OS would allocate a larger disk buffer in memory than there was physical memory available.
Unbuffered IO is not a silver bullet, and in the end can cause as many problems as it solves, on top of being difficult to get right for every type of device (due to the fact the program doing the copying has to handle intimate details of the device and filesystem or the copy will fail when buffering is disabled). (Microsoft themselves have flip-flopped between using and not using buffered file copy. In the end it doesn't make a lot of difference in most cases, and in others it's a toss-up between helping and hurting.)
Asynchronous copying makes almost no difference for the type of operation we are talking about. Buffering almost completely removes the benefit of async IO, and even without buffering async IO is most beneficial when both devices are slow. When at least one of the devices is an SSD, it's not going to gain much, IMO. I'd have to see some proper benchmarks to believe otherwise, at least.
Also, most async file-IO operations end up being turned into synchronous ones by the OS, it turns out. There are only very specific circumstances where file IO will really run asynchronously. Writes which extend the file are always synchronous, so async appears to be completely irrelevant to file-copying: blogs.msdn.com/b/oldnewthing/arc ... 15586.aspx
Have you tried doing the same copy using some other programs? (Not just Windows Explorer or things which use the shell file-copy code, although checking with them is still a good idea; most devices are at least tested against that code, though, so trying with it often won't reveal much.) Are you sure it is the way Opus is copying things, and not an issue with e.g. anti-virus or the device (or drivers) itself?
All Opus really does is open file handles and write data to them. The rest is handled by the OS & drivers (and anything that might intercept them like anti-virus, firewall, network driver/shardware (for network copies) etc.). Copying a 2GB file to any device should not slow down the entire system unless something is wrong somewhere in the system.
[quote="leo"]It's up to the operating system how data is buffered. I doubt the OS would allocate a larger disk buffer in memory than there was physical memory available.
[/quote]
Certainly not, but the system cache grows to the maximum size and is filled with the file data that is copied and as soon as all data is put to the buffers the copy call returns and DOpus closes the progress dialog even if there are 2 GB in the system's write cache waiting to be flushed to the disk. This isn't a problem as long as DOpus does not try to access the destination again (and it does so immediately). In this case everything is stopped because DOpus has to wait for the end of buffer flushing.
Fully agree, but it was my PS comment because I just want to note, that unbuffered async I/O is the fastest pattern for SSD file transfer as many benchmarks shows. Async I/O does not depend to the cache problem that leads to the freezes. That is only a question of caching or not.
I tested several programs on a clean test machine to find out a quick solution for myself.
Total Commander: Same problem here, unless you use a mode called "big-file-copy" that disables the write caching at all. In this case the progress dialog shows actual progress and the user interface keeps responsible.
XCOPY: Same problem here, unless you use /J option to activate unbuffered I/O (Microsoft recommends this option for large file transfer).
XXCOPY: Same problem here, unless you use /CA0 switch to deactivate read/write caching (whatever that means).
TeraCopy: Problem doesn't occur here. According to their website large files are not cached at all.
My current solution is to have a specific hotkey assigned to a batch file, that takes the DOpus selection and starts XXCOPY with disabled write caching to do the job. In this case everything keeps responsible and the progress reflects the actual state. If the task return the data was actually written and I can safely remove the stick.
Yes, but for specific use cases (like large file transfers) it is useful to provide some hints to the OS to get the maximum performance. The problem lies in the way in which the copy is performed - specifically buffered vs unbuffered I/O. So let's quickly define these terms. Buffered I/O describes the process by which the file system will buffer reads and writes to and from the disk in the file system cache. Buffered I/O is intended to speed up future reads and writes to the same file but it has an associated overhead cost. It is effective for speeding up access to files that may change periodically or get accessed frequently. Unbuffered I/O (or a raw file copy) is preferred when attempting to copy a large file from one location to another when we do not intend to access the source file after the copy is complete. This will avoid the file system cache overhead and prevent the file system cache from being effectively flushed by the large file data.
It doesn't make sense to use unbuffered I/O for reading and writing in general.
To be more precise, we are talking about the option to pass the unbuffered flag when creating the file handle of the target to only disable the write-caching (read-caching is still welcome).
Nice idea! I tried that for a specific USB-Stick and it immediately fixed the problem.
Unfortunately this isn't a working solution for us, since we are using Directory Opus in our team and copying large files to USB-Sticks is a daily use-case.
In other words, we have plenty of USB-Sticks that are used on plenty machines. Since the write-cache is a per-device setting it's impossible to do that for all permutations on all machines.
AKA-Mythos, how much system memory do you have in your system?
I suppose that the importance of this problem increases with huge system memory installed!?
I have a workstation with 16 GB memory and a notebook with 2 GB memory installed.
On my notebook the DOpus copy operation always performs very fine showing a realistic progress but on my workstation the progress is definitely messed up (stuttering and closing much too early).
After some testing I figured out that on my workstation I can instantly read up to 4 GB in the write cache within a few seconds (SSD), than the progress dialog is closed and DOpus is totally busy until the write cache is written to the destination (by the OS in the background). As slower the destination can write as longer DOpus is frozen.
As a side effect after such a copy operation switching the active applications seems to lag a little bit since parts of the process memory was dumped to the pagefile to assign additional memory to the system cache.
[quote="AKA-Mythos"]Nice idea! I tried that for a specific USB-Stick and it immediately fixed the problem.
Unfortunately this isn't a working solution for us, since we are using Directory Opus in our team and copying large files to USB-Sticks is a daily use-case.
In other words, we have plenty of USB-Sticks that are used on plenty machines. Since the write-cache is a per-device setting it's impossible to do that for all permutations on all machines.[/quote]
Make a registry patch for all your dvices and load to all your computers via a management tool.
This is a well-meant suggestion, but it's nothing more than a hack. I'm pretty sure that our administrators will deposit my security badge on the roof of our division building.
Anyway, the problem does not depend on USB sticks exclusively. It always occurs if the destination's write performance is slower than the source's read performance.
In other words, I see the same problem when copying from one local partition (SSD) to an other local partition (HDD) in the same system. The harddisk can't follow the fast read performance of the SSD and thus the same problem appears here also. Simply the freeze duration at the end isn't as long as in the initial use-case since the harddisk can flush the buffers much quicker than an USB-Stick (USB 2.0).
Can somebody explain me, why writing data bypassing the system cache is such difficult? There are so many tools around in the world providing this mode for years without any flaws (TotalCommander, UBCopy, TerraCopy, Xcopy, etc.). At least for me it's quite obvious that writing a large amount of data without caching makes sense.
It's not that magic, but you have to do all the copy stuff yourself (buffer handling, data alignment, etc.) and it doesn't make sense to go this way without an actual need for it. But with growing system memory and great differences in device throughput I think the time has come to at least provide a linear progress dialog for copy operations.
Buffering or not isn't the question for me. I just want to have a reliable progress dialog showing me the actual progress in a way that I can unmount the USB-Stick immediately after the dialog will be closed. And certainly DOpus should be responsible all the time during the copy operation (and even after that) which isn't the case on my machine.
[quote="AKA-Mythos"]It's not that magic, but you have to do all the copy stuff yourself (buffer handling, data alignment, etc.) and it doesn't make sense to go this way without an actual need for it. But with growing system memory and great differences in device throughput I think the time has come to at least provide a linear progress dialog for copy operations.
Buffering or not isn't the question for me. I just want to have a reliable progress dialog showing me the actual progress in a way that I can unmount the USB-Stick immediately after the dialog will be closed. And certainly DOpus should be responsible all the time during the copy operation (and even after that) which isn't the case on my machine.[/quote]
One minute this thread is about freezing (or various other issues, some that evaporated I think), the next about how much the progress dialogs reflect what's been written to disk (rather than cache), which is why I have been getting confused (as per our PMs). It's much easier to discuss things if we stick to a single issue instead of jumping around, otherwise we're all going to be talking cross-purposes.
As mentioned in my earlier private message, we will be looking into buffering/flushing changes to see if we can improve things here. I don't think any further discussion is needed right now.
Sorry for that, this issue (as shown in my video clip) has two symptoms which both are in relation to the write cache that stops the world while flushing to the target device.
Symptoms:
wrong progress dialog (1000 MB/s and quick disappear)
DOpus user interface freeze after dialog disappears
That are the three symptoms have been discussed in this thread. However, we should really wait for the feedback of the analysis.
@Leo: Can you reproduce this problem on your own machine? Have you tried to copy a 2 GB file from HDD to a slow USB 2.0 device? Thanks in advance.
No, I've never seen anything like that happen with my hardware.
But it doesn't really matter as it isn't me who is looking into this.
As mentioned, we'll look into this to see if we can make some improvements and work-around any unwanted behaviour of the OS/filesystem without introducing additional problems.
I have 3 different sys, there's no difference between 16GB on x64 or 4GB on x32 or from SSD or SATA - some USB-devices (not randomly!) freezes the sys when copying, fortunately only devices from a friend (the cheaper ones, w/o to say that they're all bad)!
I have only seen this problem on Win7 64 Bit having 16 GB installed and only a few applications opened (provide as much memory to the system cache). But on our machines I even can reproduce it when copying from SSD to HDD (no USB device involved at all).
I would appreciate, if you can perform the following test on your system having 16 GB memory.
End all applications except DOpus (64 bit version).
Connect an USB-Stick using USB 2.0 (please do not use USB 3.0).
Copy a 1,5 GB file from SSD to USB.
I'm really curious if you get a linear progress dialog without any freezes and if DOpus stays responsible (during and after the copy).
With this scenario I can reproduce the problem on all our workstations.
