I built a brand new server over the holidays. I use Drivepool to pool disks together with two SSD drives as "feeder drives" for the pool. These disks receive the files first. I copy or move files from my Laptop (SSD) to the server via Gbit Network. The server has two 1 Gbit Network connections teamed up.
Copying or or moving files via Opus: ~66 MB/sec.
Copying or moving files via Windows Explorer: 111MB/sec
What is going on here? Why is Opus so slow? I found some older threads but can't find a solution.
Is it slow in other tools as well? You can use the Network tab in Task Manager to get an idea of speeds in different apps (as long as nothing else is using the network much at the same time).
That suggests the server (or something else in between, e.g. antivirus/firewall/network) has been optimised (to some degree) for the exact way the Windows shell writes data but is pathologically slow when anything else writes data to it in a slightly different way.
If Preferences / Miscellaneous / Advanced [Filesystem]: copy_nonbufferio_threshold is set to anything other than 0, try setting it to 0 and see if it makes a difference.
I have already tried copy_nonbufferio_threshold and also copy_buffer_size.
The speed even gets worse. I did a round of tests. Copied a 2,35 GB MP4 file to and from the server.
Copy from server
Dopus = 51 seconds
Explorer = 38 seconds
Copy to server
Dopus = 36 seconds
Explorer = 21 seconds
I don't have any extra firewall. It's simply a Ethernet connection to a switch. Only the Windows firewalls are active.
We aren't experts on DrivePool but there are lots of threads about problems with it being slow with suggestions of things to try, from a quick search. I don't know if they will be relevant but it's where I would look, since the issue isn't limited to Opus.
Something else I'd try is closing all windows except the copy dialog, in case having the destination folder open while copying is causing Opus to read data back from the server, which may slow things down if the server isn't good at doing two things at once, especially if itnis sending a lot of change notifications which would trigger Opus into re-reading data during the copy.
I tested it now via a normal drive, because I have a partition of a Samsung NVMe in the server. And this one is capable of 3 GB/sec no problem.
Same effect. Dopus is consistently ~20-30 MB slower than Windows Explorer. Even if I close the tab of the copy destination.
How does Dopus act differently than the Windows built in function? I guess it simply calls a function in Windows?
You're taking about Opus doing something special to run slower with this device at the other end, but we're probably really talking about everything except the Windows shell file copy API.
Opus doesn't do anything that special, at least that the device at the other end of the network should care about. It reads from the source file into a buffer and writes that buffer to the destination file. If non-buffered IO is enabled then separate threads are used for the read and write ends (and if it isn't, then the OS itself should do something similar to cache ahead of the read position).
Some devices are only tuned to the exact nuances of how the shell file copy API works (e.g. particular buffer sizes, maybe, although I can only guess) and run slowly when everything else sends data to them.
Such devices will be slow with everything else. That includes e.g. saving files to the network share using software, like saving an image in Photoshop or a video in Premier.
At least based on the info we know so far, the question is why is the server slow when anything but Explorer (or other things using the shell filecopy API, which is very limited but also can be faster with some devices that aren't tested properly with anything else). From what you've said, Opus isn't a required factor in seeing slow transfers to the device, since you saw them with another program as well. I'm betting you'll see them with most programs.
(It's also worth noting the usual things about measuring file copy performance: Buffering can impact things greatly if doing the same copy more than once. Antivirus can slow down one program or method of copying while ignoring another. Explorer's progress dialog says it is complete abd closes a long time before it really is complete, while data is still in write buffers. Different programs calculate speed differently. Buffer sizes can also affect some network (and USB, as an aside) devices very differently. Those things may not explain the whole difference between Explorer and the other two programs you've tested, but may account for some of it. Opus being configured in a way that causes it to read data from the share or access it in other ways at the same time it is being witten is abother potential factor; Process Montior could reveal that kind of thing.)
Hi Leo,
thanks. This thread (Copy files over network - Dopus slower than Windows Explorer) also led me to investigate deeper.
It looks like since I have two network cards in the server I had some TCP congestion issue. Only one card active now and it's much better. However, I performed some tests and captured everything via stopwatch and Process Monitor while copying with Dopus, Total Commander and Windows Explorer.
Windows Explorer performs best of all the three. I can send you logs of you want.
It's a 475 MB compressed ZIP file and I spent some hours to collect all the info. Would appreciate if you look into this further.
What happens if you use Total Commander's "Big File copy mode"? Are we comparing the shell file copy speed vs everything else again, or more than two things to see if the other end is sensitive to the way the data is sent in some way?
The buffer size making such a difference suggests that trying more buffer sizes may find a sweet spot for the network or device at the other end. Process Monitor can show the buffer size Explorer (or TC) is using when copying to the network device, which is worth a try in Opus.
Have you tried with a standard Windows server at the other end to compare that? That is what I use, and the link gets saturated no matter which software is sending the data, as it should.
I only see slowness copies when doing it over wireless (which is slow anyway). When I'm hard wired to ethernet I don't see slowness issues. For instance last night I copied 13 GB of MP4's to my NAS, Wireless took about 14 minutes, hardwired to ethernet took about 1.5 minutes. That's using 5400 RPM (5900 - being helium filled WD Red 8TB NAS drives) and 5400 RPM drive in my laptop. 1.5 minutes for 13 GB isn't bad in my eyes. I'm using DirOpus 11. Maybe Opus 12 is slower than 11?
Horrible. Even slower than Dopus. Even if I set the buffer to something like 10240 k for different disks. Total Commander is as fast as the Windows Explorer (both directions) if I simply use the "Standard Copy method".
The most simple tests is copying a Windows 10 ISO file. It's totally consistent. I copy with Dopus TO the server and then FROM the server.
I have also tried this tip with the button to use nircmd.
This gives me full speed.
Reading this, I have now put in some ridiculous number in copy_buffer_size: 512 MB
And voila, I see much better speeds. But also strange fluctuations.
OK, from the speed perspective, I think I reached a better speed with that ridiculous buffer size of 512 MB. I still think it's a bug in Dopus that needs to be investigated and fixed once and for all.
If you can write me how to configure Process Monitor I can help hunt down the root cause.
Same issue for me although shares sit on a Linux FS. Opus is consistently 50% slower in xfer to the shares. Oddly, much faster from the share in comparison, bizarre!
