Hello,
I saw the same issues in different other posts but no real solution available.
If I copy a file (4GB) in Windows explorer over a gigabit link (SSD on both networked computers), the speed it around 100MB/s which is about the maximum a gigabit link can offer.
If I copy the same file in Dopus, I get half of the speed (50MB/s).
I played with the copy_buffer_size, which is 64 KB by default, if changed to 1024 KB it helps a lot.
I also have the setting copy_nonbufferio_threshold set to 1MB, but I don't think it changes anything in my case.
As it seems to be the only workaround, is there any downside to set the copy_buffer_size to 1024KB ?
how come Windows does better than Dopus by default ?
Is it a bug from Dopus which is going to be fixed ? (in which case, I will have to remember to set back copy_buffer_size to 64KB once it is fixed)
It'll only cause errors if there is a bug in the NAS or network drivers (or antivirus/firewall/etc).
There could be lots of reasons for the difference. At the end of the day, some hardware (and drivers, antiruvirus, firewalls, etc.) have problems with certain buffer sizes or when used in certain ways, and many of them are only properly tested/optimised with Explorer and might have problems with something that does things slightly differently.
Explorer can be slower than Opus in other cases, too.
The buffer size should not actually make any difference when copying to the network as the operating system and network do their own buffering on top of whatever the application uses but some things are just inefficient with certain buffer sizes. Opus lets you adjust the buffer size for when you run into such problems.
I tried at work, different Antivirus, different hardware, different switches (still Win7x64).
Copy speed in Windows explorer through the network is 80MB/s
copy speed in Dopus is 25MB/s with buffer size by default
and 56MB/s with buffer size set to 1024.
First tests: from a server to my computer
tried with a 1.68GB file, I have timed it:
tried dopus first: 48 sec (even with buffer size set to 1024)
then windows:22 sec
Then tried again with Dopus and it went down to 30 sec after some attempts, then after several attempts, the lower time I managed to get was 26 sec.
Windows was consistent with 22 sec.
tried with another file (1.84GB):
tried Windows first: 23 sec
then with Dopus: 35 sec
last test I did, I copied a file from my computer to a server, file was 4.07GB:
tried with Windows first: 39sec
then with dopus: 45sec
repeated the test several times and it was the same as above.
(see network graph attached to see the difference)
I'll see if I can reproduce that when I get a chance (sooner if you link your account).
In the past I've never been able to, or seen performance vary with different programs without any clear winner, but it's possible the recent non-buffered IO changes (which would affect the read end but not the write end when copying to a network, or vice versa when copying from the network) maybe have changed things.
What kind of server is at the other end in your tests?
I'm having the exact same problem. Copying from one Windows 7 Pro x64 machine to another on a gigabit network. I have hundreds of gigs spread over 7 folders; the files are large video files in 4GB chunks and only a handful of tiny files. Disks on both sides are RAID arrays capable of 300+MB/sec.
Right now I am copying one folder of 115GB via Explorer, and my network activity has been hammered at 98-99% the entire time, Explorer reporting ~112MB/sec.
The next folder of 175GB (all similar 4GB files) I am copying via Opus 10.0.4.0 and it fluctuates all over the place, on average between 45-55% network utilization, an average of about 58MB/sec after running for 10+ minutes.
Pretty significant difference, and I was kind of surprised.
After digging around in the preferences a bit, I found the Advanced property copy_buffer_size which was set to 64kb. Cranked up to 256kb, that improved my utilization to about a 74% average. 512kb got me up to 82% average. 2048kb yielded 90%, and that's about where I've plateaued... increasing to 4096kb still has me at 90%. A far cry from where I started with the default 64kb though.
So hopefully this helps others -- but it would still be nice to figure out if there's a way to optimize Opus' routines on par with Explorer's (a sentence I thought I'd never use in a thousand years! )
I just never got around to updating this particular machine before.
So I just did, but got pretty much identical results.. with the copy buffer size and new non-buffered IO settings at default I get poor performance. Copy buffer back up to 4096k gets me to 100MB/sec and then setting the non-buffer IO to 0 gets me to about 103MB/sec.
it is me again, I waited patiently and left you some time to work on this issue.
I tried again with DO 11 to see if things were getting better, but it seems to be still the same.
I repeated the test several times, so I give you the average times below.
For instance, if I copy from a network share (gigabit) to a SSD, it takes around 2 min using Windows Explorer.
Same copy in DO 11 takes always more than 2min 20s.
For my tests, I copied 2,82 GB (3 054 Fichiers, 374 Dossiers), with random files size (it has some big archives and small files also).
My DO copy buffer size is set to 4096.
Would it be possible to at least match the Windows default copy speed ?
If there's only a 20 second difference, and thousands of files/folders, the difference is probably not the copy speed. It's probably because Opus is configured to preserve more file metadata (attributes, permissions, descriptions etc.) than Explorer is preserving.
I've done a lot of testing of the network file copy speed in Opus and I got identical speeds with Opus and Explorer, when both were configured to do the same things. (The burst speeds varied, as did the overall times from copy to copy, but both programs had the same overall behaviour.)
This subject comes up every few months and it always turns out to be because the two programs were not doing the same things (or other testing errors, like not accounting for buffering caused by one program speeding up the other).
Six seconds is well within the amount of fluctuation that would be normal. I found both Opus and Explorer varied greatly with the time taken to copy files over a gigabit network between attempts, but both had similar overall behaviour. (The slowest Explorer ran was also the slowest Opus ever ran. Both matched for the fastest as well, and the averages were near close to identical over a series of tests.)
Sorry but I am not going to spend much time defending the copy speed in Opus unless you come up with some really good test results, with repeated tests to rule out fluctuations, accounting for filesystem buffering, with examination (e.g. using Process Monitor) that proves both programs are configured to copy the same metadata, and some theory as to why either program would consistently go faster than the other (given that all the real work is handed over to the low-level operating system anyway, and all Opus is doing is sitting in a loop calling ReadFile and WriteFile on separate threads as fast as the OS will process the data), also ruling out antivirus/firewall software treating different software differently.
I've already wasted days of my life on this subject and the result is always the same so I am setting a high bar for anyone who wants me to re-visit it, sorry.
PS: Another thing to account for is the difference in when the progress dialogs claim the operation is complete vs when it is actually complete. In one case the copy may be called complete when the last data has been passed to the OS buffers but not yet sent over the network and written to disk. In the other, it may wait until it is really finished.
Measuring copy time fairly is extremely complicated and, trust me, I've done it lots of times and just about every program copies data at exactly the same speed because at the heart of any file copy it's the exact same OS/filesystem/network code doing the exact same thing, unless there's some pathological issue (e.g. some crappy USB drives have awful performance with certain buffer sizes; antivirus can do weird things; and so on), or unless the tests are unfair (buffering, metadata, not enough tests, etc. etc.)
I tried several timesin different orders, and I didn't get a lot of fluctuation. DO was behind Windows.
For instance, I tried to copy in this order:
DO 1:22 (1min 22)
W 0:55
DO 1:16
DO 1:01
DO 1:06
DO 1:05
W 1:06
W 0:57
W 0:55
DO 1:11
DO 1:13
W 0:56
I can see that sometimes Dopus best copy times is close to the average Windows copy times, but in any cases, the best DO copy time was 1m1s compared to the best Windows copy time of 55s.
There is a difference
I did a screenshot of the network traffic while doing 2 copies in Windows followed by 2 copies in Dopus.
for my test, I used the 2 Win8 ISO files as previously, but this time I did a copy between 2 fast computers equipped with SSD (Win 7 SP1 x64 on the 2 computers).
)
