Copy files over network - Dopus slower than Windows Explorer

Hi Leo,

any update ? do you need me to try anything else on my side ?

No, not yet. I'll update the thread when there's anything new.

I'm having the exact same problem. Copying from one Windows 7 Pro x64 machine to another on a gigabit network. I have hundreds of gigs spread over 7 folders; the files are large video files in 4GB chunks and only a handful of tiny files. Disks on both sides are RAID arrays capable of 300+MB/sec.

Right now I am copying one folder of 115GB via Explorer, and my network activity has been hammered at 98-99% the entire time, Explorer reporting ~112MB/sec.

The next folder of 175GB (all similar 4GB files) I am copying via Opus 10.0.4.0 and it fluctuates all over the place, on average between 45-55% network utilization, an average of about 58MB/sec after running for 10+ minutes.

Pretty significant difference, and I was kind of surprised.

After digging around in the preferences a bit, I found the Advanced property copy_buffer_size which was set to 64kb. Cranked up to 256kb, that improved my utilization to about a 74% average. 512kb got me up to 82% average. 2048kb yielded 90%, and that's about where I've plateaued... increasing to 4096kb still has me at 90%. A far cry from where I started with the default 64kb though.

So hopefully this helps others -- but it would still be nice to figure out if there's a way to optimize Opus' routines on par with Explorer's (a sentence I thought I'd never use in a thousand years! :laughing: )

Why are you using such an old version of Opus?

The file copy code has been changed since then, for starters.

I just never got around to updating this particular machine before.

So I just did, but got pretty much identical results.. with the copy buffer size and new non-buffered IO settings at default I get poor performance. Copy buffer back up to 4096k gets me to 100MB/sec and then setting the non-buffer IO to 0 gets me to about 103MB/sec.

Hello,

it is me again, I waited patiently and left you some time to work on this issue.
I tried again with DO 11 to see if things were getting better, but it seems to be still the same.

I repeated the test several times, so I give you the average times below.
For instance, if I copy from a network share (gigabit) to a SSD, it takes around 2 min using Windows Explorer.
Same copy in DO 11 takes always more than 2min 20s.

For my tests, I copied 2,82 GB (3 054 Fichiers, 374 Dossiers), with random files size (it has some big archives and small files also).

My DO copy buffer size is set to 4096.

Would it be possible to at least match the Windows default copy speed ?

Thanks guys

If there's only a 20 second difference, and thousands of files/folders, the difference is probably not the copy speed. It's probably because Opus is configured to preserve more file metadata (attributes, permissions, descriptions etc.) than Explorer is preserving.

I've done a lot of testing of the network file copy speed in Opus and I got identical speeds with Opus and Explorer, when both were configured to do the same things. (The burst speeds varied, as did the overall times from copy to copy, but both programs had the same overall behaviour.)

This subject comes up every few months and it always turns out to be because the two programs were not doing the same things (or other testing errors, like not accounting for buffering caused by one program speeding up the other).

I tried with 2 Win8 ISO (3.21GB+2.32GB),
Windows: 1min
DO 11: 1min 6s

How many times did you try?

Six seconds is well within the amount of fluctuation that would be normal. I found both Opus and Explorer varied greatly with the time taken to copy files over a gigabit network between attempts, but both had similar overall behaviour. (The slowest Explorer ran was also the slowest Opus ever ran. Both matched for the fastest as well, and the averages were near close to identical over a series of tests.)

Sorry but I am not going to spend much time defending the copy speed in Opus unless you come up with some really good test results, with repeated tests to rule out fluctuations, accounting for filesystem buffering, with examination (e.g. using Process Monitor) that proves both programs are configured to copy the same metadata, and some theory as to why either program would consistently go faster than the other (given that all the real work is handed over to the low-level operating system anyway, and all Opus is doing is sitting in a loop calling ReadFile and WriteFile on separate threads as fast as the OS will process the data), also ruling out antivirus/firewall software treating different software differently.

I've already wasted days of my life on this subject and the result is always the same so I am setting a high bar for anyone who wants me to re-visit it, sorry.

PS: Another thing to account for is the difference in when the progress dialogs claim the operation is complete vs when it is actually complete. In one case the copy may be called complete when the last data has been passed to the OS buffers but not yet sent over the network and written to disk. In the other, it may wait until it is really finished.

Measuring copy time fairly is extremely complicated and, trust me, I've done it lots of times and just about every program copies data at exactly the same speed because at the heart of any file copy it's the exact same OS/filesystem/network code doing the exact same thing, unless there's some pathological issue (e.g. some crappy USB drives have awful performance with certain buffer sizes; antivirus can do weird things; and so on), or unless the tests are unfair (buffering, metadata, not enough tests, etc. etc.)

I tried several timesin different orders, and I didn't get a lot of fluctuation. DO was behind Windows.

For instance, I tried to copy in this order:
DO 1:22 (1min 22)
W 0:55
DO 1:16
DO 1:01
DO 1:06
DO 1:05
W 1:06
W 0:57
W 0:55
DO 1:11
DO 1:13
W 0:56

I can see that sometimes Dopus best copy times is close to the average Windows copy times, but in any cases, the best DO copy time was 1m1s compared to the best Windows copy time of 55s.
There is a difference :frowning:

I did a screenshot of the network traffic while doing 2 copies in Windows followed by 2 copies in Dopus.
for my test, I used the 2 Win8 ISO files as previously, but this time I did a copy between 2 fast computers equipped with SSD (Win 7 SP1 x64 on the 2 computers).

I'm happy to do more test if it can help.


interesting...

copy from a remote Win7 with HDD to my workstation Win 7 with SSD, same VLAN, same switch:

see attachment:


This still isn't accounting for all the different things I mentioned, and those graphs don't tell us much without knowing the type of data being copied and all the other details I asked for.

As I said, I have wasted days of my life on this kind of thread already and I'm not going to again unless everything I mentioned is covered.

Sorry I don't get what you are missing...

you can clearly see on the graph above that copying the same files in Windows is faster than in Dopus (higher transfer rate and shorter time to complete).
I still use my 2 big ISO files like I said before: 2 Win8 ISO (3.21GB+2.32GB).

I'm showing you network transfer, so it should address your remark: "Another thing to account for is the difference in when the progress dialogs claim the operation is complete vs when it is actually complete".

What would you need more ?

and by the way, it is just 2 files.. so I don't get how metadata, NTFS persmissions or anything else could add so much difference regarding the transfer rate.

Try changing the buffer size.

I have it set at 4096, what would you advise?

Try using Process Monitor (one of the things I mentioned...) to see which buffer size Explorer is using, and try similar values. (Also half or double the value may be worth a try, as well as a few arbitrary sizes like 1MB, 5MB and smaller ones like 32KB, 512KB... in case some component in the chain is sensitive to the buffer sizes used)

How do you find what buffer size Explorer is using in Process Monitor?

Thanks in-advance :thumbsup:

Look at the length of reads and writes in the Detail column of Process Monitor.

You'll probably want to disable non-buffered I/O in Opus as well, else the copy buffer size doesn't have much effect (as there's another buffer everything gets written into before asking Windows to write the data).