If there's only a 20 second difference, and thousands of files/folders, the difference is probably not the copy speed. It's probably because Opus is configured to preserve more file metadata (attributes, permissions, descriptions etc.) than Explorer is preserving.
I've done a lot of testing of the network file copy speed in Opus and I got identical speeds with Opus and Explorer, when both were configured to do the same things. (The burst speeds varied, as did the overall times from copy to copy, but both programs had the same overall behaviour.)
This subject comes up every few months and it always turns out to be because the two programs were not doing the same things (or other testing errors, like not accounting for buffering caused by one program speeding up the other).
Six seconds is well within the amount of fluctuation that would be normal. I found both Opus and Explorer varied greatly with the time taken to copy files over a gigabit network between attempts, but both had similar overall behaviour. (The slowest Explorer ran was also the slowest Opus ever ran. Both matched for the fastest as well, and the averages were near close to identical over a series of tests.)
Sorry but I am not going to spend much time defending the copy speed in Opus unless you come up with some really good test results, with repeated tests to rule out fluctuations, accounting for filesystem buffering, with examination (e.g. using Process Monitor) that proves both programs are configured to copy the same metadata, and some theory as to why either program would consistently go faster than the other (given that all the real work is handed over to the low-level operating system anyway, and all Opus is doing is sitting in a loop calling ReadFile and WriteFile on separate threads as fast as the OS will process the data), also ruling out antivirus/firewall software treating different software differently.
I've already wasted days of my life on this subject and the result is always the same so I am setting a high bar for anyone who wants me to re-visit it, sorry.
PS: Another thing to account for is the difference in when the progress dialogs claim the operation is complete vs when it is actually complete. In one case the copy may be called complete when the last data has been passed to the OS buffers but not yet sent over the network and written to disk. In the other, it may wait until it is really finished.
Measuring copy time fairly is extremely complicated and, trust me, I've done it lots of times and just about every program copies data at exactly the same speed because at the heart of any file copy it's the exact same OS/filesystem/network code doing the exact same thing, unless there's some pathological issue (e.g. some crappy USB drives have awful performance with certain buffer sizes; antivirus can do weird things; and so on), or unless the tests are unfair (buffering, metadata, not enough tests, etc. etc.)
I tried several timesin different orders, and I didn't get a lot of fluctuation. DO was behind Windows.
For instance, I tried to copy in this order:
DO 1:22 (1min 22)
W 0:55
DO 1:16
DO 1:01
DO 1:06
DO 1:05
W 1:06
W 0:57
W 0:55
DO 1:11
DO 1:13
W 0:56
I can see that sometimes Dopus best copy times is close to the average Windows copy times, but in any cases, the best DO copy time was 1m1s compared to the best Windows copy time of 55s.
There is a difference
I did a screenshot of the network traffic while doing 2 copies in Windows followed by 2 copies in Dopus.
for my test, I used the 2 Win8 ISO files as previously, but this time I did a copy between 2 fast computers equipped with SSD (Win 7 SP1 x64 on the 2 computers).
This still isn't accounting for all the different things I mentioned, and those graphs don't tell us much without knowing the type of data being copied and all the other details I asked for.
As I said, I have wasted days of my life on this kind of thread already and I'm not going to again unless everything I mentioned is covered.
you can clearly see on the graph above that copying the same files in Windows is faster than in Dopus (higher transfer rate and shorter time to complete).
I still use my 2 big ISO files like I said before: 2 Win8 ISO (3.21GB+2.32GB).
I'm showing you network transfer, so it should address your remark: "Another thing to account for is the difference in when the progress dialogs claim the operation is complete vs when it is actually complete".
and by the way, it is just 2 files.. so I don't get how metadata, NTFS persmissions or anything else could add so much difference regarding the transfer rate.
Try using Process Monitor (one of the things I mentioned...) to see which buffer size Explorer is using, and try similar values. (Also half or double the value may be worth a try, as well as a few arbitrary sizes like 1MB, 5MB and smaller ones like 32KB, 512KB... in case some component in the chain is sensitive to the buffer sizes used)
Look at the length of reads and writes in the Detail column of Process Monitor.
You'll probably want to disable non-buffered I/O in Opus as well, else the copy buffer size doesn't have much effect (as there's another buffer everything gets written into before asking Windows to write the data).
This has been a long pending issue in DOPus since 5 years ago (or longer) but without solution until this v11. You found it right that adjusting the DOPus buffer size could speed up from 50% till 80% of Windows copy speed over network; but that's it I still can't find a setting that can make it any closer.
Only TeraCopy integrated into DOPus can give you the same speed as using Windows copy over Network, and that is the only solution till now for those having slow NAS copy speed with DOPus. However the annoying part is the Teracopy icon won't grey out by itself when no file is selected; and mistakenly click the icon will cause error message and teracopy program to pop out or even crash occasionally. It cannot replace DOPus default copy/move function completely yet.
I have had 2 NAS from Synology and QNAP, 6 different PCs/Laptop from XP to Win7/8 and over 10 different routers/switches. Whatever combination all having same slow NAS copy problem with DOPus. I don't have the technical knowledge to tell where the root cause is, but with general knowledge and speed test/time taken test the difference is so obvious between Windows and DOPus file transfer speed over network. I came to know about Teracopy only 2 years ago and that become my copy handler inside DOPus for network file transfer.
If you feel that only 55MB/s vs 100MB/s with gigabit is slow; I experienced 200MB/s vs 400MB/s difference using 10GBE 2 years ago (now using laptop the 10GBE card is lying idle, waiting for the ret*rded Dell Precision series to implement Thunderbolt so that I can use it again)
Files that I transferred are big files (mainly movie file, from 50Mb up to 40GB per file)
Some people also report Teracopy being slower on their setups. The issue seems to be that some combinations of hardware, drivers, antivirus/firewalls, networks, etc. are very sensitive to minor details in how data is copied at a high level, even though the lower levels are supposed to abstract those details. This also means no method/configuration is likely to be fastest for all setups.
This is all on top of the many other factors, like whether or not timestamps and attributes are copied, and the effects of buffering.
Re greying out buttons for external tools when no files are selected, add @disablenosel to the button.
I want to point to a very comprehensive blog entry of Mark Russinovich who is explaining some of the secrets that were made in recent Windows versions to optimize file copy throughput.
Personally I was surprised that they decided to implement multiple read/write operations that are performed asynchronously in a sequence to improve triggering the read-ahead and write-behind algorithms of the underlying file system. In other words I would expect the common way of reading/writing the blocksize asynchronouly (by a common reader/writer pattern), but they decided to perform multiple block-read operations in a sequence (vice versa for write); e.g. read-64, read-64, read-64, write-64, write-64, write-64, read-64, read-64, read-64 [...]
The article mentions a lot of internal details about buffer sizes and copy strategies to max out the throughtput of network connections. As far as I understand the Windows file copy API uses different buffer sizes for different purposes (as you can easily see using ProcessMonitor as Leo mentioned).
Quote:
"File copying is not as easy as it might first appear, but the product team took feedback they got from Vista customers very seriously and spent hundreds of hours evaluating different approaches and tuning the final implementation to restore most copy scenarios to at least the performance of previous versions of Windows and drastically improve some key scenarios. The changes apply both to Explorer copies as well as to ones initiated by applications using the CopyFileEx API and you’ll see the biggest improvements over older versions of Windows when copying files on high-latency, high-bandwidth networks where the large I/Os, SMB2’s I/O pipelining, and Vista’s TCP/IP stack receive-window auto-tuning can literally deliver what would be a ten minute copy on Windows XP or Server 2003 in one minute. Pretty cool."
We're aware of that famous article from some years ago and have done some of the things in it already.
Is anyone talking about high-latency networks? People usually talk about LANs from memory.
The real takeaway from the article is that Windows allows minor differences in how files are read/written at a high level to have a huge effect on speed in certain situations, rather than making those APIs adapt as required automatically[1]. Thus Explorer (and CopyFileEx) are faster sometimes, but I've also measured Opus performing faster in some tests here. Part of the problem is there is no right answer, and a load of random things we can try will be good for some and bad for others, or may make no difference at all.
([1] For example, if multiple parallel small read requests make sense for a given driver, there's no reason Windows can't split a single large synchronous read into just that behind the scenes. We do some of that ourselves anyway, but it won't all be the same as Explorer as the exact details are undocumented and subject to change.)
But equally important is that the testing is done properly in the first place, with both programs only copying the same metadata as each other, effects of caching mitigated, antivirus/firewall turned off, and enough tests performed to account for speed fluctuations due to other activity. When I do such tests here, the speed is normally very close between any two programs. (That won't always be the case, though, as some setups have pathological cases if you use the wrong buffer size, etc. So I am not saying the speed will always be identical if the testing is done carefully, only that it will be in most cases, and has been in my own testing of various programs, not just Opus.)
Finally, it's worth noting that you can configure buttons to effectively copy files using CopyFileEx from Opus: Copy files via the shell (Windows Explorer). We're considering integrating such an option into Opus more directly, too, for people who want file copying identical to Explorer.
[quote="iieeann"]This has been a long pending issue in DOPus since 5 years ago (or longer) but without solution until this v11. You found it right that adjusting the DOPus buffer size could speed up from 50% till 80% of Windows copy speed over network; but that's it I still can't find a setting that can make it any closer.
Only TeraCopy integrated into DOPus can give you the same speed as using Windows copy over Network, and that is the only solution till now for those having slow NAS copy speed with DOPus. However the annoying part is the Teracopy icon won't grey out by itself when no file is selected; and mistakenly click the icon will cause error message and teracopy program to pop out or even crash occasionally. It cannot replace DOPus default copy/move function completely yet.
I have had 2 NAS from Synology and QNAP, 6 different PCs/Laptop from XP to Win7/8 and over 10 different routers/switches. Whatever combination all having same slow NAS copy problem with DOPus. I don't have the technical knowledge to tell where the root cause is, but with general knowledge and speed test/time taken test the difference is so obvious between Windows and DOPus file transfer speed over network. I came to know about Teracopy only 2 years ago and that become my copy handler inside DOPus for network file transfer.
If you feel that only 55MB/s vs 100MB/s with gigabit is slow; I experienced 200MB/s vs 400MB/s difference using 10GBE 2 years ago (now using laptop the 10GBE card is lying idle, waiting for the ret*rded Dell Precision series to implement Thunderbolt so that I can use it again)
Files that I transferred are big files (mainly movie file, from 50Mb up to 40GB per file)[/quote]
I agree 100% on that. When copying a 17.4GB file from my desktop to my Freenas server:
3mn with Dopus (copy buffer size tuned)
2mn30sec with explorer
But unfortunately leo keeps arguing that when you experience 200MB/sec in Dopus vs 400MB/s in explorer on a 40GB file, it's due to metadata/attributes being copied by dopus and not by explorer... Come on...
It's a fact that once tuned (copy buffer size=1MB), Dopus can reach 75 to 80% of the speed of both explorer and teracopy during network copy.
I don't say that it is a bug in dopus, it may be due to the NICs for example, but the fact is, that in the same conditions, explorer and teracopy perform better.
