interesting...
copy from a remote Win7 with HDD to my workstation Win 7 with SSD, same VLAN, same switch:
see attachment:
This still isn't accounting for all the different things I mentioned, and those graphs don't tell us much without knowing the type of data being copied and all the other details I asked for.
As I said, I have wasted days of my life on this kind of thread already and I'm not going to again unless everything I mentioned is covered.
Sorry I don't get what you are missing...
you can clearly see on the graph above that copying the same files in Windows is faster than in Dopus (higher transfer rate and shorter time to complete).
I still use my 2 big ISO files like I said before: 2 Win8 ISO (3.21GB+2.32GB).
I'm showing you network transfer, so it should address your remark: "Another thing to account for is the difference in when the progress dialogs claim the operation is complete vs when it is actually complete".
What would you need more ?
and by the way, it is just 2 files.. so I don't get how metadata, NTFS persmissions or anything else could add so much difference regarding the transfer rate.
Try changing the buffer size.
I have it set at 4096, what would you advise?
Try using Process Monitor (one of the things I mentioned...) to see which buffer size Explorer is using, and try similar values. (Also half or double the value may be worth a try, as well as a few arbitrary sizes like 1MB, 5MB and smaller ones like 32KB, 512KB... in case some component in the chain is sensitive to the buffer sizes used)
How do you find what buffer size Explorer is using in Process Monitor?
Thanks in-advance 
Look at the length of reads and writes in the Detail column of Process Monitor.
You'll probably want to disable non-buffered I/O in Opus as well, else the copy buffer size doesn't have much effect (as there's another buffer everything gets written into before asking Windows to write the data).
This has been a long pending issue in DOPus since 5 years ago (or longer) but without solution until this v11. You found it right that adjusting the DOPus buffer size could speed up from 50% till 80% of Windows copy speed over network; but that's it I still can't find a setting that can make it any closer.
Only TeraCopy integrated into DOPus can give you the same speed as using Windows copy over Network, and that is the only solution till now for those having slow NAS copy speed with DOPus. However the annoying part is the Teracopy icon won't grey out by itself when no file is selected; and mistakenly click the icon will cause error message and teracopy program to pop out or even crash occasionally. It cannot replace DOPus default copy/move function completely yet.
I have had 2 NAS from Synology and QNAP, 6 different PCs/Laptop from XP to Win7/8 and over 10 different routers/switches. Whatever combination all having same slow NAS copy problem with DOPus. I don't have the technical knowledge to tell where the root cause is, but with general knowledge and speed test/time taken test the difference is so obvious between Windows and DOPus file transfer speed over network. I came to know about Teracopy only 2 years ago and that become my copy handler inside DOPus for network file transfer.
If you feel that only 55MB/s vs 100MB/s with gigabit is slow; I experienced 200MB/s vs 400MB/s difference using 10GBE 2 years ago (now using laptop the 10GBE card is lying idle, waiting for the ret*rded Dell Precision series to implement Thunderbolt so that I can use it again)
Files that I transferred are big files (mainly movie file, from 50Mb up to 40GB per file)
Some people also report Teracopy being slower on their setups. The issue seems to be that some combinations of hardware, drivers, antivirus/firewalls, networks, etc. are very sensitive to minor details in how data is copied at a high level, even though the lower levels are supposed to abstract those details. This also means no method/configuration is likely to be fastest for all setups.
This is all on top of the many other factors, like whether or not timestamps and attributes are copied, and the effects of buffering.
Re greying out buttons for external tools when no files are selected, add @disablenosel to the button.
Woot~~ it works, thanks for that.
I want to point to a very comprehensive blog entry of Mark Russinovich who is explaining some of the secrets that were made in recent Windows versions to optimize file copy throughput.
http://blogs.technet.com/b/markrussinovich/archive/2008/02/04/2826167.aspx
Personally I was surprised that they decided to implement multiple read/write operations that are performed asynchronously in a sequence to improve triggering the read-ahead and write-behind algorithms of the underlying file system. In other words I would expect the common way of reading/writing the blocksize asynchronouly (by a common reader/writer pattern), but they decided to perform multiple block-read operations in a sequence (vice versa for write); e.g. read-64, read-64, read-64, write-64, write-64, write-64, read-64, read-64, read-64 [...]
The article mentions a lot of internal details about buffer sizes and copy strategies to max out the throughtput of network connections. As far as I understand the Windows file copy API uses different buffer sizes for different purposes (as you can easily see using ProcessMonitor as Leo mentioned).
Quote:
"File copying is not as easy as it might first appear, but the product team took feedback they got from Vista customers very seriously and spent hundreds of hours evaluating different approaches and tuning the final implementation to restore most copy scenarios to at least the performance of previous versions of Windows and drastically improve some key scenarios. The changes apply both to Explorer copies as well as to ones initiated by applications using the CopyFileEx API and you’ll see the biggest improvements over older versions of Windows when copying files on high-latency, high-bandwidth networks where the large I/Os, SMB2’s I/O pipelining, and Vista’s TCP/IP stack receive-window auto-tuning can literally deliver what would be a ten minute copy on Windows XP or Server 2003 in one minute. Pretty cool."
We're aware of that famous article from some years ago and have done some of the things in it already.
Is anyone talking about high-latency networks? People usually talk about LANs from memory.
The real takeaway from the article is that Windows allows minor differences in how files are read/written at a high level to have a huge effect on speed in certain situations, rather than making those APIs adapt as required automatically[1]. Thus Explorer (and CopyFileEx) are faster sometimes, but I've also measured Opus performing faster in some tests here. Part of the problem is there is no right answer, and a load of random things we can try will be good for some and bad for others, or may make no difference at all.
([1] For example, if multiple parallel small read requests make sense for a given driver, there's no reason Windows can't split a single large synchronous read into just that behind the scenes. We do some of that ourselves anyway, but it won't all be the same as Explorer as the exact details are undocumented and subject to change.)
But equally important is that the testing is done properly in the first place, with both programs only copying the same metadata as each other, effects of caching mitigated, antivirus/firewall turned off, and enough tests performed to account for speed fluctuations due to other activity. When I do such tests here, the speed is normally very close between any two programs. (That won't always be the case, though, as some setups have pathological cases if you use the wrong buffer size, etc. So I am not saying the speed will always be identical if the testing is done carefully, only that it will be in most cases, and has been in my own testing of various programs, not just Opus.)
Finally, it's worth noting that you can configure buttons to effectively copy files using CopyFileEx from Opus: Copy files via the shell (Windows Explorer). We're considering integrating such an option into Opus more directly, too, for people who want file copying identical to Explorer.
[quote="iieeann"]This has been a long pending issue in DOPus since 5 years ago (or longer) but without solution until this v11. You found it right that adjusting the DOPus buffer size could speed up from 50% till 80% of Windows copy speed over network; but that's it I still can't find a setting that can make it any closer.
Only TeraCopy integrated into DOPus can give you the same speed as using Windows copy over Network, and that is the only solution till now for those having slow NAS copy speed with DOPus. However the annoying part is the Teracopy icon won't grey out by itself when no file is selected; and mistakenly click the icon will cause error message and teracopy program to pop out or even crash occasionally. It cannot replace DOPus default copy/move function completely yet.
I have had 2 NAS from Synology and QNAP, 6 different PCs/Laptop from XP to Win7/8 and over 10 different routers/switches. Whatever combination all having same slow NAS copy problem with DOPus. I don't have the technical knowledge to tell where the root cause is, but with general knowledge and speed test/time taken test the difference is so obvious between Windows and DOPus file transfer speed over network. I came to know about Teracopy only 2 years ago and that become my copy handler inside DOPus for network file transfer.
If you feel that only 55MB/s vs 100MB/s with gigabit is slow; I experienced 200MB/s vs 400MB/s difference using 10GBE 2 years ago (now using laptop the 10GBE card is lying idle, waiting for the ret*rded Dell Precision series to implement Thunderbolt so that I can use it again)
Files that I transferred are big files (mainly movie file, from 50Mb up to 40GB per file)[/quote]
I agree 100% on that. When copying a 17.4GB file from my desktop to my Freenas server:
- 3mn with Dopus (copy buffer size tuned)
- 2mn30sec with explorer
But unfortunately leo keeps arguing that when you experience 200MB/sec in Dopus vs 400MB/s in explorer on a 40GB file, it's due to metadata/attributes being copied by dopus and not by explorer... Come on...
It's a fact that once tuned (copy buffer size=1MB), Dopus can reach 75 to 80% of the speed of both explorer and teracopy during network copy.
I don't say that it is a bug in dopus, it may be due to the NICs for example, but the fact is, that in the same conditions, explorer and teracopy perform better.