The Windows Shellcopy Mystery

So I have a new NAS and it has two physical drives (actually 4, but it is RAID-2). I am baffled about copy speeds, especially Windows Shellcopy (via nircmd) from one physical drive to the other.

Due to the fact that the NAS is behind a pretty slow Powerline based connection, when I use DOPUS I get around 20 MB/sec, which is the expected speed of the network as DOPUS really reads from drive A and writes to drive B.

When I use the filemanager on the NAS I can get speeds up to 250 MB/sec from one drive to the other. Nothing travels over the network and due to the RAID system I do not get maximum speeds, but again that is the expected speed.

Now Windows Shellcopy (via Explorer and / or NIRCMD) has me confused. It reaches roughly 60 to 80 MB/sec, something smack in the middle. It is too fast to go over the network, it is too slow to run just internally.

Has anybody ever found out what Windows is doing here? (Drive access is via Samba V3 by the way).

Thanks, Boris

The speeds could be misreported, which is sometimes the simple explanation.

It’s also possible that the way the NAS implements remote copy offloading is slow. Or that the way Windows does the offloading is subject to network latency, since the control/commands still go over the network even if the data doesn’t.

Nah, I checked the speeds by trying same sized (but different) files multiple times, it is 4 times faster than Dopus and slower than internal copy. As it is NAS to NAS and GB-sized files I can count out caching too. And if it really is control/commands it deepens the mystery why there is a COPY but not a MOVE implentation that could be instantaneous (if source and destination are on the same physical drive on the NAS). Large moves I need to do in the NAS file manager.

Are you sure the internal copy isn't using a large writeback buffer when the NAS does the copy by itself?

A simple rename operation would do that if the files are on the same partition (not just physical drive).

You are very right but I have several "shares" which act like partitions to the outside world, but theoretically you can issue server-side commands which Windows 8.1+ seems to do. Dopus unfortunately does not see that.
Well, if you use Windows Fileexplorer to move files between these shares, it does not issue a rename but a copy and delete and uses the faster copy function. If I do that on the internal file manager it can move by rename, because it is the same drive and partition...
It clearly is a half baked implementation in the Windows Samba client itself.
Hopefully Dopus can "see" Samba drives in the future and try server side APIs.

Writeback buffer: Theoretically possible but on large transfers of several gigs the write speed would not be "flat" but have a spike up until the buffer is filled and then flatten out. Can't see this behaviour. There are odd fluctuations but overall after several minutes the write speed is roughly the same as in the first seconds. So I would rule that one out.

We'll be adding the ability to use the shell file copy API in the future, which will make Opus work the same as File Explorer and NirCmd in that regard.

I don't think it will ever work across different shares, though. AFAIK that isn't supported by the SMB protocol (although I may be wrong).

Just a guess, if things transfer 4 times faster than network speed, could it be the data is compressed on the fly? This would only work for data which is easily compressable, if you haven't yet, maybe try a huge zip file, which cannot be compressed any further. If this transfers at 4x as well.. I don't know! o)

That's not it. These are compressed video files.

So, I tried to understand via the documentation from Samba and Microsoft what is happening here. It looks like it is a client implementation that copies stuff in chunks, so Windows asks the NAS to copy a chunck of data, waits for this to finish, then asks for the next chunk copy, and so forth. For large files that slows down the whole process quite a bit. This also gives Windows (and Robocopy) constant feedback on the copy process so it is not a black box once you kickstarted a large file copy.

1 Like

So this is a local copy operation on the NAS, with some feedback loop to the client who started it?
I would expect this kind of "server side copy" to be much faster, nearly full speed of whats possible.
But who knows..

Call it a "hybrid" copy. Windows does not tell the NAS to copy over the full file. It rather tells the NAS to copy chunks of a file and waits for feedback every chunk. Can't determine which chuck size is actually used but I guess that value is not fixed but based on what the NAS tells Windows about its buffer sizes. You can dive into this here: [MS-SMB2]: SRV_COPYCHUNK_COPY | Microsoft Docs. Theoretically this all could be sped up further by running several copy chunk operations in parallel, but I would guess that there are a lot of good reasons why not to do that. Still, this all happens deep in the Samba client implentation of Windows itself and I can't find any tool that would use this directly. Developers just take the Samba client in Windows as is and never touch the protocol. But as long as DOpus in the future can fall back to a Windows copy rather than do the copy itself, we can get this performance gain.

One more or less final remark that leads to a suggestion:

ShellCopy can run several processes in parallel for speed gains. I have just copied four files in four processes and tripled the overall copy speed. Makes absolute sense that, while one process is waiting for a response from the Samba server that the chunk has been copied, another process can do communication on a different chunk (different file). With four copies running in parallel I got close to the same speed of doing the copy with the NAS-builtin File Manager.

@Leo If I may suggest that, if you implement Shellcopy, it would be able to take a list of files and run several processes (four looks good to me) at once automatically, spliting the file list into quarters? (I will try my hand in writing an Opus script to do that, because that would be pretty useful to me).

Cheers, Boris

Did you try Robocopy.exe yet? You mentioned it, but not sure you actually tried. It also runs multiple threads when copying, in theory it should copy as fast as can be.

Of course I tried Robocopy - it uses the very same functions. For integration into DOpus going via Explorer copy sounds better though.

Yes, I don't meant to integrate the robocopy executable of course, just for testing. It has a lot of options, like setting number of threads and how to handle date stamps and existing files and folders, I'm not sure it really uses the Explorer way of copying files, as you say, but it's always worth mentioning and trying around with it, which you did, so.. good! o)