10Gb network transfers

I haven't checked using a debugger, but I expect it does.

There's also a NirSoft command-line tool which can be used to copy files using the API, which may be a bit easier to integrate into an Opus button, if that's the plan. (Although I am not sure if it can move files as well as copy them, from memory.)

[Edit] My post about using it in Opus here: Copy files via the shell (Windows Explorer)

I don't know what Total Commander uses, but Explorer uses the high level API that I'm talking about (which Opus will be able to use in the future). That is the API that has been optimized for certain hardware/network issues, and which in turn a lot of hardware/drivers/etc. have been optimized around, while leaving the normal (and much more general) way to read and write data much slower, since almost nobody measures that and thus nobody notices it is slow and complains about it.

Measure how fast something like Photoshop can read or write a large file over the network and you'll probably find it's as bottlenecked as Opus on your setup. (I may be wrong, of course, or maybe Photoshop is a bad example. But I'd be amazed if you measured a lot of software and found that most wasn't affected, and you just hadn't realised it most software doesn't report transfer speed when reading/writing files.)

All Opus does is call CreateFile to open the files and then ReadFile and WriteFile in a loop to read and write the data, with a configurable buffer size (and an option to disable filesystem buffering and do the buffering itself, but that is off by default). If that is slow, it's going to be slow for almost everything.

You can play with the buffer-size and non-buffered-IO options to see if they help with your particular setup. They sometimes do.

In time, we'll add the ability to use the high level API to Opus, but it won't come any faster by bumping threads about it. That just makes it spend time replying instead of coding. :slight_smile:

[Edit 2]: A quick search of the Total Commander forums shows it is also using the same high level API, and, now I think of it, I saw a post from the TC devs saying there's no way for anyone else to replicate the speed it gets with some hardware as it's completely arcane and Windows is just broken in this regard, so the option is to either use the API or be slower on some setups. They added an option to use the API, which is what we will do as well, but it still means anything not using the API is going to be very slow on the same setups.

The API is only good for copying simple files from A to B, not reading or writing data into memory or archive creation or extraction, so it's limited, and sad that Windows only optimized around that instead of the more general case and fixing the lower-level APIs like they should have.

1 Like