Hi
Is there any way I can improve the copy speed of Opus when copying larger 4GB+ files from my Windows 10 VM to a Linux SMB share?
VM to Share copies at 350MB/s
Share to VM (different file) copies at 700MB/s
I'm not sure why share to VM is twice the speed?
Total commander copies at 700MB/s both directions so it's not the network.
Any tips?
Try increasing the copy buffer size under Preferences / Miscellaneous / Advanced, then restarting the copy.
Tried that, it helps by increasing copy to share speed by 25% approx.I tried in 10MB increments up to around 60MB. However, it impacts share to VM copy speed by reducing it almost by 50%. So not a good solution.
Other filemanager such as TC and Yxplorer have full speed in both directions so not sure what Opus is doing different.
A lot of other file managers just use the shell file copy routines, which aren't very flexible (e.g. you can't rename-and-copy at the same time) but are what some devices are optimised for in terms of exact buffer sizes and other nuances.
You can set up a button in Opus to copy things in the same way if it helps:
(I'd also be weary of comparing file speeds or times based on progress dialogs. For example, Explorer closes the dialog and appears done when the data is really only buffered and still being sent. Opus and Explorer measure speeds differently as well. The best way to compare is to open Task Manager's network graph and compare how long data is being sent for based on wall time. Caching and other issues also come into play, and if you are copying lots of small files then the Copy Attributes settings in Preferences can have a huge impact, but are less important if we're talking about a handful of huge files.)
It's just very frustrating that the built in opus copy routine is not as fast as competitor fm's like TC and Yxplorer.
I just cant for the life of me understand why a copy of a 4GB file to an SMB share is one speed yet the reverse copy is another speed. How do you explain that? Like I said, those other fm's dont have this speed issue so what is Opus doing different?
Just so you know the copy between W10 VM and SMB share is nvme drives rated at 2GBps. My virtual NIC between them is set at 10Gbps.
What could be a valid reason for the copy to the SMB share being 50% slower than copy from the SMB share?
XYPlorer just uses the Windows shell copy function as far as I know. Not sure about TC but it may as well.
Which is why explorer copies to and from the SMB share at over 700MB/s.
It is only Opus that for some reason only copies at around 350MB/s to the share and 700MB/s from the share.
What are the possible reasons that Opus copy handler is behaving in this way?
As Leo said, one possible reason is that the SMB stack on the server is only optimised to work with how Explorer copies files.
OK, that's interesting. Does this imply that the issue with Opus and speed is limited to SMB shares only? If the share was on Windows and not Linux I wouldn't have this issue? I actually have no way of testing this as I dont have another drive capable of exceeding 350MB/s to hand.
Possibly, yes, but we can only guess as we don't have the same setup.
I set up a win server 2016 vm with file shares on the nvme drive with the samba shares on it.
I performed the same copy test (windows 10 VM to windows server VM share). Opus copies at about 350Mb/s - total commander copies at 900MB/s. The issue does not appear to be restricted to Linux SAMBA but Windows Server SAMBA also.
Please use Task Manager, and either the Network or Disk graphs (whichever best corresponds to the activity in question in your setup) to time how long the operation really takes.
Always measure the time it takes for the activity to stop in Task Manager, rather than when the progress dialog closes, or speeds reported in the dialogs.
Tests also need to be done a few times to rule out caching and background tasks.
I don't have a virtual machine on an NVMe drive, and my LAN is only 1gbit, so I can't test at the same speeds you're using, but a test to a VM using a fast SSD shows both Opus and Explorer take exactly the same time (1 minute) to copy a 12GB file to a Windows share in a VM.
That was using my copy buffer settings:
Those are not the default settings; I've enabled non-buffered I/O, which can improve things but can also create compatibility issues with some devices/filesystems, which is why it is not enabled by default.
I'd also try setting copy_buffer_size to values like 64 KB and 5 MB to see if larger or smaller buffers are preferred on your setup. (You can also use Process Monitor on Explorer to see which buffer size it is using to copy to the destination, to try the same size in Opus.)
(When copying over my LAN rather than to a VM, everything I've tried runs at the same speed and is bottlenecked by the 1gbit network, give or take the usual randomness of disk and network speeds.)
How do I get the buffer size for explorer in Process Monitor. I filtered by explorer.exe, started the copy but I can see no reference to buffer in the process monitor display log.
I think some further iternal testing needs to be perfomed by Opus engineers.
It's no longer bleeding edge to have nvme and 10gb between machines and vm's now.
I would like to see how you get on with a set up similar to mine.
As for the tests:
Opus
1.) VM to share 12GB file 36s peak 370MB/s
Explorer
1.) VM to share 12GB file 18s peak 780MB/s
Opus is almost exactly half the speed.
Changing the buffer size in 10MB increments adds a perfomance boost of approx 10-15% peaking at around 450MB/s so still some way off explorer.
@mikeyo I reported a similar effect a while back. There is unfortunately no interest to investigate and fix that issue. Cranking the buffer up to 512 MB somehow made it better. Try setting it much higher. I'm interested how this will affect the speed on a 10G network because I plan to upgrade.
See my thread.
The Details column on the right should show how large each read or write operation is in bytes.
I suspected as much, shame really. Opus excels in all aspects except its own copy handler. I scouted the forums extensively and came across your issue and others, makes for disappointing reading and apparent lack of concern over the issue.
I even set up a new vanilla w10 vm, fresh install of Opus 12 and repeated the same tests to rule out a config/env issue but alas, same results.
What frustrates me is the simplicity of the problem. I mean, copying a large file over a 10gb network to a SAMBA share sitting on nvme drives, a common setup no?
I pay my yearly sub and have done for many years, big fan of Opus since Amiga days so hence the frustration.
Happy to be a part of further testing to help narrow down the potential issue if the Dopus dev team are keen.
Have you tried putting in 512 MB in copy_buffer_size?
And leave copy_nonbufferio_threshold disabled.
Does it change something? I only have Gbit but this helped.
It's not that we aren't interested in this issue.
It's that almost every time comes up, with a handful of exceptions, it has ended up coming down to something about how the machines/network are configured that is sensitive to buffer size, or some other thing that's outside our control, impossible for us to reproduce locally, and sometimes even favors Opus over Explorer when the wind blows in another direction.
That's when it isn't just the two programs being configured to do different things. (e.g. If you're copying a lot of small files and Opus is set to preserve dates or other metadata but Explorer isn't, then it has a huge impact, and few people think of such things.)
There have been cases where a real issue has been found and we've been able to improve copy speed. One example was with the progress dialogs being updated too often, in a way that was stalling the read/write threads from continuing when using very fast media, and also causing high CPU usage during copies. So there certainly are places where improvements can be made, and we're more than happy to make them, but we're also weary from many years of threads like this where it has turned out to be factors outside our control.
We're looking into getting some 10gig network hardware but it's literally 10x the price of 1gig (and that's for the low-end stuff that's limited to connecting only two rooms/machines). It's not mainstream at all, which is a shame as 1gig is not fast enough even for HDDs these days. (If 10gig was mainstream, I'd have my entire setup on it already.)
In the meantime, I did a test using two RAM disks and localhost networking.
RAM disks are even faster than SSD, obviously, so that should tell us something about the maximum throughput Opus can reach.
localhost networking won't simulate all the network hardware between machines, of course, but it means the data is at least going through the network stack in Windows, even if it gets to take a huge shortcut without hitting a real network. Best we can really do in short notice without 10gig hardware.
Opus isn't involved in sending network packets or low-level filesystem access anyway, so if there is any difference between copying to a share on localhost and copying to a real network share, then any issues introduced there suggest the operating system or network hardware are not correctly buffering the data for smooth transmission.
If there is a way we can modify our side to make it run more smoothly, we're open to that, even if the problem really is that the OS is failing to do its side properly, but it's difficult to guess what that might be, especially without being able to reproduce the problem locally and play with different buffer sizes and so on.
The performance-critical file copy code is very simple. The code that happens before and after it to decide what to copy where and with which name & metadata, as well as error handling, streaming from archives, etc., is complex, but the main copy loop is very simple:
We call ReadFile and WriteFile in a loop, and send updates to the progress dialog every so often.
If non-buffered I/O is turned off (the default), then both calls are done on the same thread, but the OS is told to perform readahead buffering. If the OS is doing its job, the reads will be buffered in parallel to the writes.
If non-buffered I/O is turned on, we tell the OS not to do its own buffering on the read side, and do it ourselves. A separate thread calls ReadFile in parallel to the main thread calling WriteFile.
Note that non-buffered I/O is never used for the network share side of things, even when enabled in Preferences, so in today's test it isn't relevant to the write side of things, since we're copying to a network share.
I set up two 10GB RAM drives, V: and W:, located in physical memory (not the page file) using ImDisk:
A network share pointing to the W: disk was created, and I then copied a ~10GB file from the V: drive to the network share, using Opus, then Explorer, then Opus, then Explorer again. (Since everything is in RAM, caching should not play much of a part, unlike normal tests, so the order of the tests shouldn't matter too much.)
Here's a video showing what happened:
If anything, I'd say Opus is faster than Explorer in this case, just based on the progress dialogs. The two are close enough that I'd just call it even. (Unfortunately, Task Manager doesn't show performance graphs for this kind of disk, so I had to go by the progress dialogs, but those usually favor Explorer and disadvantage Opus, due to Explorer closing the dialog before buffers have been fully flushed, so if Opus looks faster/equivalent here, I'd say it is.)
We're seeing copy speeds of around 2GB/s (2 gigabytes a second) to a (localhost) network share there, copying the ~10GB file in about 3 seconds (or about 4-5 seconds in Explorer's case).
So whatever else is going on, the copy code in Opus is not limited to 350MB/s, we can definitely say that.
Thanks for the through explanation in the copy routine. I think the best way to test is a 10Gb card or create a vmware or linux kvm virtual machine x 2 and set the virtual NICs to 10Gb that way you dont need to buy a 10Gb card. For info, my setup is this...
HOST: UNRAID 6.6.6 (Linux KVM) - 2950x, 32GB RAM
MTU: 1500 HOST and VMs
Virtual NIC: 10Gb static IP4
DRIVES:
1 x 860 EVO 2TB (VM images)
1 x 960 EVo 500GB (SAMBA Shares)
1 x Corsair MP510 nvme (passed through direct to Windows 10 VM)
I would like to see your results for copying a 4GB+ file from a W10 VM to a network share on another VM or RAM disk across the 10Gb network.
When I run iperf on the host and vms I can saturate the 10Gb link so my issue is not network related. Also changing MTU to 9000 jumbo frames on affected interfaces makes no difference in my case.
If you can simulate copying a large file over a 10Gb virtual or real network between two VMs or PCs both using RAM disks and get expected results, then I would say something funky is going on with my set up and more probably related to how the SAMBA shares are set up. I also ran speed test on all my drives and get expected performance stats.
