File Move speed is slow

Opus copies the data as fast as the OS, hardware, drivers, antivirus, etc. allow it to.

If you think there are speed differences, please show the test and data, and tell us exactly which settings you have tried.

Speed differences usually turn out to be because one program was configured to do different things to another. (e.g. Explorer copies let metadata than Opus by default, so if you are copying/moving a lot of small files, there may be a big speed difference until you configure things differently.)

They can also come from antivirus treating different programs differently.

They also often come from filesystem caching, if you are running tests back to back and one program ends up reading data that is already cached in memory instead of reading it from disk.

They can come from buffer size and non-buffered I/O differences, too, which are largely dependent on hardware/driver/network quirks and need experimentation to work out the best sizes on some systems.

There are also major differences in how the progress dialogs in different software works. Opus won't say something is complete until it is flushed to the destination. Other things say the job is complete a long time before it actually is, and if you look in Task Manager you can see the data is actually still being copied.

It's a complex subject. Usually the speed is the same in every program you try. When it isn't, or doesn't appear to be, there are a lot of things that need to be looked into to understand why.

Leo,

I'm talking 40gb files here... there no chance of caching causing the HUGE discrepancy. It's not hardware, driver or network. Xplorer2 does not have this issue.

I already explained what my tests were, very large files being copied from one folder to another on the same iSCSI drive connection.

Considering that many others seem to be seeing this issue with OPUS I have to assume it's OPUS and certainly not my system. To be frank I'm not interested in messing around with settings (unless you can point ones out that can improve performance 10 fold) to figure this out. It should perform roughly equivalent to Explorer and other file managers (as I mentioned I tested xplorer2 to validate) out of the box imo. That it is consistently slow only in OPUS and not elsewhere pretty much tells me it's not my system.

Don't mean to be argumentative here but I'm a software engineer myself and I understand how software "should" work and if this is a "settings" issue then it should have been identified a long time ago and the information on correcting it should be available. If it was environmental then that also should be well known by now as it appears others have had the same issue (those that have reported back and/or tested.)

As for every program should be the same... that is completely untrue. You are using your own code for the main manager interface and for the copy process (you aren't using the Windows Explorer APIs to copy the files for example) and so it's your copy code that is the problem. I've tested both Xyplorer and Xplorer2 and neither of those products has a problem.

I really want this to be a resolvable problem. I love Directory Opus overall but this is a deal breaker. Any ideas (other than setting the copy buffer size as I've tried 1mb to 9mb and no joy) would be appreciated. If I can't get Explorer copy speeds then Opus won't work for me.

Thanks.

Agreed, assuming normal amounts of RAM/caching. That detail was not in the original post so the possibility of lots of small files was still there.

Did you test speeds with your setup before buying and find they were OK then, but then ran into issues when trying other thing after buying?

I am wondering if speeds are fast in some tests/situations and slow in others, which may point to a cause and possible solutions.

I understand that, but without seeing the data or being able to reproduce the setup, I have nothing to go on. I can tell you that when I copy large files, speed is identical to everything else. Which of is is "right?" Probably both of us, and the differences come down to the details of what is being done, which are important.

Sometimes differences also come down to how the data is being interpreted and differences in how different programs represent progress, and so on. Possibly not in this case, but I can't rule it out without knowing the details.

Have you checked copy speeds with other types of device? How do things perform when copying between two local SATA drives? If performance is good there then it means we should focus on the type of device involved, and that we may need to get hold of similar hardware to test against ourselves. On the other hand, if the same performance issue exists with a vanilla local SATA->SATA copy, then we can rule out anything to do with the type of device and look in other directions.

The vast, vast majority of people find copy speed is identical between programs when doing proper tests (avoiding caching, and per-file overheads that are different in each program, and so on). This should generally be the case because Opus is literally just sitting in a loop waiting for the OS to copy the data, and not doing the low-level work of reading/writing each bit itself.

There will always be edge cases where one program is faster than another for particular scenarios, but we aren't aware of any major ones. You may have found one, in which case we'd like to know more about it to help us reproduce what you're seeing and look into where the difference is coming from.

Many other file managers just ask the Windows shell to copy files for them, in which case (unless antivirus is doing different things to different programs) performance is identical to Explorer as it's literally the same code being used. (Which also has the same limitations, as the downside. You can call the shell file copy APIs in Opus if you wish, but it blocks off a lot of functionality. Still, it can be useful when doing basic file copies to devices that are only fast when using the shell APIs.)

As an example: Companies who make NAS devices do a lot of testing and optimising with Explorer, since it's what most people will use and judge them by, but sometimes leave giant performance issues (or worse) in their devices when other programs do things even slightly differently. I'm not saying that is the case in this situation, as I do not know and cannot test it, but it has been found to be the case in others.

e.g. People checked how fast a drive performed when Photoshop loaded and saved data vs when Explorer copied files, and they saw the drive was very slow for everything except Explorer (and other things using Explorer's file-copy API). Sometimes those issues are resolved by updating the device's firmware, but curiously they sometimes come back in new devices, indicating that some brands do not carry forward all of their fixes from one device to another.

If you don't want to try things then there is not much we can do. I can suggest settings that may help (e.g. try turning on the non-buffered I/O option in Pefs/Misc/Advanced, setting it to 1MB), but copy speed issues can be caused by a large array of issues, and we have no way of knowing which one without working through the different possibilities and trying various things.

I am willing to try things here... my main issue is that I'm not seeing what I can try that will give real results. Please tell me which tests/logs/etc you will need to diagnose.

I still need to point out that performance in every other app I've tested with show no performance issues (as noted above) with the difference being they use Windows Explorer's copy code to actually copy (dialog box etc). I wonder... is there a way to tell Opus to let Explorer do the actual file copy operation itself?

Anyway let me know what info/tests you'd like me to do. As I said I love the program but the slow copies make it unusable for me. I do wish I had tested this prior to purchase and I'm hoping sincerely we can resolve it :wink:

Note: I'll test with the unbuffered IO setting change and report back.

No change in performance with copy_nonbufferio_threshold set to 1MB. Also tested copy_buffer_size at 1024 and at 8192. No meaningful change.

42gb file copy test:
Opus - 160-200MBps.
Explorer/Xyplorer/Xplorer2 - 1.6-2.2GBps (1600-2200 MBps)

Any other tests/settings?

To confirm, we are talking about writing 2 gigaBYTES per second? That makes this test and the device very unusual.

I am assuming the DS3615xs achieves this by writing to all 12 drives at once in parallel, since no single normal HDD or SSD exists that can come close to maintaining that speed. That could also make the DS3615xs very sensitive to how the data is sent to it.

Where is the data coming from that can read that fast? Another DS3615xs, or a RAM disk or similar?

We'd like to set up a similar test, but devices that can even approach those speeds are rare and costly, and we could end up buying them just to find the problem was outside of Opus.

What happens if you use xcopy or robocopy from a command prompt? Those are examples that won't use the shell file copy API (although they won't be identical to Opus, either).

To back up what I'm saying about some Synology devices being sensitive to minor details of how data is written to them, as well as being improved by Synology's software updates, here are a couple of threads:

Synology have a File Server Performance area on their forum. It may be worth posting there to see what they think.

Another thing to check: Is there high CPU load during the copy? From dopus.exe or another process? That could indicate something is causing a bottleneck.

Have you tried extremely large buffer sizes? Things generally default to 64KB but that will be extremely small if you're transferring at 2GB/s. Setting the copy buffer size to 1MB, 10MB or 100MB to test with those is worth a try. (If you're familiar with Process Monitor, you can also use that to see which buffer sizes Explorer is using when it copies to the device. It may have logic to switch to larger sizes for extremely fast devices.)

I would also try those buffer sizes in combination with the non-buffered I/O set to both 1MB and 0. (So 6 tests in all.)

(Non-buffered I/O being turned on often helps, and will also result in the read and write sides being done in parallel by Opus rather than by the OS, but this can cause pathological issues with some devices as well, as some are only tested for buffered I/O. Note that the copy buffer size is independent of the non-buffered I/O setting; they are about different buffers: One in Opus itself, and the other in the OS/filesystem stack. The non-buffered I/O setting is more an on/off than a buffer size: It specifies the minimum file size before non-buffered I/O is used to copy that file, since it will often slow things down with very small files, even on devices which work well with non-buffered I/O.)

Yessir, GigaBYtes. The copies I conducted were From-To the NAS over a 10GB connection using iSCSI. So the data was never (or should not have been) written locally, just back and forth. The 3615xs is definitely capable of some crazy speeds.

I'll run some further tests as you mentioned. The closest I could think of you trying would be read/writes to a super fast M2 drive. I'm going to actually test that today on my main desktop that has 2 Samsung 950 drives and see how Opus compares to Explorer for the same file sizes.

Will report back...

Copy from M2 to M2 was 800-900MBps using Explorer and 400-500MBps using Opus on my desktop machine. Tested both with a fresh file copy (not cached on write) and the same file copied twice to compare cached writes. File size was 6.54GB and free system memory was 12GB of RAM.

BTW not sure if you saw my question on who handles the copy operation. Any chance there's an option to have Explorer actually handle the operation (ie opening the Explorer copy dialog) like the other products I mention here do? Assuming there is some overhead/issue/behavior with Opus that Explorer doesn't have this would potentially be a nice way to use Opus great interface without the slow down. I do know that if I drag and drop FROM OPUS to Explorer then Explorer handles the copy and I get expected performance.

Thanks

How can you get 2.2 gigabytes / sec over 10 Gb LAN?

This thread was about moving data ("File Move speed is slow"), which would be an instant rename operation if the source and destination were the same device, so I did not even consider copying from and to the same NAS.

Explorer (via the Shell file-copy API) may be telling the NAS to do the copy itself, without any of the data going over the network at all. What does Task Manager's network bandwidth graph show while the copy is happening in Explorer? I believe there is a very low-level API for doing that with newer versions of SMB (or iSCSI apparently*) if the server supports it.

(*Edit: From doing some research, SMB and iSCSI both have different but similar features which were recently added to do this, circa WIndows 8.)

That is not something Opus or many other programs support, but you can call the shell file copy APIs from Opus in cases where you want do that kind of operation, which would give you the same result as in Explorer or other programs that use those APIs. You'd lose extra functionality when doing that, but most of the extra functionality would not be compatible with a file copy which the NAS was basically performing by itself anyway;

Somehow this went unnoticed.

It's bi-directional. Copy from/to combined. Each side is around 1GBps.

I would guess that the "peaks" of 2.5 were probably something wonky with the calculations...but I can only report the numbers. I can say that it's real as I just copied a 44GB file and it took right about 25 seconds.

The copy most likely is not going over the network at all, which probably accounts for all the differences. See my previous post above.

(Copy speed also would not combine from/to speeds together. It would be the maximum of the two, at most. With a normal copy, it's the round-trip speed, always, as that is how fast data is copied from A to B. But in this case, it's probably not a normal copy and the network is probably not relevant to the 2GB/s speed you're seeing.)

I don't know enough to agree/disagree with you Leo, you are probably right, it's possible that it's not using the. That doesn't change the challenge I'm facing however. I just want to get Opus to do these with the same kind of numbers as Explorer.

Unfortunately I don't have a fast M2 drive on the server that's connected to the NAS over 10GB so I can't easily validate if it's doing the transfer completely within the NAS or not. I'm going to see if I can use Aida64 to monitor LAN utilization on the NIC with Opus and Explorer and see if Opus is using the LAN vs Explorer not (ie the iSCSI commands being processed only on the NAS.) Be an interesting test...

Lot to learn here for me at least!

Leo, just validated your theory.

Opus copy of ~40GB file from/to the iSCSI drive used the LAN fully ~125-150MBps transfer with LAN at 125000-150000KB/s.

Explorer at 2-2.5GB/s transfer speed but LAN was... about a few KB so NO LAN traffic at all on the 10GB connections. Interesting! Note: There is traffic when copying to another drive...I need to get those numbers as well. But for now this narrows it down for sure.

So it seems the issue is related to how Opus is handling iSCSI based file transfers maybe? At least when to/from the same iSCSI connection?

See my post #22 above: File Move speed is slow

Try using the linked button to do the copy from Opus (via nircmd.exe) and it should give the same results as in Explorer, bypassing the network.

I'm a Synology user as well.

And i would really really appreciate built in support for

:cry:

I can set that up... just consider this a formal request to add an option in Opus to completely bypass the Opus copy routines...somehow. :slight_smile:

Thanks.

So, hey people, I recently installed dopus and was faced with the same problem. I have samsung qvo 860 and the copy/move speed was around 50MB/s, I did 2 things, I enabled Rapid mode through samsung magician and gave the program full admin rights by default. this fixed it and now dopus is faster than FExplorer and the speed hangs around 220MB/s. I hope this helps some people who face the problem. Though I do nothing crazy and I'm a complete normal user.
I tested by moving multiple files with the total size being 50GB.

Disclaimer: Remember I am an idiot.

Try this program that someone suggested to me. I said sure and I tried it and Directory Opus started go blazing fast with respect to network stuff. It's called MSI Mode Utility Version 2. After installing it and trying it, I was like What the F....

Maybe it doesn't work for everyone and my computer was just so misaligned or something. Evidently, I know nothing about computers.

I do know that previously, when I would press the network button on the side, it would take minutes for the process to end. Now it takes seconds. And, when I access my NAS, it opens up everything right away. It's not a virus or anything like that. That was my first suspicion but evidently a lot of gamers use it to speed up something or another.

I don't have the link. You'll have to find a safe copy yourself. Actually, I might have a copy which I'll see if I can attach to this post.

I didn't do the following but the guy had said it works better if you disable fast boot and hibernation. I think I did this (I can't remember really), I don't think it made much difference. Or maybe he was saying that you need to turn it off to work. I don't know because I AM AN IDIOT. Anyway, try it out. It seems that everything on my computer goes faster. What I am wondering was why didn't people tell others about this small program years before? Maybe they did but I never knew because I am stupid.

Here is the link to the Github page for the tool. Link.

And, here is a white paper from Intel that explains what it does. Link

You'll see that everything on your computer just opens cleaner and snappier. It's pretty cool. Wondering why Windows doesn't license from Intel and incorporate it in their system.