Another Piece to the "Slow Copy" Problems

After reading many of the "Why is DOpus so slow copying my files" posts, I have some more data to contribute.

In recovering from a HD crash I had copied a few partitions over to an external USB 2.0 drive and I had purchased a Yellow Machine NAS box from Anthology Solutions (http://www.anthologysolutions.com)to relocate my recovered partitions to temporarily.

In the process of copying files to the NAS I noticed some repeatable things with large volumes of data: (permissions off, counting on)

  1. Folders with lots of small files (the Mono source tree) would drop the transfer rate under 1KB/sec.
  2. Folders with large files (VMWare virtual disks - 2G each) would be close to the expected transfer rate (6G/hr)
  3. My backup software (Dantz Retrospect) gets the expected speed. It seems to 'batch' data into large chunks as well.

Windows Explorer is closer to behavior to #2. (Timed by hand)

Is there some buffering that is going on by default in #2 and by design in #3 that is more efficient when copying files over the network? And can this be turned on in the OS or by changing the way Dopus behaves when copying to UNC shares (or mapped network drives)?

adam hill...

Do you mean that Explorer and your backup software are able to copy lots of small files (case 1), including setting their timestamps and attributes, at a fast speed, while Opus isn't? Or something else?

Correct, Explorer was much faster than copying a 'mix' of files than Dopus was. But If you were copying multi mega/gigabyte files they were much more comparable in speed.

I have the same behavior when copying from my digital camera using usb 2.0 cabel. Explorer copies everyfile in the same manner, Dopus seems to copy some, then use longer and longer time after each file...

It's not that big of a problem for me, but I can see it is a big problem for people with BIG files and lots of them...

I've just done some tests and, while the results vary by up to 20 seconds each time, copying 1.2GB worth of program files, including Photoshop which has thousands of tiny files and takes the majority of the time, takes roughly the same time in Opus and in Explorer.

I also tried disabling the pre-copy count and the attribute/description/timestamp copying options in Opus which have been said to have an effect on speed in the past, but my tests aren't scientific to really say how much and they didn't seem to make a significant difference to me. (The a/d/t options, that is. The pre-copy count takes about 20 seconds in both Opus and Explorer.) Since Explorer always does a pre-copy count and preserves attributes and timestamps (but not descriptions) I went with that for another test in Opus and it actually copied a bit faster than Explorer, for what it's worth. (Since I don't use Descriptions much I figured I'd leave that option off after my test. :slight_smile:)

The most important thing to remember when testing different programs' copy speeds is to eliminate the effects of caching. If you copy with one and then copy the same data with another program then the 2nd program will often be much faster because the directory, and sometime some of the data, will be cached. To eliminate this I made several copies of what I wanted to copy, then played a game for a while (which eats up all available memory), then did copies from each souce directory to the same destination, deleting the destination after each one.

Both source and destination are NTFS for me, and they're on separate physical drives. (One an internal laptop drive running at 7200 RPM and the other an external USB2.0 drive also running at 7200 RPM.)

Another factor that may be important is real-time virus checking. It's possible that Opus and Explorer copy the files using a different API call or maybe Opus opens the files more than once while Explorer only opens them once (or vice versa) or whatever... Things like that may cause extra slowdown with some real-time virus checkers but not others. I'm using NOD32 for what it's worth, and I haven't tried with it disabled since the results for Opus and Explorer seem so similar already with it enabled. Other virus checkers may produce differnet results, though, so try with it disabled if you can. If there is a big difference with the virus checker enabled but not much difference when it's disabled then maybe GPSoft can look at what may be triggering that kind of slowdown, but someone would have to do tests to show that this situation really does happen as I'm just guessing at things which may cause differences.

You might be able to improve this by changing the buffer size that Opus uses in Preferences / File Operations / Copying (2) -- some people have reported that other values work better with their USB memory cards, although there's no hard and fast rule.