How to optimize the copy buffer size?

Is there any optimal buffer size to serve all file sizes? I just had to stop a 22 GByte copy, which would have taken around 1 hour, 45 minutes, using the 64 kb default buffer size, with speeds around 0,8-1,4 MB/s, which is very poor.

After having changed it to 1 MB buffer size, almost half of the job is done after only 7 minutes, averaging of ~28MB/s. So, is there any known size, which gives the best balance for very small and large sizes?

You have to experiment as it will depend on your setup.

The non-buffered-io setting can also speed things up (or slow things down), although generally you want that one set to either 0 or 1MB.

Ok, i will experiment a bit. For now, it seems like 1-2MB is already clearly faster than with 64kb. Thank you.

When copying via DLAN (PC to HTPC) I always notice, that the copy dialogue shows 100%, but need some time to close/finish (with 64kb and 1,5GB file nearly a minute). With 512kb I have best/max. results to all destinations (LAN, WLAN, DLAN and different devices), while 1MB slows down a bit.

Thanks, Sasa. Indeed, 512 looks promising. I just had tested 2048kb with 1 MB of "non buffered", and it was about the half of the speed compered to the 512 kb setting, which yields around 30MB/s. I really wonder, if my next system finally will be able to provide USB3 speeds. Although i have two USB3 drives, i couldn't ever get the computer to actually function that way. I guess, my board is too old, although it is supposed to support USB3, or i can't find the proper drivers or any hint, where the problem lays. Anyway, 30MB/s with large files is a good starting point, and i don't think, i ever saw any higher transfer rates on this system. I will have to run another test later, to see, if it also works with lots of small files. :thumbsup:

As said I only had trouble with DLan (one adapter in living room is half as fast, so it's not DO).

With USB3.0 there's no noticable difference between 64kb and 512MB, it's always around 90MB/s (min. 80, max. 160) to HDD and 200+ to SSD (from SSD of course).

Is it as slow when using Explorer? Is 3.0 onboard or via PCI-card?

Edit: Overread that you board supports 3.0, with Win 8 you normally need no additional drivers, but IMO 3.0 isn't as stable as 2.0... I have an external 2-bay-case, when turning on while sys is running, Win doesn't recognize the drives and I have to restart or de- and reactivate Intel XHCI-drivers. But fortunately this is the only problem.

Thanks Sasa. I had that topic already in the coffeeshop. Maybe we can discuss the USB3.0 mess there, since it is off topic here?

https://resource.dopus.com/t/anyone-on-windows-10-already/20158/1

Isn't that the weird windows cache, that prefetches data into memory first and then fakes a finished copy-operation e.g., while actually it is still busy? Which then is the reason the 100% dialog stays for so long..

Maybe, but that happens only when copying via the one slow DLan-adapter, nowhere else.

Can someone explain how buffer sizes are related to copying files? I've read a couple of stuff on the internet and I still have no freaking clue. I guess it's such a simple topic, nobody defines what it means. I've been searching for 1 hour and I can't understand it. Bigger buffer is faster but not as safe? Smaller buffers use more system resources? These are all questions that I have no idea what they are even talking about.

What is a buffer? And, what does it have to do with copying? It seems like it's an alloted amount of space on your drive. When it's full, the program starts moving it to its destination. But, why is there a buffer in the first place? Why does the size matter? Why not just make the buffer size as big as you want like 3 gigs?

Can somebody please explain how it all works in simple language to someone who doesn't know anything about computers? Thank you so much.

My advice is to leave it alone, especially if you don't know what it is.

It's not a simple topic at all. There are a huge number of factors that can decide what the best buffer size is for a combination of devices, drivers, hardware, OS, networks, caching, antivirus, filetypes, etc. But it also shouldn't usually matter; the defaults work OK on most setups.

Measuring copy speed is also very hard to do properly, especially if you're comparing different programs which report speed and even duration differently.

Unless you're having problems that affect your ability to get things done, it is best left alone if you're already unsure what to do.

Leo,

Thanks for your help. It just really piquéd my curiousity. I wanted to know because I'm going to transfer my entire hard drive to my NAS twice. Is it normal that it takes 2 hours to copy only 200 gigs on a gigabyte network? Seems awfully long. Maybe my brain is having a fart but I thought copying was like 10 times faster. I have about 1TB on my SSD that I need to transfer and thought it would take an hour. So, I waited and waited and waited... I soon realized after about 30 minutes, I was getting about 17 MB/s which is about 136 mb/s. Is that normal?

I canceled after about an hour as I realized I wasn't going to finish and I was wasting my time. I'm wondering what is the bottleneck. The NAS itself has an average speed of 100 MB/s. Has gigabyte ports. The NAS's hard drive speed is something I'm not sure of. Specs seem to say something like 240 MB/s but that seems a little high. I don't know if I'm looking at the right thing.


(I have the 12 TB)
Cat7 cables. Switch is gigabyte too. Computer is gigabyte port. SSD is a SAMSUNG 970 EVO Plus SSD 2TB - M.2 NVMe which has a write speed of 3.5 gig/s. Where is the bottleneck?

I've checked my network speed about a year ago from SSD to SSD and I think I remember it was little less than gigabyte speed. So, why is it so slow here? (I know it's obviously a hard drive but 17 Mb/is seems awfully slow, no?

Your hardware probably has Gigabit ports. This typically results in transfer speeds of 70-80 Megabytes/s. 17 Megabytes/s is more USB 2.0 and WLAN speed territory.

Transferring 200 Gigabytes in 120 min would mean 28 Megabytes/s - still a bit slow. But 10x faster? That'd be more than you can expect.

There is some per-file overhead, so if you were copying many small files this could be part of the explanation.

1 Like

So, you're saying if I was copying a 30 gig movie, I would see faster speeds due to lack of overhead. But, if there are a lot of small files like I have, it would decrease it. What's a typical speed for these 2 situation, if you happen to know off the top of your head.

Can I increase the speed somehow by increasing the copy buffer? What is going on? What can I do to dramatically increase the speed?

Typical speed depends on a lot of things, many of which we cannot predict. One of which is whether you are copying lots of small files or a few large files.

Are you copying lots of small files or not?

Leo,

To be honest, I don't know. It's probably a lot of smaller files. I'm moving all my data folders that equal up to 1 tb. Most are probably smaller files but also some bigger video files that take up most of the space.

So you haven't looked into the details of what you're doing? I would strongly suggest doing that first, rather than messing with buffer size settings or assuming they will help. The slowdown may be elsewhere.

(I'm hesitant to give examples as I don't want to suggest more switches to flip which may or may not help, and which may also make things worse or cause confusion or problems down the road, without knowing if they are relevant.)

It's usually very easy to tell if you're copying small or large files, since you can see each file go by in the progress dialog.

With something like this, it's best to understand the problem first before you (or any of us) can evaluate potential solutions.

By the way, I am just in talks with support of a backup program, which tends to use (as far as I can tell from samba logs) variable buffer size of 1 MB or 4 MB. This results in transfers below 200 kb/s throughput for large files, with Opus copying 4 times faster with the default settings.

Xyzzy,
Thanks for the helpful info. I'll try using 1-4 mb for small files and then much larger for large files like movies.