I have plenty of memory. When I copy files, my HDD are the bottleneck. I would like to speed up copying of many small files, and extraction of many small files.
I think this can be accomplished by increasing the buffer used by directory opus. E.g. Opus should read say a hundred small files into a memory buffer first, doing a lot of reads. THEN when the buffer is full, it should write the hundred small files to the destination from the buffer.
Is this how Opus operates, and where can I adjust this buffer?
It seems currently Opus cycles between read and write for every single small file, instead of using such a buffer. Or something else is seriously slowing it down.
Under Preferences / Miscellaneous / Advanced: [Filesystem], the copy_buffer_size and copy_nonbufferio_threshold settings can be used to adjust how data is buffered during copies.
With lots of small files, buffer size is unlikely to be the issue. Windows/NTFS has quite bad per-file overheads, on top of which are optional things like copying timestamps and attributes that can cost as much time as creating the files themselves. See Preferences / File Operations / Copy Attributes for those options.
Thanks I am still wondering if Opus actually buffers the content of many small files in this buffer?
My age-old Dos Navigator âDNâ on MS-DOS did this and even informed the user about it: it would only read and copy many hundred small files into a memory buffer of 10 MB or so, until the buffer was full. Then, after that was done, it would write those 10 MB (many files) to the destination. Then read another 10 MB etc. This way, even when copying a thousand 10 kB small files, there was only one long read and one long write operation, instead of a thousand small interleaved read/write operations.
We've tested that and there's no point doing it with NTFS at least. The overhead is in creating the files, not in reading or writing the data. NTFS is extremely slow at creating files, and you can't really do anything about it when copying lots of small files.
I found the problem what made copying so slow for me: the drives I was copying to were SMR type drives, which have horrible (factor ten or worse) and erratic write performance compared to CMR type drives.