SSD usage with unbuffered async IO and file copy queues

Thanks to the new async unbuffered IO copy operation the new beta is rocket fast when copying large files between different devices.

To maximize the out-of-the-box performance I propose to extend the automatic management of file copy queues by introducing a configuration option to flag certain devices as flash-device. Since the file copy queues are vitally important to avoid frequent head seeking for old-style harddisks in case of flash-devices this mechanism will slow things down.

Particurlarly when copying from HDD to SDD (or the other way round) the SDD is able to handle multiple operations in parallel without any noticable slowdown of an individual task.

Thus I would really appreciate to have an option allowing me to disable automatic file copy queues for flash-devices to maximize the perfomance of multiple copy operations that are performed in parallel.

Example:

Having a system with an internal SSD, an external HDD and a connected USB-Stick.

Performing the following copy jobs:

  1. From SSD to HDD -> immediately started (all devices were idle) -> taking 5 min
  2. From USB-Stick to HDD -> queued (the HDD is busy) -> taking 5 min
  3. From USB-Stick to SSD -> immediately started (both are marked as flash-device) -> taking 5 min

In this case the overall copy process will be finished after 10 min (instead of 15 min) since the copy job 3 could be performed in parallel.

Why not turn off automatic queueing and create a separate copy button (or edit the main copy button so it does something different if Shift is held down, for example) which uses queueing? That way in the usual case where you don't want queueing it won't be used, and in the unusual case where you need it, you can ask for it.

Or leave it how it is and just click "Run Now" every so often to unqueue into a parallel transfer.

I did a quick test on my own SSD and the speed dropped to 70% when I added a second file transfer. Still faster than the effective 50% speed of doing the transfers in series, but a definite hit. How beneficial or detrimental it will be to avoid queueing will depend on a lot of factors of the hardware, drivers, anti-virus, etc. (not just of the target drive but of the source as well). Some drives completely fall to pieces when faced with sustained parallel requests, and sequential write speed is very different to random write speed even if there isn't a physical head to move around.

Our automatic queueing system is only meant to be a simple thing (in terms of both the code itself and of users understanding what it will do and predicting what will and won't be queued). If automatic queueing isn't doing what you want, you can either override it (the "Run Now" button) or switch to manual.

Still, we'll have a think about this.

After doing a little research, there is no reliable way to actually detect that a drive is an SSD, unfortunately.

Windows itself uses some heuristics (looks for certain strings in the drive identifier -- which are not there for all SSDs -- and also assumes drives are SSDs if their speed is above a certain limit) but I don't think it exposes the results via any API that we can use.

Maybe in the first step a simpel configuration dialog would be enough?

Maybe a solution that allows the user to disable the queuing feature for each drive individually in an (advanced?) configuration section besides the file copy queue area in the DOpus preferences.

In such a section all available drives are listed (disk number and vendor) and can be excluded from automatic file queuing. The drives are maintained by their unique disk id to allow a reliable unique identification.

I think that's be too burdensome a list to expect GPsoft to maintain...

I think it'd be more reasonable to have either:

  • some sort of "Disable/override queueing/buffering on copies to drive" list or something in Prefs that woudl let you specify drive letters or labels?
  • some sort of override switch for the copy command... that you could specify drive letters or labels with in order to exclude from queueing, even if the command the arg is included in otherwise specifically calls for queued/buffered operation?

...yes, it would require maintenance on the users part as you add new drives to teh systems you run Opus on, but it'd get the job done. To minimize the effort re: maintaining, I guess it'd be better to have a 'global' drive list in Prefs to modify the behavior of all copy operations.

Certainly not! :wink:

I meant that all available drives that are currently installed in the system are shown in a list using the vendor name and drive number to allow the user to explicitly disable queuing for each drive.

...oh. In that case, I (personally) think that it's better to be able to free-form specify drive letters or drive labels. If the general idea has any real merit, then I don't think it "helps" to display the drives with vendor strings, etc and I would imagine you'd not want it to be a dynamic-only list of devices that are 'currently' connected to the computer. You'd want Opus to 'remember' how to treat (for instance) removable drives - an eSATA attached SSD for instance, which isn't always connected to the system, etc. Yes, if you have multiple partitions, then that's "more to do" to exclude several logical drives on a single SSD, butI think most people associate drives with letters and labels more easily than they do vendor ID strings.

Just an opinion... I'll be quiet now.

Since queuing appears on disk level instead of partition level you have to exclude the disks. Personally I would propose the same name as the drives are shown in the Windows device manager.

Example:

[o] Drive-1: Samsung HD154UI (SATA-Device)
[x] Drive-2: WDC-WD30-EZRX (USB-Device)
[x] Drive-3: JetFlash TS1GJFV30 (USB-Device)