Improve Copy Queue logic for Moves when src/dest same

Tested again by moving a file when src & dest were same, where the file wasn't involved in existing copy queue operation, and it was still queued.

I know the behaviour can be overridden, but this should be the default behaviour since moves when src and dest are same partition don't really cause thrashing when a copy is already being performed on same partition.

It would be nice if moves were always immediate by default between the same drive/partition. There's no reason to worry about thrashing drives with moves even if there is a large copy operation on the same drive so there's no reason for manually moved files to be queued.

The only problem with this is exactly what Leo said: "The files you are moving could be involved in the queued copies."

There is a very simple solution:
Dopus could check all the queues first to see if the files being manually moved are in any current queues. If the manually moved files are not in any queues, Dopus moves the files immediately. If the files are in a queue, Dopus pops up a dialog box asking what to do.

You can change the Move Files button to make it never queue, if you want.

Having to check if every file about to be moved matches any file affected by a current or queued operation would slow things down too much (if done properly), I expect.

I'm jumping into a discussion of something I don't actually care about and being presumptuous enough to think I can interpret what someone else is saying, but I don't think that's what they want. I think they want moves to automatically not queue if and only if both the "old" and "new" location of the moved file are on the same partition.

[quote="rcoleman1943"]I'm jumping into a discussion of something I don't actually care about and being presumptuous enough to think I can interpret what someone else is saying, but I don't think that's what they want. I think they want moves to automatically not queue if and only if both the "old" and "new" location of the moved file are on the same partition.[/quote]That is correct. The behaviour of Explorer and the previous versions of Opus (before queue was added) would simply move a file, even if it would ventually be moved/copied as part of an existing operation - eg. suppose you start copying a large folder with lots of items, and while the copy is taking place, you navigate into this folder and move/copy one of the files/folders. Exploer/OldOpus would not tell you that this is already part of an existing copy operation, it would just copy/move it.

I can see why some people might like the current behaviour, but maybe an Advanced>Misc option could be added ?

Related to Copy Queue logic, I also noticed that a copy was NOT queued if the source was different (one removable, one fixed), but the destination was the same (fixed) - in this instance I wanted the second copy to be queued. I realised that my USB3 HDD was recognised in Windows 8.1.1 as Fixed. After googling, it would seem that Windows 8 certification requires USB flash drives be configured as fixed disks, and HD's also.

Given that the current queue logic uses the fact a drive is removable (or not), this doesn't seem to be the best way on Windows 8+.

In general, it's not a good idea to perform multiple copies on HDD's when the src or dest is used multiple times. SSD's don't suffer from thrashing, so it doesn't matter for them. So, instead of seeing if a disk is removable or not, it would be better if Opus checked if a drive was an SSD or not. You can check this by getting the rpm of a disk (which fails with erro 50 for SSD's).

The new queue logic would then be something like (note: this doesn't include all necessary logic or the Move where src+dest are same suggestion):

If ((src or dest is HDD) && (copies [or moves where src and dst are different] are being performed)) { // check if src or dest of new operation is already involved in existing operation if (src or dest of new operation is already involved in existing operation) { queue the operation return; } } run the operation

So, you treat Removable and Fixed HDD's the same, but instead base the queueing logic on whether it's an SSD or not.

Ah yes, that does make sense. We've got something for that coming in the next beta.

Confirmed latest 11.5.5 beta now moves files immediately when src & dest are same, rather than queueing them. Thanks for this!

Further to my previous suggestions for queue logic, if you start a copy from, say, D: to E: (both fixed according to Windows - ie. show up under Hard Disk Drives in Explorer/Opus), a copy from E: to D: will not be queued. For HDD's, it should be queued IMO since it will caue thrashing. I'm sure there may be other scenarios, but I think the current logic which relies on the removability of a drive is flawed.

Reviving this old thread while Opus 12 is in beta, in case anything can be improved, as I've run into a copy/move queue issue again. I started a move from D: (Fixed HDD) to E: (USB Removable HDD), and then at the same time started a copy from F: (Removable HDD Truecrypt Volume - SATA not USB) to E: (USB Removable HDD). As dest for both is USB removable HDD, I was expecting the second move to queue, but it didn't.

I know buttons/DnD can be configured to work as I need it to, but it makes sense for the second move to be queued in this instance. Same goes for if I was doing Copy for above scenario.

If dest was SSD, I would say don't queue, but for HDD it should queue IMO.

Any chance this can be changed for Opus 12 ?

If the destinations are on the same drive and auto-queuing is enabled and not being bypassed by the command, then the operations should queue automatically (and do here).

There must be some extra detail here.

First let me say I'm using Opus 11.18 x64 on Win 8.1.1 x64. Haven't got around to trying the 12 beta yet, although I will in the near future.

The command for "All files and folders" file type for "Drag-and-drop + Shift" is "copy move". Under Prefs > File Operations > Copy Options, I have "Automatically manage file copy queues enabled".

Anything else you need me to check ?

I forgot to mention that the USB dest HDD is also a Truecrypt encrypted disk (ie. whole disk, not a partition), so maybe Opus has trouble identifying it's the same disk, even though the Truecrypt mounted partition letter is the same.

Are both the copies we're talking about going into the same Truecrypt partition?

Does it have a drive letter (that is for it and only it, i.e. not a mount-point/junction below another drive's letter)?

The destination is whole disk encrypted and so, when mounted, has a single partition letter in Windows (ie. so both copies have the same partition latter in Windows).

I've done some more testing:

  1. D: (SATA Fixed) to E: (USB Truecrypt) and D: to E: at same time - second Move gets queued, as expected.
  2. D: (SATA Fixed) to E: (USB Truecrypt) and G: (Fixed) to E: at same time - second Move gets queued, as expected.
  3. D: (SATA Fixed) to E: (USB Truecrypt) and F: (SATA Truecrypt) to E: at same time (and to same dest folder) - second Move is not queued.

So, when the dest are both Removable USB Truecrypt (with same letter) and the sources are both Fixed, it queues the second Move, but when the dest are both Removable USB Truecrypt (with same letter) and first source is Fixed and the second source is Removable, it doesn't queue the second Move.

To reproduce, whole disk encrypt a USB HDD and a SATA HDD, and then mount them both. Start a copy from a Fixed SATA partition (not Truecrypt encrypted and showing as normal HDD in Opus/Explorer rather than Removable) to a folder on the USB Truecrypt HDD. Then start a copy from SATA Truecrypt HDD to the same folder on the USB Truecrypt HDD.

Neither the sources or dest are mount-points/junction.

[quote="Deipotent"]1) D: (SATA Fixed) to E: (USB Truecrypt) and D: to E: at same time - second Move gets queued, as expected.
2) D: (SATA Fixed) to E: (USB Truecrypt) and G: (Fixed) to E: at same time - second Move gets queued, as expected.
3) D: (SATA Fixed) to E: (USB Truecrypt) and F: (SATA Truecrypt) to E: at same time (and to same dest folder) - second Move is not queued.[/quote]

Now it makes sense, after reviewing the default queuing rules documented in the manual:

[ul][li]If the copy originates from a removable disc, a CD/DVD or an FTP site, the copy queue is used if an existing copy from that source folder is in progress[/li]
[li]If the copy is to a removable disc, a CD/DVD or an FTP site, the queue is used if an existing copy to that folder is in progress[/li]
[li]If the copy is to a fixed (local) hard disk, the queue is used if an existing copy to that physical hard disk is in progress (multiple partitions on the same physical drive are considered the same)[/li]
[li]For all other copy operations, the queue is used if a copy is in progress from the same source to the destination[/li][/ul]

Perhaps it would make sense for us to change the first rule so it only considers the source removable drive if the destination isn't also removable. That said, there are no right answers for every situation here.

My simplified suggestion for rules would be (in priority order):

  • If a copy/move is in progress and a move is initiated, where the source partition and dest partition are the same, then don't queue. Reason is that move within same partition doesn't usually cause thrashing, as the file data does not actually need to be copied.
  • If a copy/move is in progress and a copy/move is initiated, then the queue should always be used if either the source or destination is the same physical disk/CD/DVD or FTP site as the copy/move already in progress.

If there is a way to detect if source or dest is SSD, you could always not use the queue, since they are more suited to multiple read/write requests. This could also be made optional.

Would this not work ?

Queues work by name, so one (and only one) drive letter has to be chosen (or both letters are used, but then things only get queued if both are involved).

Given my suggestion couldn't it work like this:

  • When a copy/move is initiated, name a queue containing both drive letters and both disk physical ID's (eg. "D:E:1A2B:1A2B" means source drive is D: with physical disk ID of "1A2B" and dest is E: with ID "1A2B").
  • When another copy/move is initiated, check if source and/or dest physical ID matches that in any existing queue name and, if so, queues it to that named queue. eg. 2nd copy/move source is letter F: with source ID is 3C4D, and dest is E: with ID 1A2B. 3C4D does not match serials in "D:E:1A2B:1A2B", but dest ID does match both source and dest ID (a single match of either source or dest ID would be enough) and so should use that queue.

In the case where souce or dest is not a HDD/SSD, the queue name could consist of letters and some ID for CD or FTP.

In fact, I'm not sure if the drive letter is really needed in queue name.

As mentioned previously, the first check would be to see if the source and dest partition letter are same and a move is initiated, in which case it would never queue. Then it would check the ID's against the existing queue names.

Thinking about this some more, what appears to be a simple problem is a bit more complex :slight_smile:

Using my suggestion as an example will solve the problem of disk thrashing, but leads to inefficiencies since some queued items could run immediately once the original job (ie. the one that the queue is named after) has finished. For example,

1st job is copy to different physical disk: C: > D: (queue named "C:D:1A2B:3C4D")
2nd job (started while 1st still running) is copy to different disk E: > D: (added to queue "C:D:1A2B:3C4D" since dest matches queue name)
3rd job (started after 1st ended and 2nd job running) is copy to same disk (could be different as end result is same) C: > C: (added to queue "C:D:1A2B:3C4D" since source+dest matches queue name)

Given the above scenario, the 3rd will wait until the 2nd job has finished, but it could actually be run at the same time, since the physical disk is diferent.

The simple initial solution is to avoid disk thrashing, so I believe my suggestion is preferable to the current solution.

An ideal solution, which avoids disk thrashing, while also avoiding inefficiencies would be to allow multiple jobs in a single queue to run concurrently, if the src/dest ID's do not match. Opus could do the following when a new copy operation is initiated:

  • Check any existing queue names to see if it should be aded to the queue (ie. if source or dest disk ID matches either/both ID's in queue name, then it's a match).
  • If it finds a matching queue, the job is added to the queue, as before, but Opus will also check if the source/dest ID's of the new job match any of the currently running jobs in this queue. If a match is found, the job will be queued (ie. waiting to run). If a match is not found, the job will be run immediately (ie. so multiple jobs in the same queue could be running concurrently).
  • To kep maximum efficiency, one more thing needs to be done - when a job finishes, Opus should check all non-running jobs in the same queue to see if they match any of the running jobs. Any which don't match can be run immediately.

The ability to manually create new named queues complicates things further, since a job which has been added to one of these user-created queues could potentially clash with one of the auto-queued jobs. The solution is to also check any running jobs in manual queues before starting a new job running.

While all this may seem a little complicated, it's not overly complicated IMO. The main issue would be the UI that would be needed for running multiple jobs concurrently in a single queue.

However, as Opus' main job is file management, then it would be time well spent, as it would maximise copy/move efficiency when performing multiple copy/moves.