REQ: Improved queue handling

Help & Support
1

A suggestion to impove copy/move behavior in Directory Opus.

A simple benchmarking tool that test read/write speed for larger/smaller files with 1..3 threads. Store the results base don drive ID/serial number. (Important for removeble drives)
The queueing of files Works really great but can be improved.
When I start multiple copy actions C->D they get queued as they should be.
When I start C->D, C->F and G->C three copy threads are started.
Good or bad? Depends on the speed of the drives involved.

Let’s assume:

  • C is a very fast drive
  • D is a fast drive
  • E slow USB drive
  • F slow USB drive
  • G slow network drive

C is fast enough to “interact” with E, F and G simulataniously. Doing so improves the Total copying time.
E, F and G are slow and can’t handle any multthreaded action

So what I’m trying to say (request) is that Opus not only queues copy/move/delete actions but manages them in such a way that each drive is working at the best possible speed; which will reduce the total copying/moving time.
My system drive can handle 3 slow threads with great ease. The throughput simply triples.
But 3 threads on a USB drive (at least the ones I own) gives a Total speed that’s hardly half of single threaded performance.

I hope the above is clear. If not ask and I’ll try to re-explain.

2

I like the idea and for the first step I would propose to introduce a configuration option to exclude certain devices from automatic queue handling.

Particurlarly solid state disks can read/write data simultaneously much faster than with a single copy thread. In other words the overall throughput will increase if the file copy tasks are not queued.

Thus I would like to have an option in the preferences that allows me to disable the file copy queues for fast flash-memory based devices (SSD, USB3-Sticks, etc.).

3

I think adding complicated heuristics, requiring drive benchmarking in advance, would just make it unpredictable when Opus will/won't queue things, for the few people who cared enough to benchmark all their drives.

Even if we could come up with a good way to benchmark how various combinations of drives and loads would perform, and an algorithm to use that information to decide how to queue things, I'm not sure how well it would work or how many people it would really benefit.

If someone else wants to go down this path, we could provide a simple plugin API where Opus passes the plugin code/script the source and destination paths and it replies with the name of a copy-queue. The plugin could then apply whatever logic, benchmark results, etc. it wants.

IMO, though, it's something better handled by manual queueing. You can set up buttons which copy/move files and force that job to be added to a particular named queue. If the simple automatic queueing isn't doing what you want, you can use buttons like that to manually queue different jobs to different queues, and that allows you to evaluate what makes sense at the time.

4

But even among the fast media you mentioned (SSD, USB3-sticks, etc) large performance difference can be found. So, imo, in that repect they are no different from any other media. But obviously it's much more difficult to reach the limitation of a quality SSD than of a cheap USB2 drive.
So, with the proper benchmarks and program logic there shouldn't be much/any difference between a floppy and a high end SSD. Configuration wise that is.

User configs to exclude drives is always a good thing.

Maybe some extra performance can be gained by using large copy buffers. (frequently I have a few GB memory waiting to be used)

5

[quote="leo"]I think adding complicated heuristics, requiring drive benchmarking in advance, would just make it unpredictable when Opus will/won't queue things,[/quote]The only thing I care about is getting the job done ASAP. I think that's the same for most people. 1 or 10 active threads? I personally don't care at all. Speed is all that matters to me :sunglasses:

[quote]for the few people who cared enough to benchmark all their drives.[/quote]Perhaps on-the-fly logic? Or simply don't use it if people don't care to benchmark.

[quote]Even if we could come up with a good way to benchmark how various combinations of drives and loads would perform,[/quote]It doesn't have to be very advanced to work. I'm not asking about the squeezing out the last 1%.
A very simple user config that limits the max number of threads to/from a certain drive would help. I think it's already half of what I requested. In other words: Opus should never queue if it hasn't reached the configured number of threads for a certain drive.

[quote] and an algorithm to use that information to decide how to queue things, I'm not sure how well it would work or how many people it would really benefit.[/quote]The same can be said for many things in Opus Leo. 95% what can be found in Opus is useless for me. But I bought Opus 6...10 based on the other 5% :thumbsup:
I upgraded to Opus 10 for the queue and picture upload. That was worth my money. I'll only upgrade to v11 if there is something new in it for me. Improved queue handling would cause me blindly to upgrade without even checking out other new things.
But, yes, I'm just one user and I have no idea the programming costs will ever be earned back by selling copies based on that functionality.

[quote]IMO, though, it's something better handled by manual queueing. You can set up buttons which copy/move files and force that job to be added to a particular named queue. If the simple automatic queueing isn't doing what you want, you can use buttons like that to manually queue different jobs to different queues, and that allows you to evaluate what makes sense at the time. [/quote]Better than nothing I guess :smiley:
But in that case all the queues should be able to be merged into one "transfer management".
But please! build in some logic (number threads) because reconfiguring copy actions every time is a waste of time.

6

Title of topic is just "Improved queue handling", so I hope that it is right place to ask:

Why automatic copy/move queueing is not bidirectional?

Currently if I move big file from pendrive P: to harddisk partition H:,
and then
Copy other big file from H: to P:

both operations will run in parallel.

If P: is (rightly) considered as slow device it can be assumed, that if P: is source of FROM operation, it cannot be at the same time destination of TO operation without hurting performance.

7

That's exactly the problem I experience too, Konrad.
I really would like my first post to be implemented but setting the max number of threads shouldn't be that difficult.
Setting it to 1 for your pendrive would solve our problem.

In this case there is an additional bonus.
Say your pendrive is full. First you queue moving the large file FROM the pendrive.
Then you queue moving a large file TO the pendrive.

8

[quote="FileNerd"]
In this case there is an additional bonus.
Say your pendrive is full. First you queue moving the large file FROM the pendrive.
Then you queue moving a large file TO the pendrive.[/quote]

Indeed. My example was based on such real life story. :slight_smile:

9

How often do you have lots of large, concurrent copy jobs running between such a varied combination of devices that you need the computer to decide how to queue them? :slight_smile:

We'll never be able to make the automatic queuing guess correctly for every situation, and I think it is better to keep it simple and predictable rather than make it really complex and hard to predict/understand.

Right now, by default, copies are added to a queue if they are going to the same destination, and you can override that if you want by forcing some or all copies to go into one or more named queues.

When doing automatic queuing, Opus internally generates a queue name to assign each job to, and right now it uses the name of the destination drive. You can already make it use the source drive if you prefer, e.g. if you tend to copy from source to lots of destinations rather than from lots of sources to a single destination. Using both the source & dest drives in the queue name wouldn't work, of course, unless you only wanted jobs to queue if they were working in the same direction between the same pair of drives.

We could change the internals to add special logic for the bi-directional case, but it becomes complicated quite quickly, for the user interface as well, if you think things through. For example:

[ul][li]Drive A is allowed 3 threads, which are used for copying to drives C, D and E.[/li]
[li]Drive B is allowed 3 threads, which are used for copying to drives F, G, and H.[/li]
[li]A copy is then started from Drive A to Drive B. Which queue does it get added to?[/li][/ul]

There's no good answer to that. If you do find yourself in such a complex situation, I think using buttons which manually push the jobs to specific queues makes more sense.

I guess what you're all really asking for is to have copy jobs which are held up until certain conditions are met, which not the same as queuing. (Jobs would be able to jump the queue or form new queues depending on the current conditions.) That would be quite a redesign and would make the progress and configuration user-interfaces more complicated than we feel is necessary, at least right now.

[quote]Say your pendrive is full. First you queue moving the large file FROM the pendrive.
Then you queue moving a large file TO the pendrive.[/quote]
A button which you can click, in special situations like that, to put both operations into a specific queue solves that problem already.

That seems like a good example of what I'm trying to say: When the automatic queuing doesn't work, simple manual queuing solves the problem well already.

10

I don´t think that´s a good idea. Who really needs to "optimize" stuff like that? Just be patient, or (sorry) buy all fast drives.

11

[quote="leo"]How often do you have lots of large, concurrent copy jobs running between such a varied combination of devices that you need the computer to decide how to queue them? :slight_smile:

We'll never be able to make the automatic queuing guess correctly for every situation, and I think it is better to keep it simple and predictable rather than make it really complex and hard to predict/understand.[/quote]

From time to time.

In case of transfers between slow devices, like USB, and fast devices, like local HDD, there is no chance for bad quess. It should be queue of slow device*.
If both source device and destination device are already commited in queued operations (in two separate queues) AND both are slow or both are fast, there is chance for bad guess. In case of bad guess the queued operation will run too early (but never too late).
Not quessing at all and running the operation as first operation of third queue is worst case - it will be always too early.

*) assuming that all queues are automatic and there are no conflicting named queues started by user.

12

[quote]How often do you have lots of large, concurrent copy jobs running between such a varied combination of devices that you need the computer to decide how to queue them?[/quote]Often. Why is the a queuing system in Opus? Must be for a reason.

[quote]A copy is then started from Drive A to Drive B. Which queue does it get added to?[/quote]Doesn't really matter for me Leo. The main point is that the drives are maxed out as much a possible. Ok, it never gets perfect for all situations. Buy it's far better than using only 15MB/s when the system could easily handle 10x more. The order in which the queue is processed might not be perfect but the total time taken will be much less.

[quote]A button which you can click, in special situations like that, to put both operations into a specific queue solves that problem already.

That seems like a good example of what I'm trying to say: When the automatic queuing doesn't work, simple manual queuing solves the problem well already.[/quote]It would be an improvement if it existed :slight_smile:

[quote]When doing automatic queuing, Opus internally generates a queue name to assign each job to, and right now it uses the name of the destination drive.[/quote]How about "sorting"the drive letters and then creating a name from them. That would solve the problem Konrad described.
Add to that max number of simultanious read/write actions for each drives and things improved quite a bit without any AI.

13

It would also be great to have ALL transfers in one "transfer window". (including totally unrelated copying threads)
Plus the ability to re-order queued transfers.
I think that's a must if things are going to be handled manually.

14

You can do that already by sending everything to the same queue. Of course, only one thing will run at a time then.

15

[quote="abr"]I don´t think that´s a good idea. Who really needs to "optimize" stuff like that?[/quote]If you don't need it then simply don't use it,

[quote] Just be patient, or (sorry) buy all fast drives][/quote]I think you don't (fully) understand what my request is about. Even buying the fastest possible drives won't solve what I described.

16

[quote="leo"]You can do that already by sending everything to the same queue. Of course, only one thing will run at a time then.[/quote]Opus decides the queues for me.
?

17

Even fastest drives have some limits of transfer and seek time greater than 0.
Optimization is about achieving of maximum from existing situation.

P.S.
I have heard about Samsung HDD with infinite capacity. It will come to retail as soon as it will be formated so... be patient.

18

[quote="FileNerd"]Opus decides the queues for me.
?[/quote]
You can send things to any queue you want. The Copy command can be given the name of a queue to use, and anything else that is sent to the same queue will wait in turn.

If you edit all the things that copy files to use a specific queue name then they'll all use the same queue.

(You can turn off automatic queuing via Preferences, too. And you can have different queues on different buttons, or if you hold Shift down, or things like that.)