Flat view locking up

Help & Support
1

I have a 5TB drive, 86% full, with 660 dirs at root level. Each top level dir has between 10 and 50 sub-dirs (give or take). These sub dirs have files in them, let's say they are book reviews. The sub dir is named " - ". These have been created over the years, and there is no relationship between where one book review by one author is and another. In other words, any one author's reviews may be in any number of different top level dirs.

I keep track of this via a spreadsheet, which works, but is becoming unweildy.

I decided to use flat view and a filter, and then checking filter folders, to pull out selected and often used authors, so I could move them all into one dir, rather than have them spread over dozens, and requiring me to refer to a spreadsheet to find anything. Problem is, Opus locks up when I try this, and I tried it three times, using three different examples, and some of the numbers that were showing at the time of the lock up are all quite close to one another, making me wonder if this is significant in some way. BY lock up, I mean 'Opus is not responding' rather than an actual crash. Each time I exited Opus via task manager.

From left to right on the filter bar, the numbers were as follows for each of the three attempts.

3385, 131, 568,820, 19,361 and 588,545 hidden
4715, 105, 570,454, 19,657 and 590,205 hidden
237, 21, 571,968, 19,741 and 591,803 hidden

So, is this a bug, or ?

Is there another way to do what I am trying to do? I guess I could work around it by temporarily moving say 50 or 100 dirs at a time into a temp dir, then doing it, but that seems like cheating, and it is a pain in the butt :slight_smile:

2

How long did you wait?

What was the CPU usage like?

3

I didn't wait for long, because from start to finish, the numbers kept ticking over, then they just stopped. The first time it happend, when I saw the numbers stop ticking over, I thought nothing of it, but after a bit of time, who knows, a minute maybe? I tried to abort and it all went downhill from there. CPU, not sure, I will try again, using one of the same filters, and see. ta

4

Ok, tried again. CPU for opus task at 8%. Same result.

3385, 131, 568,820, 19,361 and 588,545 hidden << as per first post
3396, 137, 568,509, 19,625 and 588,228 hidden << same filter, tried again now. Eerily similar numbers again.

Opus can count the bytes on the drive, so I guess it isn't a bad block, or is that not enough of a test?

I have a pic if that helps?

This time I let it sit, until the numbers stopped ticking over, then the lister greyed out and, title bar saying NOT RESPONDING. After some minutes, it spat the dummy and reverted to the root of the 5TB, as if nothing had happened, although the lister was empty! (ie, it was pointed at the root of the drive, but nothing was showing.) I refreshed that lister, and all was well.

5

If it comes back eventually then it doesn't sound like it is crashing, but running very slowly.

Flat view and real-time filters aren't really designed for dealing with (literally) a million files, so you may be running into limitations there.

6

Well, if it came back eventually, I could just wait, but it doesn't, so looks like I have to do my workaround. I don't get why a million files give or take is an issue, then again, I don't write code. I could understand if RAM was an issue, or CPU, but I have 32 GB of RAM, and about a zillion CPUs, so that isn't it.

I guess I might have to say that this is a job Opus cannot handle. Oh well, thanks Leo.

So, is there another way to do this, barring me moving say 100 dirs at a time into temp dirs? Some sort of recursive wildcard copying thing that doesn't require me seeing and selecting the dirs.

7

Using Tools -> Find Files might work, depending on exactly what you're trying to do.

8

...and that's why they pay you the big bucks Leo. Works perfectly, and much faster too, though no surprise there I guess. I should have at least tried this. Oh well. Live and learn eh?

Thing is, why is it called "Find files" when it can find dirs too?

Anyway, problem solved circumvented Many thanks :slight_smile:

9

[quote="leo"]If it comes back eventually then it doesn't sound like it is crashing, but running very slowly.

Flat view and real-time filters aren't really designed for dealing with (literally) a million files, so you may be running into limitations there.[/quote]
My two cents:

Although the OP found a different solution for his problem, this does not fix the filtered flat view issue then. Flat view offers different functionality than "find files," and I think this might also impede large filtered sync operations.

I am a bit concerned about this in general. Having millions of files is not that unusual nowadays. I have a 5 TB HD and last time I performed a sync using Syncback, it reported >2 million files and that progream sync'ed them all fine and snappily. I don't use DOpus for such operations, because I just don't trust it with so many files. For example, on a first try, just refreshing the checkmark view before a large sync took forever in DOpus, while it only took a minute or too for a similar summary view in Syncback.

Thinking about it more, I feel like I should be able to trust my file manager easily with millions of files on any operation. For example, I use flat view and filters very frequently and soon might want to use it on folders that contain >1 million files. So I would hope this could be fixed in the future. At the very least it should not lock up but give some (slightly embarassing) message such as "too many files for filtering flat view".

10

Having 1 million files is common. Wanting to maintain a live view of all 1 million files at once in a big, actively updated, real-time filtered list that can detect any change to any of the million files and show it immediately is at least fairly uncommon and stretching what those parts of Opus were designed for. Different tools for different jobs.

Still, we do have plans to improve it in the future, both in terms of speed and filtering & related functionality.