I have created Filters using Find/Advances in the Utility Panel. Those filters work well, even when the folder I choose and its subfolders contains over 70,000 files, but I do not want results to be in a Collection because I cant move the files from there and preserve folder structure.
I tried using 'Select by Pattern' > Advanced in the Edit menu, but if I select any of the filters I created, and use it on the same folder (with over 70,000 files), the program freezes and 'Not Responding' is displayed:
I tested using smaller and smaller folders, and I always get the 'Not Responding' message unless I'm working with under 10,000 files, and even then it only works some of the time.
So I tried another method: using commands. The commands do work on folders with smaller quantities of files, but, like 'Select by Pattern' > Advanced, it freezes with larger numbers of files.
Is this a bug? Is there another method I could use without the results being in a Collection?
Does Task Manager show CPU or disk usage while this is happening, or is nothing going on?
If it needs to read metadata out of 70,000 files then it may take some time to apply. (Something we could probably improve, if it's just that. I suspect it was designed before filters could do that and hasn't been revisited since then.)
I use 3 monitors. If I move DOpus to a side monitor and use 'Select by Pattern', and do not let my mouse touch that screen, I still see this after 3 or more hours:
If I click on the screen, I get the 'Not Responding' message right way.
I will do 'Select by Pattern' now, and take screenshots:
I See 'Not Responding' in task manager, but not not here:
If I use Find > Advanced, it does not freeze and there are indications that it is working:
Using this method, it took 12 minutes to make a collection of all photos without an Exif DateTimeOriginal (10, 351 files). I find that 12 minutes to be quite acceptable.
Obviously the filter works, but not when applying it using 'Select by Pattern'.
I have saved a few different filters, and they all work well in 'Find' but not when using 'Select by Pattern'
Are you using an Exiftool column in your filter? Then this might be the culprit. The script log will tell you, how busy Exiftool is.
If you haven't, try out the newest version of the script. It checks on a per-file basis if the metadata needs to be extracted. This can make a huge difference, especially when handling collections.
Thank you but I do not believe it is the culprit. I have tried several different filters and there is no detectable difference in performance. One of the filters I tired does use an ExifTool Column.
The filter using the ExifTool column works well in the Utility Panel - Find > Advanced.
I see. It any case, you would see the delay only once. When the cache is ready, the columns should be displayed just as fast as any other, if not faster.
Could you make some process snapshots and send them to us? We should be able to get an idea of where it's freezing from those. Since it's taking a very long time, it might make sense to wait 15 seconds or so between making each snapshot, so we can see if it progresses from one file to another.
Here's how to make them:
With that filter, you may be able to speed things up by reversing the two lines, so it checks the file type group first, then checks the image description second. That would do the fast check first, and avoid trying to calculate the descriptions of any non-image files.
I did more testing, and things seem to work better if the metadata field I am selecting/searching is NOT displayed. However, even when it does eventually work, 'not responding' is often displayed and there is no indication that anything is happening.
I agree with the UI principal that the user should be able to see an indication that something is happening--as one does when using Find > Advanced.
The DMP files show Opus is always waiting for the ExifTool column script, which in turn is waiting on ExifTool to run and return a result.
That doesn't match the filter shown above where it was matching on the internal Image Description property, as that shouldn't involve a script column. A script and external tool being involved complicates things a lot more, but in any case the bottleneck is in waiting for the information to come back from the external tool, since that is what is running in each of the snapshots.
(The UI could definitely handle this situation better, but wasn't really designed for filtering operations which take that long to calculate.)
Leo, Thank you. I'd like to send you more dump files - this time I'll use a Filter that does not involve the ExifTool Custom Columns script, because I am getting the same results with other Filters.
Did I send you the correct information? I meant to send you only the last page of the document. The last page documents the test I did when I created the Dump files.
I just tested again with a filter that does not involve any ExifTool Custom Columns fields. I have sent an email with a my notes and a link to the dump files.
I do not mind that it took 7 minutes.
I do mind that it appears to not be working:
and that there is no indication that there is a process running.
When I use the same filter in the Utility Panel - ‘Find Files’ there are 3 indicators that the process is running:
Using 'Select by Pattern' in the Edit menu is so helpful - especially since results are not in a collection.
I hope there will be an improvement soon.
If I use 'Find' in the Utility panel, it works and I can see progress indicators, but there are restrictions to what I can do when results are in a collection.
When I use the same filter in 'Select by Pattern' (advanced), I almost always have problem- unless I am working with very few files.
I can't open a new lister.
I can't move the app to another screen.
There is not progress indicator.
If I touch anything with my mouse, I have to re-start DOpus.
