Filter by Column : Powerful filtering/searching tool for all columns

I usually focus on a specific column while filtering data. However, if the filter requires sub-filtering or other complex steps, I tend to use a similar filtering strategy for other folders. These multi-step processes can be time-consuming, so it would be beneficial to save them as favourites. Let's face it, humans are creatures of habit, and we tend to use well-rehearsed strategies repeatedly.

I can't think of a way to include that in the dialog (like Excel macros recorder?). When I developed it, I had in mind a more visual filtering experience, making decisions based on the results, so the query are not constant.

If you already know what you want to search for, you can create buttons that use the command with your query already defined.
(In beta 2, which I will release tomorrow or the day after, you can make multiple queries, similar to the Filter Bar with Evaluator, but with the syntax of this command. Eg. ext==jpg && created==thismonth)

Update v1.4.9 Beta 2 :

Needs at least DOpus v13.4.3

CHANGELOG

GENERAL :

  • Some changes in the code to support the new features.
  • Renamed flag Ignore non qualified items to Ignore undefined values.
  • Now you can check if a column has data or not, regardless of its type. For that simply use colname=="" or colname!="".
  • You can now compare one column with another (not available in Dialog Command mode). To do that use colnameoperator{othercolname}
    • In both cases, colname can be the column keyword or a custom name (or be ignored in implicit mode).
    • {othercolname} is a column keyword (only built-in ones are supported). Keywords are listed here.
  • Improved logic when you want to ignore unavailable (undefined) values. Should now work in all modes.
  • Other minor fixes.

COMMAND :

  • Redesigned DIALOG mode.

    • Major changes in data handling:

      • Values are now read only at the beginning, and kept in cache for future use. This may seem trivial, but it is a substantial improvement, as it avoids unnecessary reading (and consequent delay), so subsequent load times are greatly reduced, especially when using multiple files/values that take a long time to calculate.
      • The refresh button does not reread the data from the files (it only synchronizes the files in the tab with those in the dialog).
        • If you need to update the data (because of some change in the tab, like new items) you must refresh the tab, and the dialog will update the data (reread it from the files) automatically.
    • Ported FAYT syntax to the dialog's filter box

      • Allows to use the implicit FAYT mode when searching for matches in the dialog list
      • Cache data for quick comparison, so that unlike FAYT mode, the search is genuinely Find-As-You-Type
      • Regular expressions, complete words, case match and diacritics available (if the column is of text type)
    • Now includes built-in, Evaluator and Shell columns in the columns combobox.

    • Now has more options and uses icons instead of buttons.

    • Options to select all/invert/deselect list entries

    • Option to choose search type : FAYT Syntax or Filter as plain text

    • Option to use regular expressions, case match, whole word and diacritics (if applicable)

    • Option in configuration to change the method of custom columns obtention. By default use Rename Presets, which is a bit slower, but more consistent.

    • New hotkeys:

      • Alt + 1 to select all list entries.
      • Alt + 2 to invert selection.
      • Alt + 2 to deselect all list entries.
      • Escape to clear filter box content.
      • F9 to save the results to a collection and close the dialog.
      • Shift + F9 to save the results to a collection without closing the dialog.
    • The dialog now behaves better with tab events:

      • When switching to another tab, the dialog is disabled, and re-enabled when returning to the same tab.
      • Updates its data if the folder is read again.
      • It closes if the path is changed or the tab is closed.
    • You can edit the saved categories from the dialog by clicking on the pencil icon.

    • You can ignore empty values when searching by pressing the :no_entry_sign: button in the list header (similar to Ignore undefined values flag).

  • New argument : MULTI, allows to use multiple filters, referencing one or more columns, using logical operators && or ||. (Experimental, need more tests, be patient with this)

    • E.g. FilterbyColumn MULTI QUERY ext==exe && created>lastyear
    • In this mode, Evaluator and textual filters are used exclusively.
      • You must use parenthesis where applicable. To search for an operator or literal parenthesis, enclose the values in double quotes.
  • New argument : IGNOREEMPTY, similar to the Ignore undefined values flag. Can be combined with other arguments.

  • It should work now properly in Libraries as well (let me know if it doesn't)

TODO/IDEAS :

  • If you use the advanced dialog (via More Options), the options in the list are not updated to match the results. This is understandable, since the options were not supposed to be the necessary ones, but since it can cause some confusion I imagine that it should be updated as well... (done in beta 2)

  • Currently you get the values each time a column is accessed via the combobox. The ideal would be to do that once for all items (even if they are hidden) and then in subsequent calls, filter those not currently visible based on the values already calculated. As this implies more work on my part, it can be implemented in later versions (so use/change columns or refresh the dialog list when necessary, especially for custom columns that create a cache in an exotic way). (done in beta 2)

  • Implement timers for the three buttons at the bottom, this in order to avoid overloading Opus, since someone can think of pressing them continuously (not necessary in beta 2)

  • If the Lister is refreshed, update also the dialog ? (I'll have to see how complicated it would be to implement that). (done in beta 2)

  • Handle the Attributes column as multi value, with more friendly options (e.g. archive instead of a). (done in beta 1)

  • List Shell and Evaluator columns (done in beta 2)

  • Edit those categories saved from the dialog (done in beta 2)

  • Add more quick filters in More Options (I'm open to suggestions :blush:).

  • Translate missing strings to other languages in the dialog, for better immersion (you can help with this).

  • List Script columns (done in v1.6.0).

4 Likes

Update v1.4.9 Beta 3 :

GENERAL :

  • Improved filtering of nested items (either by expanded folders or grouped flat view) in all modes (except where Textual Filters are used, there is no possibility in there). Now the command is able to show matches in these cases, showing also the ancestor folders.

DIALOG :

  • Now you can use a number as value, referring to the number that occupies the column (zero based index) in the current Lister (eg. DIALOG=3).
  • Now if DIALOG is used without values, or with an unsupported column name, fallbacks to 'Name' column instead of closing.
  • When getting values for custom columns, now does not show an error for each item in case the column does not exist (very rare but possible).
  • Fixed an error in columns with multiple values always showing "no value".
  • Possible Workaround when trying to select nested items from the dialog. Now, to show them, it also shows their ancestor folders.
  • Improved use of 'AND' for multi-value columns.
  • Improved the case of 'Genres', that in despite of being a multi-value column, it has a separator different from the rest.

KNOW ISSUES :

  • The command works very randomly in Libraries when nested items are involved. Since the workarounds I have implemented do not always work 100%, I think this is the best that can be achieved in terms of compatibility at this point.
  • If there are expanded folders, sometimes, after hiding/showing items, not all items are shown again. This seems to be a problem with the Select command, not with this command alone.

(I have not received any reports of any errors of any kind, so I imagine that there are no others bugs, or that nobody uses the command :slightly_smiling_face: :neutral_face:).

3 Likes

There is much I do not know yet about your script, but I congratulate you on your achievement.
Filtering expanded folders is almost working now. I have found one issue, but as is posted above, you may already know of it.

To simplify the discussion I am using only two expanded folders.
If a value is selected that is in both folders all works well.
Selecting a value that is only in one folder after a value has been selected that is in both folders also works.

However, when a value is selected that is only in one folder and then selecting a value that is in both folders, does not display the shared value in the other folder.

If you press F9 or shift + F9, and the correct files are stored in the collection, the problem is related to the Select command, already reported here

Yes !
The correct files are stored in the collection.
It is the same problem.

Update v1.4.9 Beta 4

CHANGELOG

GENERAL:

  • Improved conversion and comparison of columns set as number, in all modes. Decimals are now correctly taken into account.

FAYT :

  • Fixed a problem when using Rename Presets, where sometimes it did not correctly display matching items if they were nested.

DIALOG :

  • If you delete a saved category, and this column is currently in use by the dialog, now it detects and updates the data automatically.
  • Now the dialog hopefully takes into account if new items have been added (including if a folder has been expanded) when updating the data (eg. by pressing F5 or the 'Refresh' button) (Previously only updating the tab caused this, and was not really useful, in the case, p.e. of having expanded a folder). Note that this doesn't respond for data changes. For that, you have to refresh the tab.
  • Fixed some cases when 'Use AND' checkbox was taken into account when it shouldn't.
  • Possible fix for some controls not displaying correctly, due to custom font settings.
  • Workaround for current issues with Select not unhiding nested items (in case this become the new behaviour)

For those who use this, the dialog should look similar to this:

Please, let me know if it does not look like this for you.

1 Like

v1.5.0 is out and I have updated the main post explaining the major changes.

Changes since beta 4:

  • Corrections when querying for metadata type in FAYT mode, it should now be faster.
  • Removed the workaround for nested items, as it is not necessary in v13.4.5.
  • Other minor corrections.
1 Like

Directory Opus 13.4.6 beta with v1.4.9 Beta 4 solves the problem I posted above.
It works great with at least two expanded folders as the problem was shown in my post with many screen shots above.
Sorry, v1.5.0 doesn't at least as of tonight.

Note: I experienced a massive power outage for a few days and am only looking again after a local infrastructure interruption.

Yes, I have experienced the same issue with 13.4.6. Beta 4 was using a workaround, which I removed in v1.5.0 because when I tried it with v13.4.5, it was working fine.

Whether or not tagged columns are supported when filtering (ps: tags are automatically assigned through tag assignment)

2024-04-07_105229

@David thanks. I've sent you an updated version, with some workarounds for nested files (plus some changes in the 'refresh' button). FWIW, as far as I remember, v1.5.0 did work correctly with DO 13.4.5. And it's no longer available for download either (from https://cdn.gpsoft.com.au/files/Opus13/Beta/DOpusInstall-13.4.5.Beta.exe), so I can't confirm that. But it should work again now even with 13.4.6.

AFAIK, Tags are displayed correctly, at least on my side.
(Don't know what automatically assigned means)
Update: What column are the one you're referring? Because his header doesn't look like 'Tags' in chinese.
Update 2: Ok, I manage to find that you mean 'Label' instead of 'Tags'. For labels, the dialogue only lists those explicitly set. In retrospective, perhaps it would be more useful if it listed all the labels (including "Status")

Is there a specific font value in preferences the dialog uses? The dialog text is a bit hard for me to read due to my bad vision.

To be honest, I thought the script dialogs automatically used the size and font set by the user in Preferences when not specified in the script editor. But it seems that's not the case. And there's no way to do it from the script editor either. However, I've found a way that does work. Would you mind if I send you a version for you to try out?

If you set fontsize="0" for the Dialog in the resource XML, it will use the Preferences dialog font size.

Currently, the dialog editor UI will mess this up if you then edit the dialog, and also won't let you specify zero in the editor itself (only in the XML), but we'll change change in 13.5.1.

Thanks Leo, I was doing that.

And not only when you actually edit the dialog, but also if you edit the script with an external editor, and then save the changes with the dialog editor. So I guess the only suitable option in the meantime, will be embed the entire dialog into the script as a string, correct?

(May I suggest also add a property to query this value from a script? So you can decide how to show things based on user font size for dialogs, e.g. replace icons with text because they don't scale based on font size)

I took the following two pictures in English language

2024-04-08_132700

Update v1.5.9 Beta 1 :

GENERAL:

  • Dialogs should now use the font name and size configured in Preferences / Colors and Fonts / Fonts > Dialogs.
  • Now is possible to show matches for nested items (either from an expanded folder or when grouped flatview is on) from within a library.
    • Note than show matches for nested items is only available when not using Textual Filters (when enable_wilcards==false and preferred_filter_mode==1)
    • Not available when using Multi Mode.
  • Changed command's label (because it was too large)

FAYT:

  • Multi mode can now be used via FAYT mode (by enabling the Multi Mode flag)

DIALOG:

  • Changes to 'Refresh' button in the dialog. Now always shows all items in the tab and synchronizes them with the dialog (before it was a bit confusing about which items to show and which not).
  • Checked values are remembered under certain circumstances (e.g. when changing/editing a category).
  • Columns that have the same header are now suffixed with their keyword (so overlapping columns are avoided and ALL built-in columns should be listed now)
  • All Labels (including non-explicit ones) are now listed.
  • Fixed filtering issues with nested items.
  • The dialog now is able to avoid continous call for the same tab.

For those who are actually using this on a daily basis, please try this new version and report if anything isn't working as it should.
(@David @JCDARK @evernessince you may want to update as well)

4 Likes