EVColumnizer Script Add-in : Feedback needed

EVColumnizer : Everything-powered values as custom columns

:warning: EXPERIMENTAL
This script is provided for testing and feedback only. It's not a final release and may contain (serious/funny) bugs.

What it does

EVColumnizer is a Directory Opus add-in that deepens Opus's integration with Everything. It lets you surface any Everything column value inside Opus as if it were a native Opus column.

EVColumnizer 0.1.opusscriptinstall (28.8 KB)

Features

  • Built with speed in mind.
  • Read any value from Everything inside Directory Opus. Performance is best when those values are indexed.
  • Configure everything from an easy UI: pick which Everything columns you want to expose in Opus. Enable only the columns you need.
  • Fine-grained column customization, including which file types each column applies to. (see note 1)
  • Preview data using a sample file/folder.

Requirements

  • Everything 1.5 or newer, with Opus integration already enabled.
  • Everything CLI (es.exe) 1.1.0.34 or newer.
  • Directory Opus 13.19.6 or newer.

Recommended to set everything_autolaunch in Opus settings.

es.exe can be placed in that same path or configured from the script's Settings dialog.

How it works

The script uses Everything's CLI and tries to minimize the number of calls required to fetch values (see Current limitations for details).
To start using it, after install, just click in Configure in the Script Management window.

Usage

Usage is similar (if not equal) to my other script add-in : MediaInfo++.

Notes

  • Columns are divided into three categories: indexed, fast non-indexed, and regular.

    • Indexed: columns Everything always keeps indexed (for example: folder file counts, custom columns, native Everything properties).
    • Fast non-indexed: columns not indexed by default but that return values very quickly, the difference versus indexed is tiny.
  • For speed, indexed and fast columns are fetched in groups: the script requests values for the whole folder in as few calls as possible and asks for all enabled columns of the same category at once. This drastically speeds up retrieval.

  • Regular columns are fetched by running es.exe per item per column (see Current limitations for why).

  • An in-memory cache is used to avoid re-requesting values that were just calculated. The cache is volatile and tied to each Tab.

Note 1: Indexed columns cannot be restricted by file type because they're retrieved for everything at once rather than filtered per type.
Note 2: To reduce query length, the script requests the data list using parent: with the parent folder (valid only for filesystem files). Ideally it would use filelist: to target only related files, but parent: is used as a workaround because of Everything's query-size limits.

Current limitations

  • ES currently has a bug where long queries (>255 characters, per void's explanation) fail and return no data. This release adds workarounds. void is aware of the issue and we expect a fix in a future release; the script will be updated accordingly.
  • For now the script only works on the regular filesystem (no Collections, etc.) because workarounds would otherwise require running es.exe per item, which defeats the script's speed goals.
  • Some columns visible in Everything's GUI may not return values via es.exe. Please report any such cases.

Technical notes / feedback

  • There isn't a reliable programmatic way to tell whether a column is indexed (beyond common ones like size, etc.). Some properties can be indexed only for specific file types or paths. Here, "indexed" means properties Everything maintains without user intervention. If a column is indexed in Everything, its values should be faster to retrieve.
  • "Fast" means the value can be retrieved quicker and indicates those columns are fetched in batches rather than individually.
  • "Indexed" columns are hardcoded, so you can't choose specific filetypes.
  • Folders-only columns can't choose specific filetypes.
  • The in-memory cache is a workaround to allow batch retrievals and avoid recalculating values (for example, after expanding a column). A disk cache was avoided to keep things simpler. The cache is cleared when the tab is refreshed (unless you hold Ctrl+Shift during refresh), when you change folder, or when the tab is closed. Opinions on whether the memory trade-off is worth the speed are welcome.
  • Everything returns raw values, so some column types may not make sense here (I'm considering dropping date/time columns). Some values come back in hexadecimal; a few require hardcoded conversions to be human-friendly (attributes, etc.). Ultimately the user chooses the data type for each column.
  • You can preserve the cache across a tab refresh by holding Ctrl+Shift while refreshing. The cache is always cleared when changing folders or closing the tab.

About handling "regular" columns

I haven't settled on the best approach for regular columns. Options I'm weighing:

  1. One call for multiple files per column (current approach).

    • Faster overall: the initial call does most of the work and each subsequent item takes ~2–3 ms. Files are pre-filtered and non-related files are skipped.
    • Drawback: data becomes available only after the batch completes, which can be bad for very large sets.
  2. One call per file for multiple columns.

    • Useful if the user enables many regular columns, but still suffers from the ES call delay (albeit reduced). Slower than the first option.
  3. Multiple files with an item limit (e.g. 50).

    • This was the plan for ALL the columns, but ES 1.1.0.34's query-size limit breaks it. I'll revisit this in future releases.

If you want to help

If you're interested, try the script and report edge cases (columns that return nothing when it should, weird raw values, or performance issues) or any suggestion you might have. Feedback will guide the next improvements.

Any suggestions or test results are appreciated. :wink:

(Main icon from All SVG Icons)

4 Likes

How does one set the es.exe path in the dialog?

Entering the path, either via browse or cut/paste, places the path in the Fullpath to es.exe field. However, EVColumnizer does not offer a clear way to save the path.

Once done, where do the selected columns in the dialog appear in the column right-click menu?

The last value entered is saved automatically after closing the dialog. Alternatively, the script will look for es.exe in the same folder from which Everything auto-starts if configured in Opus, as a fallback.

Once the dialog is closed, the columns appear listed in the categories where you placed them. (Same as Mediainfo++).

The script dialog is not saving parameters for me. I load this:

After entering the info and closing the script, nothing changes.

Everything 1.5a is running in the background. The latest es.exe is set. DOpus is set to auto-launch Everything.

The column I selected is not available in any column. When reopening the script dialog, the info is not there.

The ES localization thing really does look like a bug. A pretty funny one, actually. Fixed. Thanks!

Assuming you meant "is not available in any Category"...
Have you checked the Size and Count category? Since that's where you put it. :slightly_smiling_face:

You can test v0.2 with all the fixes related to the ES path in the dialog. FWIW, es.exe itself is NOT required when setting up the columns. It's only needed for the preview/actual value retrieval.

BTW, it looks like this script isn't something many people are interested in, even though I'd expect the opposite, since in theory it lets you pull property values much faster when an index is involved. So v0.2 might be the "latest" version for a while (or at all :upside_down_face:)...

v0.2-test: Download from here

  • Fixed ES path not being saved.
  • Fixed runtime detection of ES path existence in config dialog.
  • Other minor fixed for the dialog.

(I'm testing moving all my releases to Github, so from now on, all future releases can be found there)

2 Likes

Both the es.exe location saves and I am now able to see the activated column in the appropriate category, thanks!

Do these columns have keywords?

Also, I am trying to take the Folder Depth and Max Child Depth columns and add the values together. Not having luck there, yet, but will keep at it.

v.0.3.0-test:
(Needs ES v1.1.0.35 or greater)

  • Huge speed improvement! :smiling_face_with_sunglasses:
  • Now it should work with other non-filesystem path types (collections, libraries, etc.).
  • Added a CONFIG argument to launch the configuration dialog via commands.
  • Preview redesigned. Now you can stop the task instantly, since the value retrieval runs in the background.
  • Workaround to show values for name-frequency, ext-frequency and size-frequency in the preview.
  • Skipped the preview for certain properties (e.g. checksums) since they could potentially take a long time to calculate. (They can still be used in columns; that's up to the user).
  • Other bugs fixed.

A technical note (and a rant :upside_down_face:):

When a "column" it's used for filters/searches, there's no concept of Tab or other related items. That means es.exe gets run once per item for each column, and you also can't access the value cache.

About the cache, I might be able to change the context, though that could bring new problems. Since this script doesn't seem to be widely used (other than by me :slightly_smiling_face:), I'd prefer using Everything filters, etc., to run searches instead of relying on the column values directly. Another option is creating "satellite" Evaluator columns, there you could access the cache.

In any case, I hope this script has shown the current limits of columns via scripting (and other related concepts), and that even though JScript is pretty old, it's not as slow as people assume and shouldn't be avoided for that reason. You can run benchmark tests with this columns if you don't believe me.

And I hope that in the not-too-distant future a native way to create multi-columns for multiple items at once is considered :wink:, since this script demonstrates (I hope) that the time difference is huge.

This script signs off here. It may be updated via GitHub, so if you use it you can follow updates there.

There won’t always be a tab. Filters can be evaluated in situations where there aren’t any tabs open.

Wouldn’t you want the cache to work across tabs and windows anyway?

I thought about that at first, but I see some drawbacks. Keep in mind the cache's purpose here is just to avoid reprocessing items that were handled in the immediately previous operation (e.g. when expanding a folder), not to avoid recalculation in every context (as I said, this cache is very volatile). A script-scoped cache could work, but then, IMO, it becomes unnecessarily complicated to figure out and perform a per-item cache flush. A Tab-scoped cache is much simpler and more practical, though it has downsides. So:

:folded_hands:

I notice the columns do not have keywords. Maybe those are DO-specific.

That said, I have continued, and failed, to add the Folder Depth and Max Child Depth columns together in an Evaluator column.

How can this be done here?

I don't think that's possible or even valid for any column...

Again have you checked the Size and Count category?

I suggest not changing the category to anything other than "Script" without understanding what that means.

Thanks for responding. Both the Folder Depth and Max Child Depth columns are available in Columns > Scripts > EVColumnizer and can be used as such.

If there's no way to add their values via an Evaluator column then I will let it go.

Thanks, again!


Note that sometimes you might want to check if the columns are set first.

1 Like

THANK YOU!

All this time, I was trying combinations of

Val("scp:folder_depth") + Val("scp:max_child_depth")

instead of

Val("scp:EVColumnizer/folder_depth") + Val("scp:EVColumnizer/max_child_depth")

Once again, you taught me something. Thanks Cris!

1 Like