EverythingDopus - A utility to integrate Everything with Directory Opus

[Zoc]

EverythingDopus

An utility to integrate voidtools' Everything with Directory Opus

This utility was inspired by apocalypse's SearchEverything.

Introduction

voidtools' Everything is a search engine that locates files and folders instantly for Windows using NTFS USN Journal.
It can index 1 million files under a minute, uses a small memory footprint, and monitors and updates the index in real-time.

EverythingDopus is a bridge between both software, allowing the search results of Everything to be displayed as a collection inside Directory Opus.

List of features

• Written in C
• Blazing fast - on average a search takes around 45ms!
• Written as a Windows application (No sneaky consoles popping up when you run it)
• Does not require Administrator permissions to run
• Cleans up any file operations it needs to use (i.e., leaves no temp files behind)
• Bundled and easy to use button and scripts for Directory Opus
• Has a neat logo

How to install?

1. Extract the application package somewhere you're comfortable with (e.g. I use D:\Utils\EverythingDopus). Please avoid folder where you require Administrator permissions to run (i.e. avoid Program Files!)

2. Open Directory Opus and enable toolbar customization mode

3. Drag the file EverythingDopus.dcf to your toolbar:

4. Close the Customize dialog in Directory Opus.

5. Copy the file EverythingDopus.osp to /scripts. If you have trouble accessing that folder, that is just a shortcut for: %AppData%\GPSoftware\Directory Opus\Script AddIns

And you're done!

The first time you attempt to search using the new toolbar button, it will ask for the location of ed.exe, just choose it from the directory you previously extracted the files.

This utility also supports the custom /regex/ search implemented by apocalypse

Does it work with Everything 1.5 alpha?

Yes, as long as you only have this version installed and running.
To make it compatible, you need to disable the alpha instance (the feature that allows it to run side by side with 1.4). If you do this, it will behave exactly like 1.4, being the main instance running.

Be sure you only have 1.5a installed before doing this, or you will run into issues with multiple installed versions!

1) Start Everything 1.5a
2) Open Everything 1.5a and go to Tools -> Options
3) Select Advanced on the left pane, choose alpha_instance and set it to false:

4) Press OK and restart Everything 1.5a

Acknowledgements
This utility wouldn't exist without apocalypse's original implementation.
This utility also uses a script helper created by wowbagger to aid in selecting the executable when clicking the button.

[NotNull]

So, what should this button/script do? Adn what is the difference with the one it is based on?

(I'm new to Dopus, but familiar with Everything)

[Micky]

Greetings NotNull, nice to see you here, too

[NotNull]

Thank you!
Not sure if I will hang around here often, though.

But I do really like Dopus. It is the best I have seen so far (of the 40+ file managers I checked)

But this is all going way off-topic ...

[Zoc]

Hey @NotNull

I initially thought about submitting pull requests to improve SearchEverythingCoreCLI, but I quickly learned as not as experienced with C# when compared to C/C++, so I decided to build my own version in C (for fun, as it was initially C++) and address my concerns there.

I also warned to make this easy to setup and use for new users of both Directory Opus and Everything - so I added utilities to ease the learning curve.

The main differences are:

• Built in C
• Doesn't need admin permission to run with the default setup, while SearchEverythingCoreCLI default setup requests admin permissions
• Doesn't pop up a console window when using, not needing the @runmode:hide workaround
• Internal DopusRT calls already take care of the Go NEWTAB command, preventing de-sync
• On average a search with EverythingDopus takes around 45ms, while searches with SearchEverythingCoreCLI take on average 250ms. The worst times I got for each are 58ms (EverythingDopus) vs 441ms (SearchEverythingCoreCLI) - I didn't go through extensive testing though
• SearchEverythingCoreCLI seems to leave temp files after it's done running, while I made sure to cleanup and temp files created by EverythingDopus
• There's a drag and drop button ready to use bundled with the release file, as well a script and icons
• I added a custom icon that mixes the Directory Opus and Everything, so it's clear it's not a builtin function
• Has a simple help dialog when running it

I planned to update the first post with this info once it was approved, but I still haven't time to write it appropriately.
I want to make sure I make it clear overt there that this application wouldn't exist without apocalypse's work on SearchEverythingCLI, he did an excellent job there

Since you're experienced with Everything but not with Directory Opus, you would be an excellent test subject. If you do happen to give it a try, can I get some feedback on how was your experience?

[NotNull]

@Zoc

If you do happen to give it a try, can I get some feedback on how was your experience?

Sure. My experience:
(Tested with Everything 1.4.1.1022 (x64) / Directory Opus 12.30 (x64) @ Win11 22H2)

• I definitely needed the extended installation instructions on GitHub. Those were easy to follow.
• The Everything part worked without any issues. "Everything" I threw at it was processed as expected. And fast too!
• It is even Unicode-proof (I saw CLI somewhere and had my doubts. (codepages and such can give a lot of issues .. )
• Suggestion: File Collection name = Everything. Maybe change it to EverythingDopus? I already had an Everything collection with subcollections, which gave strange results. Had to delete those first.
• File collection is always opened in the upper/left pane. Independant of which pane is currently active. I did expect it to be opened in the inactive pane (not based on anything, btw)
• Suggestion: Clean up older Everything file collection tabs (they are empty)
• Empty search (= get all Everything files/folders) : Warning + assistance. Nice!
• Suggestion: Add Everything_GetTotResults() query before "downloading" reasults.
  If above a certain amount (let's say 1000), show a dialog "123456 results. Refine your search or continue anyway" or similar
  With ~300K objects (files/folders), Dopus/ DopusEverything uses ~1GB RAM
• After closing all Everything file collection tabs, not all memory was released (still using 360 MB ..)
  After deleting the file collection from the Directory Tree (which I don't have visible) that was reduced to ~90 MB. Dopus itself uses ~20 MB on my system.
• Suggestion: Instead of using complex OR constructions in the search query, an option to add to the existing file collection?
  That way you can build the collection out of multiple, simpler searches.
• Nice logo btw!

All in all: Well done!

Uninstalling is removing the button fromn the toolbar and deleting EverythingDopus.osp from the Sripts Addin fodler?

[Zoc]

Thank you for taking your time, testing and providing feedback!

@NotNull

I definitely needed the extended installation instructions on GitHub. Those were easy to follow.

Yeah, the instalation instructions on the main topic are indeed too simplistic. They follow the standard instalation procedures for any script or button, but aren't immediatelly clear - Even using dopus for 6+ years now I had forgotten how to do those when attempting to create this project

@NotNull

The Everything part worked without any issues. "Everything" I threw at it was processed as expected. And fast too!

Yay!

@NotNull

It is even Unicode-proof (I saw CLI somewhere and had my doubts. (codepages and such can give a lot of issues .. )

Yeah, to be honest using the whole TCHAR wrapper from Microsoft to support Unicode is a pain, but I made sure to use it so it supports "everything" you throw at it.
That was the main reason why I used C instead of C++ for this project (Other than the challenge, ofc)

@NotNull

Suggestion: File Collection name = Everything. Maybe change it to EverythingDopus? I already had an Everything collection with subcollections, which gave strange results. Had to delete those first.

Hmm I didn't consider that. I think it's a good change to have to avoid conflicts indeed

@NotNull

File collection is always opened in the upper/left pane. Independant of which pane is currently active. I did expect it to be opened in the inactive pane (not based on anything, btw)

Oh, this is a good change as well, change the internal call from /cmd to /acmd, targeting the active window.

@NotNull

Suggestion: Clean up older Everything file collection tabs (they are empty)

They should update to the contents of the Everything collection.

I'd like to have some sort of "Go to existing tab named , otherwise create new tab" command, but I don't think DopusRT.exe support that.
It's interesting that the documentation for commands doesn't include the command Go NEWTAB. Maybe there's a way I'm not aware of

Update: While searching more in the documentation, I found there is a Go NEWTAB findexisting. A quick test shows me that .\dopusrt.exe /acmd Go NEWTAB findexisting path=coll://Everything/ works, so I'll add the change later

@NotNull

Empty search (= get all Everything files/folders) : Warning + assistance. Nice!

@NotNull

Suggestion: Add Everything_GetTotResults() query before "downloading" reasults.
If above a certain amount (let's say 1000), show a dialog "123456 results. Refine your search or continue anyway" or similar

Ohhh I like this! I'll add a dialog with a sensible threshold (Though 1000 sounds good)
I guess it would be good to set the default operation (i.e., if the user just press enter) to stop the search

@NotNull

Suggestion: Instead of using complex OR constructions in the search query, an option to add to the existing file collection?
That way you can build the collection out of multiple, simpler searches.

I'm having some trouble to imagine how would be the the workflow for this in a way that doesn't conflict with the searches. Need to put some more thought on this hmmmm.

@NotNull

Suggestion: Instead of using complex OR constructions in the search query, an option to add to the existing file collection?
That way you can build the collection out of multiple, simpler searches.

Thank you <3

@NotNull

Uninstalling is removing the button fromn the toolbar and deleting EverythingDopus.osp from the Sripts Addin fodler?

There's no dependencies, so yeah
Remove the button, delete the script file (it's a zip file, if you want to peek on what's inside) and delete the application folder

@NotNull

After closing all Everything file collection tabs, not all memory was released (still using 360 MB ..)
After deleting the file collection from the Directory Tree (which I don't have visible) that was reduced to ~90 MB. Dopus itself uses ~20 MB on my system.

I left this for last, as it is the most complicated answer. I didn't notice this previously, but at the same time this is not related to the application I wrote, it should be related to how dopus handles collections internally.
I'm running Directory Opus 12.30 and Everyting 1.5.0 1337a

I did try the search string a - that way I'd get a silly amount of hits. In total this matches 1051025 objects on my system (about 60k directories, the rest are files)

Before running the search, initial Dopus memory: 15700KiB
Dopus memory used after opening the collection: 2234608KiB
After closing the collection: 1088376KiB
After deleting the collection: For some reason it instantly grew to 17983186 KiB - And kept growing by about 10mb / second. After about a minute it went back to around 1.1GiB memory use
I suspect the high memory usage is due to watermark system, but it's probably better to ask @Leo or @Jon for help here figuring out what's going on

@NotNull Again, thank you for all the tests and feedback!
I'll incorporate the changes above in the next few days

[Jon]

@Zoc

It's interesting that the documentation for commands doesn't include the command Go NEWTAB. Maybe there's a way I'm not aware of

That's just the list of commands that were new/changes in Opus 12 vs Opus 11.

The full internal command set is documented here.

[sfuller]

I'm getting this--

I'm running 1.5a, and have added alpha_instance=0 to the end of the ini, have tried restarting the Everything service and the application. I was having troulble saving the ini, so also tried storing settings in appdata.

[Zoc]

@sfuller From what I read in Everything forums, you need to modify Everything-1.5a.ini that's inside the program files folder.
To do that, you will need to open your editor (e.g., Notepad) as Admin, so you can open and save the file inside C:\Program Files\Everything (Or your equivalent installation path).

Once you have that file open with administrative privileges, add alpha_instance=0 under the section [Everything] (usually at the end of the file works, unless you have more sections in there), save it and restart Everything

[Zoc]

Thanks for the link @Jon

Do you happen to have any insight in the memory usage behavior we got?

[Leo]

@Zoc

Do you happen to have any insight in the memory usage behavior we got?

Collections stay in memory for at least a while, to avoid having to constantly re-load them. Sounds like normal behavior.

[WKen]

I tested other methods, and the latency is high. It seems that using the SDK will be faster. It would be great if DOpus could be integrated.

762-285=477
557-87=470
607-136=471
var WshShell = new ActiveXObject("WScript.Shell");
var DOpusFactory = DOpus.Create();
DOpus.Output(DOpusFactory.Date.ms)
var oExec = WshShell.Exec('"D:\\Program Files\\Everything\\es.exe" -dm -size -full-path-and-name -sort-date-modified test -export-txt D:\\test.txt');
while (oExec.Status == 0)
{
   DOpus.Output(DOpusFactory.Date.ms)
}
DOpus.Output("Exitcode1: " + oExec.ExitCode)


876-501=375
992-637=355
817-465=352
var WshShell = new ActiveXObject("WScript.Shell");
var DOpusFactory = DOpus.Create();
DOpus.Output(DOpusFactory.Date.ms)
var result = WshShell.Run('"D:\\Program Files\\Everything\\es.exe" -dm -size -full-path-and-name -sort-date-modified test -export-txt D:\\test.txt',0,true); // true == wait for process to finish
DOpus.Output(DOpusFactory.Date.ms)



975-630=345
797-352=445
519-61=458
var DOpusFactory = DOpus.Create();
var cmd = DOpusFactory.Command();
 cmd.addline("@runmode:hide")
 cmd.addline('@sync:"D:\\Program Files\\Everything\\es.exe" -dm -size -full-path-and-name -sort-date-modified test -export-txt D:\\test.txt')
DOpus.Output(DOpusFactory.Date.ms)
 cmd.run
DOpus.Output(DOpusFactory.Date.ms)



904-508=396
588-115=473
910-434=476
var DOpusFactory = DOpus.Create();
var cmd = DOpusFactory.Command();
 cmd.addline("@runmode:hide")
 cmd.addline('@sync:cmd.exe /s /c ""D:\\Program Files\\Everything\\es.exe" -dm -size -full-path-and-name -sort-date-modified test -export-txt D:\\test.txt"')
DOpus.Output(DOpusFactory.Date.ms)
 cmd.run
DOpus.Output(DOpusFactory.Date.ms)

[NotNull]

@sfuller

I'm running 1.5a, and have added alpha_instance=0 to the end of the ini, have tried restarting the Everything service and the application. I was having troulble saving the ini, so also tried storing settings in appdata.

First of all: the obvious.
Was Everything 1.5 running? Check the system tray (/notification area) to see if the Everything icon is shown.

Assuming Everything was indeed already running:

• Check if Everything is using an instance or not. A couple of ways to check:
  • Hover the systemtray icon.
    If it says Everything no instance is being used
    If it says Everything (something) an instance something is being used. Most likely instance: 1.5a
  • Check the titlebar of the Everything GUI
    If it says Everything 1.5.0.nnnn no instance is being used
    If it says Everything (something) 1.5.0.nnnn an instance something is being used.
  • Type or paste the follow in the Everything searchbar and press ENTER:
    /alpha_instance
    The statusbar will briefly show the current value of alpha_instance.

Everything has 2 Everything.ini files. One with system-level settings, one with user-level settings.
The first one - the one you need to edit for alpha_instance - is located in the same folder as the executable (Everything64.exe) and has a size of ~ 1KB.
The usersettings are in %APPDATA%\Everything\Everything-1.5a.ini (~ 95 KB)

As @Zoc already pointed out, :

• Exit Everything completely (right-click its systemtray icon and select Exit
• From the Start menu, search for: notepad
• Right click the Notepad app
• Select Run as administrator.
• Menu:File => Open "c:\Program Files\Everything 1.5a\Everything-1.5a.ini"
• Add alpha_instance=0 at the last line.
• Save and exit
• Start Everything
• Check if Everything now runs without instance.

[WKen]

I tested your plugin / script for latency, several times faster.
692-480=212
235-151=84
121-31=90
898-816=82
141-65=76
237-124=113

[sfuller]

@NotNull

alpha_instance=0

I think it had something to do with not originally not storing settings in appdata. Originally there was no alpha_instance entry--now there was one with a 1 instead of a 0--and I missed that. Working now!

[Zoc]

Thanks for the test, happy to know it performs better

[WKen]

@NotNull

It is even Unicode-proof (I saw CLI somewhere and had my doubts. (codepages and such can give a lot of issues .. )

There is indeed a problem, no name displayed, for example: ไทย

[Zoc]

Unicode is indeed supposed to be supported, not sure why it isn’t working

Once I have some time at my pc, I’ll try to create a file named “ไทยไทย.txt” and try to search for it

(This is a busy week, but I’ll try to do that as soon as possible)

[khalidhosain]

@Zoc

The first time you attempt to use it, it will ask for the location of ed.exe. Just select it from the path you previously extracted the application to.

hi Zoc. its asked for ed.exe as you told. But I have no idea where to locate for the ed.exe file?