Everything Dupes is a script command for Directory Opus, that allows you to automate duplicate search, using the combined power of voidtools Everything and DOpus. Using the file database searching ability provided by Everything, coupled with the file property access capabilities provided by the Directory Opus scripting interface, you can now find duplicates of single or multiple files throughout your system in a matter of seconds!
You can search duplicates based on their size, extension, name, modification, creation, and many other extended properties, combined or separately, and you can specify the locations to search for them.
Its main features are:
- Duplicate search using the Everything database, then refine the search with DOpus Duplicate File Finder tool.
- Search by size, extension, base name, creation and modification date (either by itself or combined).
- Search by extended properties (according to compatibility, see list below) (It is recommended to use those properties indexed by Everything).
PLEASE READ BEFORE PROCEED
This command was designed and tested using Everything Alpha v1.5.0.13.70 or later.
HOW TO INSTALL
Download the file below. Then just double click on it or run Prefs SCRIPTINSTALL and select the downloaded file.
IMPORTANT: Needs at least v13.6
v1.6.1 : SearchDupesEV.opusscriptinstall
IMPORTANT: Needs at least v13.7.4
v1.7.0 Beta : SearchDupesEV.opusscriptinstall
USAGE
You can use Everything Dupes by simply invoking it with EVDupes.
By default (without using any value for PROPERTIES), the search is performed using the size and extension of the selected files.
Based on the type and number of items to be used with the command, there are multiple search modes, which are detailed below:
| CASE 1 | selected files only | Search for dupes GLOBALLY for the selected files. |
| CASE 2 | 1 or more selected files, plus 1 or more folders | Search for dupes for the selected files ONLY in the selected folders |
| CASE 3 | 1 or more selected folders | Search for dupes for all the files included ONLY in those folders |
COMMAND ARGUMENTS
The following arguments are available:
| Argument | Type | Value | Description |
|---|---|---|---|
| FILES | /K/M | "filepath1" "filepath2"... | Specifies the name of the file or files to use with this command. If you don't provide this argument the command operates on all selected files in the source Lister. (and destination, where applicable) |
| IN | /K/M | "dirpath1" "dirpath2"... | Specifies the name of the folder or folders to use with this command. If you don't provide this argument the command operates on all selected folders in the source Lister. (and destination, where applicable) |
| LOCAL | /S | Convert CASE 1 to CASE 2 using source path as the folder to search in. Do not combine with others args. | |
| NODODUPES | /S | Don't perform a Opus Duplicate Finder task, only search based on Everything results. | |
| READFOLDER | /S | Use folder content (convert CASE 2 and 3 to CASE 1) | |
| RECURSIVE | /S | In conjunction with READFOLDER, read folders recursively (for CASE 2 and 3) | |
| USEDEST | /S | Add destination items to all cases above | |
The arguments used to define what information to use for the search (modifiers) are:
| Argument | Type | Value | Description |
|---|---|---|---|
| MD5 | /O | Force use MD5 comparison when searching for dupes. | |
| no | Force disable MD5 comparison when searching for dupes. | ||
| PROPERTIES | /K | colname | Include file properties info in the search. See list below. |
List of supported properties:
| Name | Keyword |
| Documents:Authors | author |
| Documents:Category | category |
| Documents:Company | companyname |
| Documents:Copyright | copyright |
| Documents:Document created date | doccreateddate |
| Documents:Last Saved By | doclastsavedby |
| Documents:Last Saved Date | doclastsaveddate |
| Documents:Pages | pages |
| Documents:Subject | subject |
| Documents:Title | title |
| General:Name | name |
| General:Extension | ext |
| General:Size | size |
| General:Created Date | created |
| General:Modified Date | modified |
| General:Access Date | accesed |
| General:Attributes | attr |
| General:Location | path |
| General:Rating | rating |
| General:Keywords | keywords |
| Image:Altitude | altitude |
| Image:Aperture | apertureval |
| Image:Aspect Ratio | aspectratio |
| Image:Authors | author |
| Image:Bit Depth | picdepth |
| Image:Camera Make | cameramake |
| Image:Camera Model | cameramodel |
| Image:Contrast | contrast |
| Image:Creation software | software |
| Image:Date Digitized | datedigitized |
| Image:Date Taken | datetaken |
| Image:Digital Zoom | digitalzoom |
| Image:Dimensions | picsize |
| Image:Exposure Bias | exposurebias |
| Image:Exposure Program | exposureprogram |
| Image:Exposure Time | exposuretime |
| Image:Focal Length | focallength |
| Image:Focal Length (35mm) | 35mmfocallength |
| Image:Height | picheight |
| Image:Latitude | latitude |
| Image:Lens make | lensmake |
| Image:Lens model | lensmodel |
| Image:Longitude | longitude |
| Image:Metering Mode | meteringmode |
| Image:Resolution (X) | picresx |
| Image:Resolution (Y) | picresy |
| Image:Saturation | saturation |
| Image:Sharpness | sharpness |
| Image:Subject | subject |
| Image:Subject Distance | subjectdistance |
| Image:Title | title |
| Image:White Balance | whitebalance |
| Image:Width | picwidth |
| Movies:Artists | mp3artist |
| Movies:Aspect Ratio | aspectratio |
| Movies:Audio streams | audiocount |
| Movies:Composers | composers |
| Movies:Conductors | conductors |
| Movies:Dimensions | picsize |
| Movies:Directors | director |
| Movies:Duration | duration |
| Movies:Frame Rate | framerate |
| Movies:Genre | mp3genre |
| Movies:Height | picheight |
| Movies:Publisher | publisher |
| Movies:Release Date | releasedate |
| Movies:Subtitle count | subtitlecount |
| Movies:Title | mp3title |
| Movies:Video Bit Rate | datarate |
| Movies:Video Streams | videocount |
| Movies:Width | picwidth |
| Movies:Year | mp3year |
| Music:Album | mp3album |
| Music:Album Artist | mp3albumartist |
| Music:Artists | mp3artist |
| Music:Audio Bit Rate | mp3bitrate |
| Music:BPM | mp3bpm |
| Music:Composers | composers |
| Music:Conductors | conductors |
| Music:Duration | duration |
| Music:Genre | mp3genre |
| Music:Release date | releasedate |
| Music:Title | mp3title |
| Music:Track | mp3track |
| Music:Year | mp3year |
| Programs:Copyright | copyright |
| Programs:Module Description | moddesc |
| Programs:Module Version | modversion |
| Programs:Product Name | prodname |
| Programs:Product Version | prodversion |
Once the search with Everything is done, then proceed the search with DOpus Duplicate File Finder tool, from the previous results (if applicable, read the details in NOTES, below).
SCRIPT CONFIGURATION
In the Script Management window, select Everything Dupes and click in the gear icon. (or just run Prefs SCRIPTS=SearchDupesEV.js*)
| log level | Script Log | Logging level to be displayed. OFF to shows critical errors only. DEBUG to shows all the messages. STANDARD to only shows the most relevants. WARNING to shows error conditions that are of consideration. |
| collection name | Results Collection | Main collection name to use for results. |
| delete on startup | Results Collection | If enabled, all results collections will be deleted after start. It only affects collections created by this command. |
| create Everything collection | Search Options | If enabled, 2 sub-collections will be created, one with Everything results, and one with the DOpus dupes search results. If disabled, only one subcollection will be used for duplicates. Note that `NODODUPES` overrides this value and set it to True |
| max secs timeout | Search Options | Max seconds to wait for the Everything search results. |
| min items to show progress | Search Options | Minimum number of items used by the command to display progress dialogs. If READFOLDER is used, the dialogs are always displayed. |
| select for delete | Search Options | Set to True to show the dialog for dupes selection at the end. |
| use MD5 | Search Options | Use MD5 comparison in Opus dupe finder. Only applies when search includes name/size. |
NOTES
- Although DOpus officially supports Everything v1.4, to use this command you need v1.5.0.1370 or higher. You can grab the latest one from here and here are some information on how to make compatible v1.5 compatible with DOpus.
- If you do not specify a file or folder (or they are not valid), the command will try to use all files in source (or destination if
USEDESTis used). - If no property is specified for the search, the files size and extension are used.
- You can override
use MD5config value by specifying the argumentMD5. Note that MD5 comparison makes sense only when also size/name are used. - Everything results are always stored, if you use
NODODUPESargument. - If you encounter an endless Everything search, you can cancel the search by clicking on the 'Abort' button in the displayed progress dialog.
- Avoid such behavior by exclusively using indexed properties, or refine your query by specifying directories to search within.
- Since v13.5.1, you can search for duplicates in all the columns that support it.
- Just a heads-up, though: Opus currently only lets you compare using one column (except for Name and Size). But with Everything, you can use multiple columns, so you're getting all the results you need if you set
create Everything collectionto True.
- Just a heads-up, though: Opus currently only lets you compare using one column (except for Name and Size). But with Everything, you can use multiple columns, so you're getting all the results you need if you set
CHANGELOG
v1.7.0 (Jul 06, 2024) : SearchDupesEV.opusscriptinstall (11.6 KB)
- Improved recognition of whether Everything is running and if the version in use meets the minimum requirements (thanks to new implementations in DO).
- Everything can now auto-start to execute the command if
everything_autolaunchis set in Preferences/Miscellaneous/Advanced.- If auto-started by the command, the same command will close Everything upon completion.
- Enhanced the way progress dialogs work:
- Now, the Skip button bypasses listing files and folders directly and skips everything else when reading folders and file properties (since skipping individual files didn't make much sense given the speed).
- The Cancel button now cancels the command in all cases.
Complete changelog:
v1.6.1 (Jun 23, 2024) : SearchDupesEV.opusscriptinstall (11.5 KB)
- Added a workaround for when the user doesn't have enough privileges to obtain info using WMI (used to get Everything's path and version).
- Some corrections for cases when the search in Everything is so fast that the script couldn't detect results properly.
v1.6.0 (Apr 27, 2024) : SearchDupesEV.opusscriptinstall (11.3 KB)
- Only compatible with Everything v1.5.0.1370 or later. The command now detects the version of Everything being used and does not proceed if it's an older version.
- Nothing new here, just that now the version is identified due to potential syntax incompatibilities with the EV syntax used by the command in older versions.
- New argument:
NODODUPES, allows for an Everything search without a subsequent search for Opus duplicates.- Added due to popular demand, as the command alone generates quite useful queries for Everything.
- Improved the method indicating when an Everything search is in progress and allowing it to be stopped if it takes too long.
- Now a progress dialog is shown, along with the elapsed time and the maximum allowed configured in
max secs timeout.
- Now a progress dialog is shown, along with the elapsed time and the maximum allowed configured in
- If the property to use is empty for all used items, now it searches for those whose property value is empty, instead of omitting the property.
- Other minor changes.
v1.5.1 (Apr 20, 2024) : SearchDupesEV.opusscriptinstall (10.7 KB)
- Added extra protection when closing Everything results tab while waiting, resulting sometimes in an unexpected error.
- Now the script log displays as an 'Error' when Opus dupes finder can't be performed due to various reasons. Please note, these aren't errors per se, just a nudge to catch the user's eye on the log.
v1.5.0 (Apr 20, 2024) :
- Notable Upgrade: Some major improvements in how the command build queries for Everything. Now, we're avoiding duplicate values and following Voidtools' advice to make things run smoother.
- Removed some arguments:
NOUSESIZE,NOUSEEXT,USECREATED,USEMODIFIED,USENAME. Now, if you want to use a property, you have to specify it inPROPERTIES.- e.g.
PROPERTIES=size,picheight
- e.g.
- Updated some variable names in the Script Configuration. Added some, removed some. It's worth checking out to stay up-to-date.
- New argument:
MD5. You can override the value ofuse MD5in the Script Config to search for duplicates using Opus's MD5 comparison. UseMD5=noto turn it off. - Additionally, a progress dialog has been added to the command. Therefore, if you choose to interrupt the reading of folders/properties, you can do so, and the command will continue from where it left off.
- The command now (hopefully) will end if you close the Everything results tab. Just to prevent things from dragging on forever.
v1.0.2 (Jan 20, 2024) :
- Removed some extended properties that were not working.
- Added search by bit depth.
- Fixed search by rating.
- Overall code cleanup (should be faster, especially with various files).
- Minor changes in script configuration dialog.
v1.0.1 (Dec 16, 2023) :
- Added support for tags in extended information (needs at least Everything v1.5.0.1364a).
v1.0.0 (Dec 12, 2023) : Initial release
