Maybe only for inspiration, there is a script addin called CompareEx, which is able to select items based on MD5.
You were able to find it here (in the german DO support forum): https://www.haage-partner.de/forum/viewtopic.php?t=4956
But since the linked thread/homepage and forum is offline for some time, try this one to get the script addin at least: Want to Compare two listers/folders to check similarity?
Run it like so:
CompareEx TABS=by-md5
It won't work with archives I guess, did not try.
Not sure what size of files and archives you need this for, but if you go huge in number of files or file sizes/archive depth, scripting this thing is probably not optimal. Heavier lifting could be done by some specialized C++ code/executable which only reports back to logfile. Logfile content could be read by DO script and applied to current view (the selecting/deselecting part), separating logic/comparison and view by that.