I am migrating data to a new backup hard drive and wish to reduce unnecessary files being transferred across.
The data consists of:
- Size: 2.16 TB
- Files: 2.64 million
- Folders: 371,000
- File Types: Windows Executables (.exe), Word documents (*.docx), Adobe PDFs and text (.txt) files
I have tried running a duplicate file search (using the MD5 checksum matching option) on the 4 main folders which contain the above data, Opus Version 64-bit 12.19.6 Beta became unstable since almost 500,000 files were found totaling over 900 GB. RAM usage by Opus reached over 5 GB. The duplicate file search in-progress dialog box disappeared while the file list continued to grow and could not be scrolled through or sorted.
While I could perform a duplicate search on each of the 4 main folders individually, there is an overlap between them and thus searching all of them would give the best result.
Should I reduce the files returned by limiting them to 5 MB and above? Any other suggestions to effectively process and reduce the size of this data would be much appreciated.
Thanks in advance for your time.
My system specification:
- CPU: Intel Core i9 Extreme 7980XE (18 cores, 36 threads)
- RAM: 64 GB DDR4 (4 modules in quad channel mode)
- Hard Disk (with the duplicate files): 5 TB WD Black (7200 RPM, 128 MB Buffer)
- OS Disk: 800 GB Corsair Neutron NX500 PCI Express (NVMe) SSD
- OS: Windows 10 Pro for Workstations 64-bit Version 1909 (Build 18363.693)