I am copying huge amount of data from one drive to another drive for backup. Once the copy is over, I plan to compare the copied folders to see if each and every file matches.
For this,
At first level I will do Filename, Size, Date, Time comparison which should be pretty fast. Next, I plan to compare the copied folders including sub folders and files using the Byte comparison. It is here that I have few queries. I am afraid that Byte comparison will be too time consuming for around 3.5 TB of data.
I was reading in Directory Opus Manual about using MD5 for finding duplicate files. It has options like MD5 accuracy and also in another setting max_md5_file_size. This allows to calculate the MD5 for a fix percentage of file and if the MD5 matches for say 5% of each file, it can be presumed that the file is copied correctly. This is a kind of balance between calculating full MD5 or byte comparison which is time consuming and just plain comparison of filename, size, date, time.
But I do not see any MD5 in Synchronization related options. Is limited file actual content comparison available for Synchronization internally to Directory Opus based on the max_md5_file_size setting for Byte comparison also.
The max_md5_file_size only affects the behaviour of the MD5 column if you have it displayed in the file display. It lets you stop enormous files having their checksums calculated automatically.
The Synchronize tool does not have an MD5 mode; it only has a straight byte comparison option, which compares the full file byte-by-byte. There would be no real advantage to an MD5 option as the whole file has to be read anyway in order to calculate the checksum.
The option for MD5 accuracy you mention is only applicable to the Duplicate finder tool.
Thanks for the update. That means Byte comparison will be accurate but I have to be ready for slow comparison at the same time.
I will also add in feature request sometime but the same principle of comparing a small percentage of size using MD5 instead of full file Byte comparison will be useful if provided in Synchronization file compare also as here too the files are compared anyway so in addition to Byte comparison the MD5 option and Slider can be provided in Synchronization too.
I don't really see what the point of that would be to be honest.
The filesystem and your hard drives already use checksums to ensure data integrity. The only real way you can lose data when copying files is due to a hardware fault. But if you suspect a hardware fault or don't trust your hard drives then only checking a portion of the file data that you copy probably wouldn't be enough to prove that the copy was successful anyway.
In an ideal world, I agree. However, Software Industry (to which I belong) definitely does not fit in the ideal world definition as per me.
Hardware faults do happen and even without hardware faults, files do get corrupted at times. That is the primary reason why backups are required.
So, it just makes extra sense and time to do at least a one time detailed comparison for getting more comfortable and have better peace of mind for important backups.
I agree that it will be not enough to prove copy was 100% successful but it will be a midway balance between time taken for 100% content checking be it using Byte comparison or even with MD5 or other hashing algorithms and not checking any content and only relying on filename, size, date and time.
It is similar to random checks that are done in many industries but with every file instead of few files.
I will also go ahead with suggesting an idea that when something like part of file is checked to increase speed at cost of accuracy (e.g. in Duplicate File Finder with MD5 Accuracy slider kind of concept) while comparing files, the percentage of file to be checked should be randomly selected from anywhere in the file. It other words, it need not be the first part of the file always. (e.g. If I am comparing 5% of a file, it could be initial 5% for some files, last 5% for few others and 5% anywhere in the middle for remaining). Probability of file compare accuracy will increase this way without any additional time or resource usage for comparing.
If you're worried about hardware corrupting data, then you have to check all of the data. Nothing else really makes sense.
If the hardware fault was so large that it would be likely to show up when only checking a subset of the data, then it would probably reveal itself quite quickly without doing any checks. The drive would completely fail, or the OS or SMART would start reporting errors. (I'm assuming you're already using SMART monitoring if you're worried about this type of thing.)
OTOH, if it's only a small hardware error, say a corrupted cluster somewhere on the disk, then you'd only find it if you checked all of the data.
Also, if you only want to check some of the data, then just check some of the files/folders (completely) and ignore others. That check would be just as valid as checking all of the files only partially.
Agree with above. This too makes sense for partial data.
As for the complete checking of all files for full content, it is for peace of mind that the process provides although it is time consuming for huge data volume.