DST Compensation [BUG]

EDIT: The original thread title was: "DST Compensation for FAT32 <-> UDF and NTFS <-> UDF, et al."
However, it has since been verified that the issue has nothing to do with any file system; it lies within DST Compensation itself. I have renamed the thread subject accordingly. Reading the entire thread should bring the reader up to speed. Furthermore, the reader is encouraged to just disregard the remainder of my first post here (which I have left unchanged from its original posting).—KRA 2006-04-14

Issue:
DST (Daylight Savings Time) Compensation doesn't support synchronizing between a FAT32 volume and a Universal Disk Format (UDF) volume.

SUBMITTED TO GPSoftware:

------------------------------------------------------------- Tracking Number: 330010015045 Support Issue: Program Suggestion Submission Date: 4/10/2006 4:00:40 AM -------------------------------------------------------------
Background:
The UDF file system is used primarily on optical media. However, many removable storage devices also utilize this file system. Directory Opus users that utilize programs which allow writing files directly to rewritable optical media may also be affected (e.g. DirectCD). For more information on UDF, see this Wikipedia entry:

http://en.wikipedia.org/wiki/Universal_Disk_Format

Related Settings:[ul][li] Synchronize Utility Pane > DST Compensation Option[/li][/ul]Related Documentation:
Note that while the current option is called DST Compensation, it is specific only to time differences between FAT/FAT32 and NTFS volumes, UDF was not included. When synchronizing files between a FAT/FAT32 volume and a removable storage drive that uses the UDF file system (e.g. an Iomega Rev Drive) DOpus will not ignore time differences of exactly one hour, regardless if this option is used or not.[quote="The authors of the DOpus V.2 Changes.pdf"]Miscellaneous changes and bug fixes for 8.2.1.0 (30/01/06)

...

• Added "DST Compensation" option to the Synchronize function. When enabled, time differences of exactly one hour are ignored to compensate for different handling of daylight savings time between FAT/FAT32 and NTFS drives.[/quote]

Steps to Recreate This Issue:[ol][li] Change system date to 5 minutes before a DST time change event that will automatically move the clock forward one hour.[/li]
[li] Copy a folder full of files from a FAT32 file system drive to a UDF file system device, where they did not exist before.[/li]
[li] Allow time to pass, so the system automatically triggers the DST time change event. This ensures that the minutes and seconds on the time stamps will still be synchronous, and that only the hour changes.[/li]
[li] Synchronize files (two-way) between the two folders, ensuring that the DST Compensation option is enabled.

DESIRED RESULTS: All files with time stamps exactly one hour different should be omitted from the operation, regardless of the file system used by the two volumes.

ACTUAL RESULTS: All the files on the UDF Device, even though they all have time stamps exactly one hour different and DST Compensation is enabled, will be identified as newer and included in the operation.

NOTE: A similar issue exists when the time change goes backward, except the perception of which file copy is newer changes.[/li][/ol]Rationale:
UDF is becoming a more popular file system format for the makers of removable storage drives. It is a reasonable File Management requirement to be able to synchronize files between any two volumes. It provides a huge productivity boost when two copies of a synchronized file are not interpreted as: newer, older, or different, based solely upon a difference of exactly one hour in their timestamps. Such files can usually be safely skipped during synchronization.

I don't understand why this wouldn't currently work. Opus doesn't care what filesystems you are using - it doesn't actually look to see if one is FAT and one is NTFS. It simply looks to see if the file dates are exactly an hour apart (actually, the tolerance on this is two seconds - so it checks if the difference between the two file dates is 3599, 3600 or 3601 seconds) and if so, treats the dates as identical.

If it works with FAT there's no reason I can think of that it wouldn't work with UDF.

Might it have something to do with Synchronizing between FAT32 and UDF? That is, there is no NTFS volume in the equation?

Every time change event (on or off DST) occurs, I have had to re-synchronize everything, with time-date stamps off by exactly on hour. Last Fall, I thought it was just me, so I wrote the issue down as something to watch and be sure I did right next time. I just went through forward DST change times last couple of weeks here in US. And I made sure to have DST Compensation on, same results not a single file skipped in all my watched folders.

Give me steps to try, and I do it. I also turn on the Ignore Seconds when comparing TimeStamp option, to no avail.

I noticed something recently where the DST compensation only worked one way around. (LD00750)

That is if I synced A to B it would ignore files with a 1 hour time difference and no other changes, but if I synced B to A all of those files were listed for synchronizing.

Not sure if there was more to it than that or if it's been fixed, but maybe you're seeing the same thing?

I have often thought that I have also encountered what Nudel mentions above. Since reading his post, I have thought some more about this issue, and I finally resolved to do a much more thorough (and painstakingly time consuming) analysis which should provide much more useful information. Ultimately, I resorted to brute-force testing. It is now very clear to me that the real issue is that DST Compensation does not correctly work in both directions. It just so happens that the way I was synchronizing ended up in a certain direction.

@Jon,

Here's my analysis. Please fix this in an upcoming release.

Test Setup
The tests use the following, except where explicitly noted otherwise:[ul][li] Related Directory Opus Preferences or Synchronize options:[ul][li] Show Seconds in time columns – ON[/li]
[li] Preserve the timestamps of copied files – ON[/li]
[li] DST Compensation – ON[/li][/ul][/li][li] Two-way synchronization operation.[/li]
[li] Two instances of the same Test Folder; one instance will be referred to as the Older instance, the other will be referred to as the Newer instance.[/li]
[li] Several files in the Newer instance are one hour to-the-second newer, than the timestamps of their counterparts in the Older instance.

DESIRED RESULTS: These files should never be included in the operation.
[/li]
[li] Several other files are otherwise legitimately different (aside from DST considerations) in each folder instance.

DESIRED RESULTS: These files should always be included in the operation.
[/li]
[li] Three volumes, each with a different file system:[ul][li] A local FAT32 volume[/li]
[li] A local NTFS volume[/li]
[li] A removable UDF Iomega Rev drive[/li][/ul][/li][li] Each volume above contains both an Older and Newer instance of the Test Folder. There are 6 copies total of the Test Folder. Three (3) copies are Older instances—one on each volume type—that are all exactly the same. The other three (3) copies are Newer instances—one on each volume type—that are also all exactly the same. The three Older Instances are different from the three Newer Instances, as described above. This is done so that Older-Newer instances between any two volume types (i.e. Older FAT32 to Newer NTFS, Newer FAT32 to Older NTFS, Older FAT32 to Newer UDF, etc) can be tested and the results of each possible combination verified.[/li][/ul]Test Iterations
Each test iteration should always include one Older instance, and one Newer instance. What should differ with each iteration is which volume (and what File System) these instances are located on. A test iteration is performed for each of the following possibilities:[ol][li] Newer FAT32 to Older FAT32[/li]
[li] Newer FAT32 to Older NTFS[/li]
[li] Newer FAT32 to Older UDF[/li]
[li] Newer NTFS to Older FAT32[/li]
[li] Newer NTFS to Older NTFS[/li]
[li] Newer NTFS to Older UDF[/li]
[li] Newer UDF to Older FAT32[/li]
[li] Newer UDF to Older NTFS[/li]
[li] Newer UDF to Older UDF[/li][/ol]Test Script
The test script below is performed exactly as listed, once for each Test Iteration.[ol][li] Make the Newer instance the Source.[/li]
[li] Turn the Ignore Seconds when comparing timestamps option – OFF[/li]
[li] Click on [Compare].[/li]
[li] Record Expected and Actual Results[/li]
[li] Make the Older Lister the Source.[/li]
[li] Click on [Compare].[/li]
[li] Record Expected and Actual Results[/li]
[li] Turn the Ignore Seconds when comparing timestamps option – ON[/li]
[li] Make the Newer instance the Source.[/li]
[li] Click on [Compare].[/li]
[li] Record Expected and Actual Results[/li]
[li] Make the Older Lister the Source.[/li]
[li] Click on [Compare].[/li]
[li] Record Expected and Actual Results[/li][/ol]Test Results
Yes, I actually went through that 14-step test script 9 different times, and recorded all the results. As a result, I can confirm that the results are identical, regardless of the volume type. Thus, the issue is not with UDF (or any other file system) as I originally thought. However, there is still an issue, but I now have a better idea of where it actually resides. Since all 9 iterations test results were identical, I will only publish one set of results.[ol][li] Make the Newer instance the Source.[/li]
[li] Turn the Ignore Seconds when comparing timestamps option – OFF[/li]
[li] Click on [Compare].[/li]
[li] Record Expected and Actual Results

EXPECTED RESULTS: Only files that do not exist in either location, or whose timestamps differ by more than one hour (to the second) should be included in the operation.[ul][li] Copy from Source (Newer) to Destination (Older): Files 5 Folders 1[/li]
[li] Copy from Destination (Older) to Source (Newer): Files 1 Folders 2[/li][/ul]
ACTUAL RESULTS: Actual Results equal Expected Results. In this case:[ul][li] Copy from Source (Newer) to Destination (Older): Files 5 Folders 1 – correct[/li]
[li] Copy from Destination (Older) to Source (Newer): Files 1 Folders 2 – correct[/li][/ul][/li]
[li] Make the Older Lister the Source.[/li]
[li] Click on [Compare].[/li]
[li] Record Expected and Actual Results

EXPECTED RESULTS: Only files that do not exist in either location, or whose timestamps differ by more than one hour (to the second) should be included in the operation.[ul][li] Copy from Source (Older) to Destination (Newer): Files 1 Folders 2[/li]
[li] Copy from Destination (Newer) to Source (Older): Files 5 Folders 1[/li][/ul]
ACTUAL RESULTS: Actual Results do not equal Expected Results.[ul][li] Copy from Source (Older) to Destination (Newer): Files 1 Folders 2 – correct[/li]
[li] Copy from Destination (Newer) to Source (Older): Files 33 Folders 4 – incorrect[/li][/ul][/li]
[li] Turn the Ignore Seconds when comparing timestamps option – ON[/li]
[li] Make the Newer instance the Source.[/li]
[li] Click on [Compare].[/li]
[li] Record Expected and Actual Results

EXPECTED RESULTS: Only files that do not exist in either location, or whose timestamps differ by more than one hour (to the second) should be included in the operation.[ul][li] Copy from Source (Newer) to Destination (Older): Files 5 Folders 1[/li]
[li] Copy from Destination (Older) to Source (Newer): Files 1 Folders 2[/li][/ul]
ACTUAL RESULTS: Actual Results equal Expected Results. In this case:[ul][li] Copy from Source (Newer) to Destination (Older): Files 5 Folders 1 – correct[/li]
[li] Copy from Destination (Older) to Source (Newer): Files 1 Folders 2 – correct[/li][/ul][/li]
[li] Make the Older Lister the Source.[/li]
[li] Click on [Compare].[/li]
[li] Record Expected and Actual Results

EXPECTED RESULTS: Only files that do not exist in either location, or whose timestamps differ by more than one hour (to the second) should be included in the operation.[ul][li] Copy from Source (Older) to Destination (Newer): Files 1 Folders 2[/li]
[li] Copy from Destination (Newer) to Source (Older): Files 5 Folders 1[/li][/ul]
ACTUAL RESULTS: Actual Results equal Expected Results.[ul][li] Copy from Source (Older) to Destination (Newer): Files 1 Folders 2 – correct[/li]
[li] Copy from Destination (Newer) to Source (Older): Files 5 Folders 1 – correct[/li][/ul][/li][/ol]
Conclusion
I inspected and verified that each file's timestamp to-the-second. I knew which should be included which should not have been included. The number of files and folders to be copied should have been the same for steps number: 4, 7, 11, and 14. All four compare operations should have reported the same numbers of files and folders to be copied. The only variance should have been switching in which direction they should have been copied.

The issue is in DST Compensation, specifically in the logic branch that does not use the Ignore Seconds when comparing timestamps option. More specifically, the issue occurs only when the Source folder contains the file copy that is older by exactly one hour.

One could say: "just turn on Ignore Seconds when comparing timestamps." In my opinion, that is a separate option and should not be related to this issue. The basis for my assertion is that without that option Compare actually works in one direction. Therefore, Compare should work in the reverse direction without that option. Also, I would argue that DST Compensation should work irrespective of Ignore Seconds. If I have a file that is legitimately a few seconds off (not becuase of DST, but becuase of an actual edit), It should overwrite its counterpart, but one that is exactly an hour? No, that is DST. And these two cases need to be handled differently.

Ok, I've been able to reproduce this. It's a bug caused by having "ignore seconds" turned off. If that is on, it works fine in both directions. If it's turned off, it fails in one of the directions.

We'll fix this in the next version, which hopefully will be out before the next DST-related time change :slight_smile:

Thanks for your help!

Glad all that testing and typing paid off! I'm glad the issue is out in the open now. Thanks Jon!