FTP timestamp wrong sometimes

I suspect I'm inquiring about a bug that has crept into Directory Opus (DO) recently. I say that because for years I have used DO to keep multiple NAS units in sync via FTP and haven't seen this issue until recently. At first, I thought it was a problem with a new NAS I bought to replace a unit that failed, but I've tracked that down as far as I can go with Synology support, and they've convinced me it's a problem with DO instead.

So let me explain. The following two screenshots show side by side directory listings of a single directory on one of my NAS systems, a Synology DS 1520+ if it matters. In each case, the left pane shows the contents of the folder via an SMB connected network drive (U:), whereas the right pane shows the contents of the same folder via an FTP connection instead. And to be more precise, that's an SFTP connection (though I see the same problem with regular FTP) with timestamp-adjustment features disabled, so it should simply be showing the file times as they are on the NAS. The following first image is taken from FileZilla, and it shows everything correctly, whereas the second image is taken from DO, which has the wrong timestamps for all but the first file listed in the folder.

I find that both incredibly weird and extremely frustrating as it makes it impossible to sync folders accurately. For the record, I've tried enabling the time-zone adjustment and daylight-savings-time adjustment in the DO FTP address book, but it makes no difference.

Help?!

Some of the times have an hour difference but some don't have any difference. That most likely means the differences are in how times on the other side of a daylight saving change are being displayed. There is no "correct" convention there; either method is arguably correct (and indeed, Windows itself has changed what it does over the years).

Opus should consistently use the same method which your version of Windows uses, but the FTP server at the other end may be doing something in addition which confuses things, perhaps. Maybe Opus and FileZilla are using different commands to list the directory or something. Difficult to say without access to the server to look at the data it's sending.

I don't think anything related to this has changed in Opus for a long time.

Ok, but why should daylight savings have anything to do with it when I've disabled that in DO? Shouldn't it simply report the file times it gets from the FTP server? And if so, then shouldn't the details it displays match what FileZilla gets?

I don't see how it's possible for DO to "...use the same method which your version of Windows uses" because there's a known issue with the way Windows Explorer just shows midnight for every file when connected via FTP. I can reproduce that bug at the command line reliably. I don't know what DO is using, but I don't see how it could be using the same method as Windows.

Better yet, how can I help debug/fix this issue? It's been a problem for me only since standing up my new Synology NAS, and I can absolutely see why they want to disavow the issue and blame DO.

Opus only applies a single timezone adjustment to all files returned from the server; it doesn't do it per-file. If it gets it wrong, all files will be off by the same amount. So if you're seeing individual files with different offsets, I think that information can only be coming from the server.

So help me understand how only some files could have the offset applied while others do not? If DO gets only a single adjustment from the server and applies it to everything, then all of the files should be off by that amount, right? And beyond that, why would DO be applying the offset in the first place if I've disabled that option in the address book for the site? I'm so confused.

As I said, Opus only applies a single adjustment for the whole site.

The value it uses for the adjustment can come from the server (if it reports its timezone), or from the FTP addressbook entry, if you've turned on the option and selected a specific timezone.

The daylight savings time option in the addressbook entry applies to YOUR timezone, not the server's. If the option is turned on, and your local system currently has DST in effect, the extra hour will be added or subtracted from the adjustment value used for the site. It's not something that happens on a file-by-file basis.

Any variation in individual files must be coming from the server because Opus simply doesn't apply time adjustments on a per-file basis.

1 Like

It might shed some light on things if you log into the FTP site using a command line FTP client, run the dir or list command, and see what the timestamps it returns are, and the differences (if any) relative to each other, as well as what the various software reports.

(That said, if you can access the NAS via SMB, why use FTP at all? SMB is generally far better, at least over a LAN.)

As I said, Opus only applies a single adjustment for the whole site.

If that's true, then how is the adjustment affecting only some of the files? That doesn't make sense.

I'm not sure this really sheds any light on anything because the command-line client I have on Windows is the busted one that doesn't include file times. But here's what I get when I try it:

C:\Users\john\Downloads> sftp [site name redacted]
john@[site name redacted]'s password:
Connected to [site name redacted].
sftp> cd "/Media/Graphics/Comics/Bloom County"
sftp> ls -al
drwxrwxrwx    1 John     users         522 May 14  2021 .
drwxrwxrwx    1 John     users       31764 Jan 18 12:51 ..
-rwxrwxrwx    1 John     users       15360 Jan 14  2020 Thumbs.db
-rwxrwxrwx    1 John     users      149471 Mar  9  2018 blmc180221.jpg
-rwxrwxrwx    1 John     users      167945 Mar 19  2019 blmc190314.jpg
-rwxrwxrwx    1 John     users       92046 Mar 28  2019 blmc190328.jpg
-rwxrwxrwx    1 John     users       58104 Jun 11  2019 blmc190531.jpg
-rwxrwxrwx    1 John     users       67986 Jun 11  2019 blmc190611.jpg
-rwxrwxrwx    1 John     users       73127 Jun 12  2019 blmc190612.jpg
-rwxrwxrwx    1 John     users       72216 Jun 13  2019 blmc190613.jpg
-rwxrwxrwx    1 John     users      153292 Jun 14  2019 blmc190614.jpg
-rwxrwxrwx    1 John     users      110753 Jun 17  2019 blmc190615.jpg
-rwxrwxrwx    1 John     users       46348 Jun 17  2019 blmc190616.jpg
-rwxrwxrwx    1 John     users      187827 Jun 18  2019 blmc190618.jpg
-rwxrwxrwx    1 John     users      179737 Jun 23  2019 blmc190623.jpg
-rwxrwxrwx    1 John     users       89414 Jul  3  2019 blmc190703.jpg
-rwxrwxrwx    1 John     users       48405 Aug 15  2019 blmc190815.jpg
-rwxrwxrwx    1 John     users      142865 Aug 20  2019 blmc190820.jpg
-rwxrwxrwx    1 John     users      100281 Nov 17  2019 blmc191114.jpg
-rwxrwxrwx    1 John     users       64367 Jan 14  2020 blmc200111.jpg
-rwxrwxrwx    1 John     users      166813 Apr 18  2020 blmc200403.jpg
sftp> exit

Does that help at all? I certainly don't see any file offset being provided that DO would apply to all the files as has been suggested is the issue.

The response comes from the server. The client does nothing but print what the server outputs.

You'll need to work out the command that gives you timestamps for that (S)FTP server.

With FTP I would try LIST, DIR, MLSD, MLST but with SFTP it seems to depend on the falvour of Linux at the other end, as the ls command's arguments are not standardised from what I can find.

(It may be worth trying all of them as well, in case some do timestamps differently to others. That in turn could explain differences between clients, if one is using one command and another using another.)

My bet is some of the timestamps are on the other side of a DST change and some are not.

But we still don't know if the difference is caused by Opus, the SFTP server, or a combination of things.

On further investigation, I may be wrong about that. It works that way with FTP but SFTP looks like it's different and it is up to the client how it displays the listing. There's probably a better command line client but I don't know which to recommend for this.

I'm investigating this at the moment, and we may have found something that could be causing this. Stand by!

Nice! Keep me posted! I'm writing some sample programs in C# to query the servers myself to see what they're actually returning. I'll post if I find anything that might be helpful.

For what it's worth, the very simplest utility I could gin up to grab the file information from the server shows the file dates/times from FileZilla, not those displayed by DO. Here's the output of a simple program listing the contents of that same folder:

. has timestamp 5/14/2021 08:39:29
.. has timestamp 1/18/2022 12:51:14
blmc180221.jpg has timestamp 3/9/2018 16:28:53
blmc190314.jpg has timestamp 3/19/2019 05:47:30
blmc190328.jpg has timestamp 3/28/2019 07:27:24
blmc190531.jpg has timestamp 6/11/2019 09:51:19
blmc190611.jpg has timestamp 6/11/2019 09:50:36
blmc190612.jpg has timestamp 6/12/2019 09:10:25
blmc190613.jpg has timestamp 6/13/2019 09:27:22
blmc190614.jpg has timestamp 6/14/2019 09:12:28
blmc190615.jpg has timestamp 6/17/2019 09:26:04
blmc190616.jpg has timestamp 6/17/2019 09:26:19
blmc190618.jpg has timestamp 6/18/2019 09:06:01
blmc190623.jpg has timestamp 6/23/2019 14:44:35
blmc190703.jpg has timestamp 7/3/2019 08:11:16
blmc190815.jpg has timestamp 8/15/2019 09:30:18
blmc190820.jpg has timestamp 8/20/2019 10:05:21
blmc191114.jpg has timestamp 11/17/2019 09:58:12
blmc200111.jpg has timestamp 1/14/2020 09:38:30
blmc200403.jpg has timestamp 4/18/2020 10:51:55
Thumbs.db has timestamp 1/14/2020 22:30:51

And here's the source code, host and credentials redacted:

using System;
using Renci.SshNet;

// Got this sample from: https://ourcodeworld.com/articles/read/369/how-to-access-a-sftp-server-using-ssh-net-sync-and-async-with-c-in-winforms

namespace SshNetFtpGetListing
{
    internal class Program
    {
        /// <summary>
        /// List a remote directory in the console.
        /// </summary>
        private static void ListFiles()
        {
            string host = @"[host redacted]";
            string username = "[user ID redacted]";
            string password = @"[user PW redacted]";

            string remoteDirectory = "/Media/Graphics/Comics/Bloom County";

            using (SftpClient sftp = new SftpClient(host, username, password))
            {
                try
                {
                    sftp.Connect();

                    var files = sftp.ListDirectory(remoteDirectory);

                    foreach (var file in files)
                    {
                        Console.WriteLine($"{file.Name} has timestamp {file.LastWriteTime}");
                    }

                    sftp.Disconnect();
                }
                catch (Exception e)
                {
                    Console.WriteLine("An exception has been caught " + e.ToString());
                }
            }
        }

        static void Main(string[] args)
        {
            ListFiles();

            Console.ReadKey();
        }
    }
}

This built and ran just fine in Visual Studio 2022 with nothing but the SSH.NET NuGet package installed if you'd like to monkey around with it.

We've put out a beta with a change which will hopefully improve this. Please give it a try and let us know.

BOOM! That did it! DO now shows the correct dates and times for all the folders I check. My sync operations are working again. Just out of curiosity, what did you change, if you're willing to share? Either way, thanks so much!

Glad it's working! Thanks for the info you provided, it made it easy to track it down.

It did turn out to be nothing to do with the time adjustments in our own code (i.e. the options in the addressbook) , but I found a call to _localtime64_s in the SFTP function that converts the incoming Unix timestamp to a Windows timestamp. Although the documentation for it is unclear, I suspected that it might be responsible.

It looks like that function takes daylight savings time of the source timestamp into account when converting to local time. The fix was as simple as changing that function to use _gmtime64_s and then doing the conversion to local time using the native Windows functions which ignore daylight savings.

2 Likes

I've been writing software for over forty years, and the number one thing I absolutely despise is dealing with time zones, date/time conversions, etc. It is literally the bane of my existence. I'm so glad I could help you find it and even happier to have the fix.

3 Likes

I'm afraid I have to come back to this topic again. I'm having two problems, one of which isn't related to DOpus but is evincing the second. The first problem is that I can no longer connect to my NAS via SFTP as I have been doing since this issue was resolved for me back in January. That's a separate issue I have to take up with I believe the Firewalla people as their recent update appears to have botched my ability to forward that port successfully.

So that leaves me with connecting via regular FTP, which still works, and that has all the time-stamp synchronization problems (and worse) that I was complaining about before. Is there any way to get reliable time stamp synchronization working on regular FTP without SFTP? Perhaps some variant of that same bug exists in the regular/non-SFTP code? Thanks in advance.