I suspect I'm inquiring about a bug that has crept into Directory Opus (DO) recently. I say that because for years I have used DO to keep multiple NAS units in sync via FTP and haven't seen this issue until recently. At first, I thought it was a problem with a new NAS I bought to replace a unit that failed, but I've tracked that down as far as I can go with Synology support, and they've convinced me it's a problem with DO instead.
So let me explain. The following two screenshots show side by side directory listings of a single directory on one of my NAS systems, a Synology DS 1520+ if it matters. In each case, the left pane shows the contents of the folder via an SMB connected network drive (U:), whereas the right pane shows the contents of the same folder via an FTP connection instead. And to be more precise, that's an SFTP connection (though I see the same problem with regular FTP) with timestamp-adjustment features disabled, so it should simply be showing the file times as they are on the NAS. The following first image is taken from FileZilla, and it shows everything correctly, whereas the second image is taken from DO, which has the wrong timestamps for all but the first file listed in the folder.
I find that both incredibly weird and extremely frustrating as it makes it impossible to sync folders accurately. For the record, I've tried enabling the time-zone adjustment and daylight-savings-time adjustment in the DO FTP address book, but it makes no difference.
Some of the times have an hour difference but some don't have any difference. That most likely means the differences are in how times on the other side of a daylight saving change are being displayed. There is no "correct" convention there; either method is arguably correct (and indeed, Windows itself has changed what it does over the years).
Opus should consistently use the same method which your version of Windows uses, but the FTP server at the other end may be doing something in addition which confuses things, perhaps. Maybe Opus and FileZilla are using different commands to list the directory or something. Difficult to say without access to the server to look at the data it's sending.
I don't think anything related to this has changed in Opus for a long time.
Ok, but why should daylight savings have anything to do with it when I've disabled that in DO? Shouldn't it simply report the file times it gets from the FTP server? And if so, then shouldn't the details it displays match what FileZilla gets?
I don't see how it's possible for DO to "...use the same method which your version of Windows uses" because there's a known issue with the way Windows Explorer just shows midnight for every file when connected via FTP. I can reproduce that bug at the command line reliably. I don't know what DO is using, but I don't see how it could be using the same method as Windows.
Better yet, how can I help debug/fix this issue? It's been a problem for me only since standing up my new Synology NAS, and I can absolutely see why they want to disavow the issue and blame DO.
Opus only applies a single timezone adjustment to all files returned from the server; it doesn't do it per-file. If it gets it wrong, all files will be off by the same amount. So if you're seeing individual files with different offsets, I think that information can only be coming from the server.
So help me understand how only some files could have the offset applied while others do not? If DO gets only a single adjustment from the server and applies it to everything, then all of the files should be off by that amount, right? And beyond that, why would DO be applying the offset in the first place if I've disabled that option in the address book for the site? I'm so confused.
As I said, Opus only applies a single adjustment for the whole site.
The value it uses for the adjustment can come from the server (if it reports its timezone), or from the FTP addressbook entry, if you've turned on the option and selected a specific timezone.
The daylight savings time option in the addressbook entry applies to YOUR timezone, not the server's. If the option is turned on, and your local system currently has DST in effect, the extra hour will be added or subtracted from the adjustment value used for the site. It's not something that happens on a file-by-file basis.
Any variation in individual files must be coming from the server because Opus simply doesn't apply time adjustments on a per-file basis.
It might shed some light on things if you log into the FTP site using a command line FTP client, run the dir or list command, and see what the timestamps it returns are, and the differences (if any) relative to each other, as well as what the various software reports.
(That said, if you can access the NAS via SMB, why use FTP at all? SMB is generally far better, at least over a LAN.)
I'm not sure this really sheds any light on anything because the command-line client I have on Windows is the busted one that doesn't include file times. But here's what I get when I try it:
C:\Users\john\Downloads> sftp [site name redacted]
john@[site name redacted]'s password:
Connected to [site name redacted].
sftp> cd "/Media/Graphics/Comics/Bloom County"
sftp> ls -al
drwxrwxrwx 1 John users 522 May 14 2021 .
drwxrwxrwx 1 John users 31764 Jan 18 12:51 ..
-rwxrwxrwx 1 John users 15360 Jan 14 2020 Thumbs.db
-rwxrwxrwx 1 John users 149471 Mar 9 2018 blmc180221.jpg
-rwxrwxrwx 1 John users 167945 Mar 19 2019 blmc190314.jpg
-rwxrwxrwx 1 John users 92046 Mar 28 2019 blmc190328.jpg
-rwxrwxrwx 1 John users 58104 Jun 11 2019 blmc190531.jpg
-rwxrwxrwx 1 John users 67986 Jun 11 2019 blmc190611.jpg
-rwxrwxrwx 1 John users 73127 Jun 12 2019 blmc190612.jpg
-rwxrwxrwx 1 John users 72216 Jun 13 2019 blmc190613.jpg
-rwxrwxrwx 1 John users 153292 Jun 14 2019 blmc190614.jpg
-rwxrwxrwx 1 John users 110753 Jun 17 2019 blmc190615.jpg
-rwxrwxrwx 1 John users 46348 Jun 17 2019 blmc190616.jpg
-rwxrwxrwx 1 John users 187827 Jun 18 2019 blmc190618.jpg
-rwxrwxrwx 1 John users 179737 Jun 23 2019 blmc190623.jpg
-rwxrwxrwx 1 John users 89414 Jul 3 2019 blmc190703.jpg
-rwxrwxrwx 1 John users 48405 Aug 15 2019 blmc190815.jpg
-rwxrwxrwx 1 John users 142865 Aug 20 2019 blmc190820.jpg
-rwxrwxrwx 1 John users 100281 Nov 17 2019 blmc191114.jpg
-rwxrwxrwx 1 John users 64367 Jan 14 2020 blmc200111.jpg
-rwxrwxrwx 1 John users 166813 Apr 18 2020 blmc200403.jpg
sftp> exit
Does that help at all? I certainly don't see any file offset being provided that DO would apply to all the files as has been suggested is the issue.
The response comes from the server. The client does nothing but print what the server outputs.
You'll need to work out the command that gives you timestamps for that (S)FTP server.
With FTP I would try LIST, DIR, MLSD, MLST but with SFTP it seems to depend on the falvour of Linux at the other end, as the ls command's arguments are not standardised from what I can find.
(It may be worth trying all of them as well, in case some do timestamps differently to others. That in turn could explain differences between clients, if one is using one command and another using another.)
My bet is some of the timestamps are on the other side of a DST change and some are not.
But we still don't know if the difference is caused by Opus, the SFTP server, or a combination of things.
On further investigation, I may be wrong about that. It works that way with FTP but SFTP looks like it's different and it is up to the client how it displays the listing. There's probably a better command line client but I don't know which to recommend for this.
Nice! Keep me posted! I'm writing some sample programs in C# to query the servers myself to see what they're actually returning. I'll post if I find anything that might be helpful.
For what it's worth, the very simplest utility I could gin up to grab the file information from the server shows the file dates/times from FileZilla, not those displayed by DO. Here's the output of a simple program listing the contents of that same folder:
. has timestamp 5/14/2021 08:39:29
.. has timestamp 1/18/2022 12:51:14
blmc180221.jpg has timestamp 3/9/2018 16:28:53
blmc190314.jpg has timestamp 3/19/2019 05:47:30
blmc190328.jpg has timestamp 3/28/2019 07:27:24
blmc190531.jpg has timestamp 6/11/2019 09:51:19
blmc190611.jpg has timestamp 6/11/2019 09:50:36
blmc190612.jpg has timestamp 6/12/2019 09:10:25
blmc190613.jpg has timestamp 6/13/2019 09:27:22
blmc190614.jpg has timestamp 6/14/2019 09:12:28
blmc190615.jpg has timestamp 6/17/2019 09:26:04
blmc190616.jpg has timestamp 6/17/2019 09:26:19
blmc190618.jpg has timestamp 6/18/2019 09:06:01
blmc190623.jpg has timestamp 6/23/2019 14:44:35
blmc190703.jpg has timestamp 7/3/2019 08:11:16
blmc190815.jpg has timestamp 8/15/2019 09:30:18
blmc190820.jpg has timestamp 8/20/2019 10:05:21
blmc191114.jpg has timestamp 11/17/2019 09:58:12
blmc200111.jpg has timestamp 1/14/2020 09:38:30
blmc200403.jpg has timestamp 4/18/2020 10:51:55
Thumbs.db has timestamp 1/14/2020 22:30:51
And here's the source code, host and credentials redacted:
using System;
using Renci.SshNet;
// Got this sample from: https://ourcodeworld.com/articles/read/369/how-to-access-a-sftp-server-using-ssh-net-sync-and-async-with-c-in-winforms
namespace SshNetFtpGetListing
{
internal class Program
{
/// <summary>
/// List a remote directory in the console.
/// </summary>
private static void ListFiles()
{
string host = @"[host redacted]";
string username = "[user ID redacted]";
string password = @"[user PW redacted]";
string remoteDirectory = "/Media/Graphics/Comics/Bloom County";
using (SftpClient sftp = new SftpClient(host, username, password))
{
try
{
sftp.Connect();
var files = sftp.ListDirectory(remoteDirectory);
foreach (var file in files)
{
Console.WriteLine($"{file.Name} has timestamp {file.LastWriteTime}");
}
sftp.Disconnect();
}
catch (Exception e)
{
Console.WriteLine("An exception has been caught " + e.ToString());
}
}
}
static void Main(string[] args)
{
ListFiles();
Console.ReadKey();
}
}
}
This built and ran just fine in Visual Studio 2022 with nothing but the SSH.NET NuGet package installed if you'd like to monkey around with it.
BOOM! That did it! DO now shows the correct dates and times for all the folders I check. My sync operations are working again. Just out of curiosity, what did you change, if you're willing to share? Either way, thanks so much!
Glad it's working! Thanks for the info you provided, it made it easy to track it down.
It did turn out to be nothing to do with the time adjustments in our own code (i.e. the options in the addressbook) , but I found a call to _localtime64_s in the SFTP function that converts the incoming Unix timestamp to a Windows timestamp. Although the documentation for it is unclear, I suspected that it might be responsible.
It looks like that function takes daylight savings time of the source timestamp into account when converting to local time. The fix was as simple as changing that function to use _gmtime64_s and then doing the conversion to local time using the native Windows functions which ignore daylight savings.
I've been writing software for over forty years, and the number one thing I absolutely despise is dealing with time zones, date/time conversions, etc. It is literally the bane of my existence. I'm so glad I could help you find it and even happier to have the fix.
I'm afraid I have to come back to this topic again. I'm having two problems, one of which isn't related to DOpus but is evincing the second. The first problem is that I can no longer connect to my NAS via SFTP as I have been doing since this issue was resolved for me back in January. That's a separate issue I have to take up with I believe the Firewalla people as their recent update appears to have botched my ability to forward that port successfully.
So that leaves me with connecting via regular FTP, which still works, and that has all the time-stamp synchronization problems (and worse) that I was complaining about before. Is there any way to get reliable time stamp synchronization working on regular FTP without SFTP? Perhaps some variant of that same bug exists in the regular/non-SFTP code? Thanks in advance.