I keep experiencing incomplete copy operations using Directory Opus on a site. I thought at first this was due to too many listing requests for counting files and sizes, but apparently this is not the case. To further strengthening my case I tried the same operation with FileZilla, and Filezilla has happily completed the operation in a single pass. No warnings, no errors.
I'd like to keep using Opus for FTP but it looks like it's not too reliable for dealing with high file count. At least not to me, or my settings, or my site. It looks like my IP gets blacklisted or something with Opus and not with FileZilla.
For example the site in question has 1.34GB, 18.780 files and 2.517 folders.
Do you have some suggestions? I kept pretty much default configuration both in Opus and FileZilla. I'd really like to get to the bottom of this of this issue.
Since we got no timestamps, here is a snap where the incident has happened:
14052583 bytes received in 0.547 seconds (25087.912 Kbytes/sec)
226 Transfer complete
--> PASV
227 Entering Passive Mode (...).
FD_CONNECT - WSAETIMEDOUT: Connection timed out
--> PORT ...
FD_CLOSE - WSAECONNABORTED: Software caused connection abort
Connection closed
Timeout on Command 1003.
I've got the entire log, but I'd rather not have it publicly exposed. I can provide you directly with full user/pass and log if you wanna take a deeper look? It's an interesting case really. Looks like some Opus actions are incurring temporary ban, as opposed to FileZilla which runs smoothly.
I just talked to the provider guys, forcing them to investigate what they see on their side.
Yesterday I managed to copy like 600MB, whilst today on a test before breaking down like 130MB...
Today's snippet:
226 Transfer complete
--> PORT ...
200 PORT command successful
--> RETR Seminar-24-of-107-300x300.jpg
425 Unable to build data connection: Connection timed out
--> PORT ...
FD_CLOSE - WSAECONNABORTED: Software caused connection abort
Connection closed
Timeout on Command 1003.
It seems odd since it's working one moment and then starts failing commands the next, without any new control connections being made (which is usually what causes FTP servers to start blocking clients; new non-control, data connections are normal for each file transfer).
You might need to ask them what their site's rules are for blocking things. Presumably they have logs which report this.
It could also be a stateful firewall somewhere, deciding it doesn't like the activity (in combination with the process name, possibly) for some reason, and blocking things for a while.
I have asked them at least 5 times by now and they could have never found my IP blacklisted in their logs, which is apparently the case, as the web interface is also intermittently down for like 15 minutes after the incident occurs. The FTP comes back online in next couple of hours. Nevertheless, I still keep pushing them for some answer, they should have the information on what has caused this.
But the strangest thing this is only happening with Opus. How come it does not happen with FileZilla? I just don't get it.
It could also be something your router is doing, maybe, triggered by some difference in what Opus and FileZilla do. It could be a lot of things, of course, but that's one that could cause the router to get confused about the server for a while, without anything being done on the actual server side.
If your router has any kind of FTP ALG (Application-level gateway - Wikipedia), I've found they are often really badly written, at least on consumer routers. And if you're using FTP in passive mode (which it looks like you are) then they aren't needed. I always turn any ALG options off on my routers, as I've found they cause more problems than they solve, and they aren't really needed these days when most protocols people use have adapted, or been replaced, to handle NAT (NAT traversal - Wikipedia) themselves.
Firewalls on the PC or in the router could also be doing the same if they see some pattern as an attack and shut down connections to the server, without the server side being part of the problem.
This has been happening on two entirely different routers. I never noticed an FTP ALG option on my previous Sercom router. I'm now using TP-Link AX1500 (the provider has enabled bridge mode to me) and I just turned off the options there. I rebooted the router and I still can't connect.
I shall retry when I got unbanned and let you know how it will go.
Looks like we are moving somewhere. I just turned off the FTP ALG and made sure to use PASSIVE mode, and now I am at least not getting auto-banned.
Now, Opus is just facing timeouts on accessing the files.
Moving on like below, on each file in queue:
--> TYPE I
200 Type set to I
--> PASV
227 Entering Passive Mode (213,202,100,28,197,222).
--> RETR masonry.php
550 masonry.php: No such file or directory
** CloseFile NO FD_CLOSE Response from Server **
Timeout on Command 1020.
Connection closed
Opus doesn't rely on an ALG (unless you turn off passive mode, which won't work in general if you're behind NAT, without an ALG; that's how FTP works, and not specific to Opus).
If turning the ALG on seems to fix things, that's pretty weird. ALG usually causes problems, rather than solving them (other than the combination of non-passive FTP and NAT).
I tried dynamic with ALG FTP off and it didn't work, as you have said.
Nonetheless, it's now better when I turned ALG FTP off even with passive mode, since I'm not getting banned anymore, but Opus still can't finish simple copy operation. That's super weird, since Filezilla had no problem whatsoever.
I'd like to keep using Opus FTP, but it apparently has some problem, at least with this site.
It doesn't sound like the problem is really happening in Opus, since you're also unable to access the same server via HTTP when the problem starts happening. Nothing Opus does should even be capable of affecting HTTP traffic to and from a server.
Something is breaking routing between you and the server, probably at one end of the other.
I've given some guesses as to what that might be, but ultimately it is something only you and the server admins can work out, since we can't see your network or theirs.
My understanding is that when incident occurs I'm temporary banned for like 15 mins on HTTP and for couple of hours on FTP. I'm unable to access the site in question only, not any of the other internet sites.
Another thing is that the incident is only occurring by using Opus, not with FileZilla. Of course, further digging should be required.
Anyhow - I'll abstain from using Opus for backup purposes on this site. Not really a solution, but as long as it works.
If the provider guys should find something, I'll make sure to share.
Just ran some more tests. I can't explain this, but there are some issues with FTP and the clients. These below are fresh tests. The fact is that FileZilla was the only client so far, that hasn't failed not even once (with my server), whilst both Opus and another popular client: WinSCP have both ran into connection issues, and were unable to complete site copy operation.
