I'm having an issue with some files transferred via FTP. This is for my job. We were using a regular web host for these files. They are CSV files created by an external system. When I download them via DOpus FTP on the old host (bluehost.com, if I recall correctly), the end of line (EOL) for the files was just fine.
However, we also have an Amazon Web Services (AWS) account and it was recently decided we should consolidate all this stuff to save money. So, the files are now being uploaded to our AWS account. When I download the files as ASCII or Binary, the EOLs are always <CR><CR><LF> instead of just <CR><LF>.
Given the popularity of AWS, I thought it would be a good idea to get to the bottom of this as there might be more and more people that will have this issue.
Note: I tried FileZilla (which I had to install because I only use DOpus for FTP these past many years) and it also resulted in the same EOL issue.
@Leo and @Jon, I'm going to private message you the OXC file so you can log in and see if I'm doing something wrong. I have over 50 files I do on a weekly basis. Right now, I load them all into Notepad++ and search for \r\r\n and replace with \r\n. Many of the files are quite large and this can take a few minutes when using the "Replace in all open documents" feature.
Here's how it looks in Notepad++ with Show All Characters turned on (names were changed)...
That sounds like the problem is in the files themselves, especially if you get the same results using FileZilla.
Binary mode should always give you the file data exactly as it is on the server. That suggests that whatever is generating (or uploading) the files on the server has already doubled all the <CR> characters for some reason.
Note that when my company switched from bluehost to aws, this started occurring. The software generating the csv files is the same. In fact, I went in to that system and just changed the FTP information so it would upload them to aws instead of bluehost.
I'm suspecting it's more of an aws issue. It's possible there's a configuration setting that allows for ASCII transfers or disallows them. Maybe it's not being allowed at all and is forcing binary all the way through.
I'll ask my work colleagues if I can have access to do some administration of the aws account.
If it's forcing binary, what you get is what's on the server, without any conversion. So if that's messed up (there's no OS I know of that uses CR CR LF as a line ending), it's most likely how things are on the server (or something very strange is happening).
Are the CSV files generated on the server directly to disk, or are they being FTP'd to the server by something?
It could be a difference in the runtime that the CSV generation code is using compared to before. Maybe on the old system it had to use CR LF to get Windows-style EOLs, but now that second LF is being converted by the runtime to CR LF itself, resulting in CR CR LF.
(This is what some C Runtime functions like fprintf will do if used incorrectly, as an example. An explicit CR LF would be needed if the code was running on Unix/Linux but, atypically, wanted Windows-style EOLs in the files it generated, maybe for Excel. The same code running on Windows would end up outputting CR CR LF, because the runtime outputs the CR as-is and then automatically converts LF into CR LF. That might affect other runtimes or scripting environments, but I don't know enough about them.)
I was soooo confused when I saw it was CR CR LF. I don't think I've ever seen that in nearly 40 years of computing.
When I set up an automated report in the software, I provide FTP connection details (address, username, password, filename, and directory). It then uploads them via FTP once week. I download them.
I'm also going to email the software vendor. If aws is only allowing binary, why in the world is the software creating the files with such a strange EOL?
Thanks, Leo. I'll work on getting aws to fix the issue and/or the software vendor.
Does this mean all my sites are defaulting to Binary transfer mode? Note that I'm looking at just one site here. Also, I am noticing an oddity. If a site is secured, I can't click the Transfer Mode's Use defaults. If it's not a secured site, I can. In this screenshot, Use defaults is not checked, but I also can't click any of the three options. Kind of a cross-eyed way of looking at it. lol
It means they're using binary mode unless any of them override the defaults.
SFTP doesn't have ASCII/binary modes like FTP; it's always in binary mode.
I don't think your issue will be on the downloading client side of things, though. The file on the server is already wrong before you are downloading it, based on everything you've said so far.
Based on what you've said above, I think either the file is being generated incorrectly, or the extra lines are being added to the file when it is uploaded, not when it's downloaded.
