FTP not workng correctly - data loss

LITWINCZUK · March 11, 2011, 9:52am

I cannot copy marked file. KSIPWWW~2.
On server side its a 108KB file .
When i trying to copy to my comp. i get only 64 KB.

I tried:

copy other files - as you see, they all copied without problems.
get this file via ftp from command-line in other server = OK
change filename to 232323 = still only 64 KB.
swich Active/Passive modes = without success.

I enabled DEBUG mode on FTP:

227 Entering Passive Mode (xxxxxxxx,134,209) 150 Opening BINARY mode data connection for KSIPWWW~OC4J_KSIPWWW~KSIPWWW~2.log (111548 bytes). 65536 bytes received in 0.93 seconds (687.512 Kbytes/sec) 226 Transfer complete. Opening Connection xxxxxxxxxx:21 220 xxxxxxxx FTP server (Revision 1.1 Version wuftpd-2.6.1(PHNE_36065) Mon Apr 14 06:16:29 GMT 2008) ready. 331 Password required for root. 230 User root logged in. 215 UNIX Type: L8 500 'FEAT': command not understood. 200 Type set to I. 257 "/root" is current directory. 250 CWD command successful. 257 "/u01/oracle/product/AS1013/opmn/logs" is current directory. 200 Type set to A. 200 PORT command successful. 150 Opening ASCII mode data connection for /usr/bin/ls. 1620 bytes received in 0.27 seconds (58.608 Kbytes/sec) 226 Transfer complete. 227 Entering Passive Mode (xxxxxxxxxxxx,79,36) 150 Opening BINARY mode data connection for KSIPWWW~OC4J_KSIPWWW~KSIPWWW~2.log (111548 bytes). 65536 bytes received in 0.110 seconds (581.56 Kbytes/sec) 226 Transfer complete. 200 PORT command successful. 150 Opening ASCII mode data connection for /usr/bin/ls. 1686 bytes received in 0.27 seconds (60.560 Kbytes/sec) 226 Transfer complete. 227 Entering Passive Mode (xxxxxxxxxxxxxx,237,92) 150 Opening BINARY mode data connection for 232323 (111548 bytes). 65536 bytes received in 0.109 seconds (586.936 Kbytes/sec) 226 Transfer complete. 227 Entering Passive Mode (xxxxxxxxxxxx,31,232) 150 Opening BINARY mode data connection for 232323 (111548 bytes). 65536 bytes received in 0.94 seconds (680.680 Kbytes/sec) 226 Transfer complete. Connection closed

What wrong is with this file?
Opus: 9.5.3.0.3728.x86

LITWINCZUK · March 11, 2011, 10:00am

Tested with WinSCP - ftp working correctly in them.
This file has been copied.

Leo · March 11, 2011, 10:35am

[ul][li]Does this happen for other files on that server, or just that one? Is it only files in that logs directory or every file you try?

[/li]
[li]I think 64KB is the default buffer size used by Opus for FTP transfers, so there may be a connection there.

See if changing the buffer size either fixes the problem or results in the new size being the amount that is transferred. That may shed light on what's happening.

The buffer size can be set under Settings - Preferences - Miscellaneous - Advanced: ftp_copy_buffer_size. The size is in bytes and 65536 (64KB) is the maximum (anything higher will be ignored), so try a lower value like 32768 (32KB).[/li][/ul]

LITWINCZUK · March 11, 2011, 2:43pm

Before i changed it, i have a 0 (ZERO) here. I never changed it before.

Receive this file. i copied in on OS into new file with other name (original also exists)

Opening Connection xxxxxxxx:21 220 xxxxxxxx FTP server (Revision 1.1 Version wuftpd-2.6.1(PHNE_36065) Mon Apr 14 06:16:29 GMT 2008) ready. 331 Password required for root. 230 User root logged in. 215 UNIX Type: L8 500 'FEAT': command not understood. 200 Type set to I. 257 "/root" is current directory. 250 CWD command successful. 257 "/u01/oracle/product/AS1013/opmn/logs" is current directory. 200 Type set to A. 200 PORT command successful. 150 Opening ASCII mode data connection for /usr/bin/ls. 1694 bytes received in 0.27 seconds (60.560 Kbytes/sec) 226 Transfer complete. 200 Type set to I. 200 PORT command successful. 150 Opening BINARY mode data connection for ML_nie_kasowac (111548 bytes). 65536 bytes received in 0.109 seconds (586.936 Kbytes/sec) 226 Transfer complete.

ftp_copy_buffer_size = 32 kb. Restarted Dopus.

Opening Connection xxxxxxxxxx:21 220 xxxxxxxx FTP server (Revision 1.1 Version wuftpd-2.6.1(PHNE_36065) Mon Apr 14 06:16:29 GMT 2008) ready. 331 Password required for root. 230 User root logged in. 215 UNIX Type: L8 500 'FEAT': command not understood. 200 Type set to I. 257 "/root" is current directory. 250 CWD command successful. 257 "/u01/oracle/product/AS1013/opmn/logs" is current directory. 200 Type set to A. 200 PORT command successful. 150 Opening ASCII mode data connection for /usr/bin/ls. 1694 bytes received in 0.27 seconds (60.560 Kbytes/sec) 226 Transfer complete. 200 Type set to I. 200 PORT command successful. 150 Opening BINARY mode data connection for ML_nie_kasowac (111548 bytes). 65536 bytes received in 0.94 seconds (680.680 Kbytes/sec) 226 Transfer complete.

Other file? Log with receive a 445 KB file (opmn.log). File has been successfuly copied!
Look at "32768 bytes received in 0.110 seconds"

Opening Connection xxxxxxxx:21 220 xxxxxxxx FTP server (Revision 1.1 Version wuftpd-2.6.1(PHNE_36065) Mon Apr 14 06:16:29 GMT 2008) ready. 331 Password required for root. 230 User root logged in. 215 UNIX Type: L8 500 'FEAT': command not understood. 200 Type set to I. 257 "/root" is current directory. 250 CWD command successful. 257 "/u01/oracle/product/AS1013/opmn/logs" is current directory. 200 Type set to A. 200 PORT command successful. 150 Opening ASCII mode data connection for /usr/bin/ls. 1694 bytes received in 0.27 seconds (60.560 Kbytes/sec) 226 Transfer complete. 200 Type set to I. 200 PORT command successful. 150 Opening BINARY mode data connection for opmn.log (456457 bytes). 226 Transfer complete. 32768 bytes received in 0.110 seconds (290.40 Kbytes/sec)

LITWINCZUK · March 11, 2011, 2:48pm

I have several different servers. I many times copied files like that one from them.
This one is a HP-UX B.11.23

Strange is, that i can get them by:

WinSCP
from other linux machine via ftp commandline
i renamed this file - still not working
i copied it to other file (232323 in first post, now its other file ML_nie_kasowac).

I think DOpus react on specified data inside this file.
and copy them to other path.

LITWINCZUK · March 11, 2011, 3:26pm

Tested:
copied file to other Unix machine with similiar configuration.
Other path, other filename, same problem...

LITWINCZUK · March 11, 2011, 3:30pm

Copied file to HP-UX B.11.31 - downloaded ok.

LITWINCZUK · March 11, 2011, 3:34pm

I don't know where problem is located... why large file is copied ok, but this one cannot be received?

LITWINCZUK · March 11, 2011, 3:40pm

HP-UX 11.11 also ok
HP-UX 11.00 ... copied 96 KB

LITWINCZUK · March 11, 2011, 3:42pm

cp ML_nie_kasowac ML_nie_kasowac2
cat ML_nie_kasowac >>ML_niekasowac2

217 KB received OK. Ehhhhhhhh....

Wang · March 11, 2011, 3:47pm

Did you compare the partially copied files with the original files (i.e. using beyond compare). Maybe it's some weird character within the file that prevents DOpus from copying.

...just a thought.

LITWINCZUK · March 11, 2011, 3:55pm

SOLVED !!!!!!!!

File is copied successfully.
I turned-off in FTP:

Speed -> Limit download Speed Maximum Speed 50 kB/s

this is problem.....

LITWINCZUK · March 11, 2011, 4:04pm

Yeah, 40, 500, 9999
1 = ok. But... file copy is immediate (1 sec) - what this option doing?

OK, Leo, i performed more tests:
If we come into FTP -> Edit Address Book, there are general settings.
If i enter here Limit Download Speed = 50, my file is cut on FFFF (65535 = 64 KB).
If i turn this off and go to specified server entry, turn off here "use defaults" and enter here 50 = BAD
If i turn on "use defaults"... (no speed limits in general, no speed limits on entry) = OK.

Problem is located with this option.
PS. Why when i set this limit to "1", i got file in one second?

LITWINCZUK · March 11, 2011, 4:07pm

Yeah - i think any value other than 1 = my file is partially copied...

Sorry for spam in this thread but i cannot edit my posts.
Please look at Dopus code, maybe You find something what made this behavior..

LITWINCZUK · March 11, 2011, 4:08pm

PS. every i change ftp settings, i log off from server and log on them again.

LITWINCZUK · March 11, 2011, 4:22pm

One more test:
Speed limit test to "2".

Trying to copy file 200 MB.... slowly... 32 kb copied... wait... wait... 64 kb copied... - this take some seconds.... i cancel transfer, working ok.
Trying to copy opmn.log , filesize 445 KB with same setting, in same ftp session. Copied in 1 second.

--------------- OK, and here is a problem:
filesize = 531.897 bytes = copied in one second
filesize = 538.260 bytes = speed limit is working.

LITWINCZUK · March 11, 2011, 4:24pm

problem is located between filesize 531.897 and 538.260 bytes.

I don't have now time to split them for smaller parts to find this border where it's located.
Leo, if you want to do more tests, contact with me via PM.

LITWINCZUK · March 11, 2011, 4:29pm

ok, i going home...

Filesize 528.289 copied in one second.

LITWINCZUK · March 11, 2011, 4:42pm

535412: copied only 96 KB, in one second.
535413 - slow copy, speed limit working ... i wait for them. They copied ok

Again with 535412 - 90% copy operations = 96 KB (98.304). Sometime file is copied in full length (one time per 10x).
Every copy take one second.

LITWINCZUK · March 11, 2011, 4:46pm

i making this filesize with different files, like
cat aaaa >>bbb
split -b 535412 bbb ccc

and copy first file ccc wth my filesize. File content is not a problem, i think..