Compress files before copying from sFTP site

daroc · August 12, 2013, 11:55am

I'm a web developer and very often have to copy a lot of files from/to sFTP site. Those files are usually very small (<10 KB) but there are thousands of them which causes transfer to slow down awfully.

Is there any chance DOpus could compress/decompress files on sFTP site via command in context menu? It would be much easier than using Putty or other utilities.

Another idea is to compress and decompress files on-the-fly but it might be a little messy, especially when file transfer is interrupted.

It would be nice to have similar feature on regular FTP sites but I'm afraid it might be impossible.

daroc · August 12, 2013, 12:01pm

Oh, and to be clear - by "compressing" and "compressing on-the-fly" I mean compress files using ZIP/GZIP/TAR or other formats, not compressing connection which might be quite useless if there are a lot of very small files.

Leo · August 12, 2013, 1:45pm

I don't think Opus could do the decompression on the remote end because each instance of Linux/Unix seems to have wildly different decompression tools, versions and command lines. (It takes me long enough to work out how to do it by hand on a new shell account .)

daroc · August 12, 2013, 1:54pm

And this is why I wanted DOpus to automate this process.
Maybe it could somehow check which tools are present on the remote server. I don't think there are hundreds of them. I guess there are 5 or 7 compression/decompression methods that will cover 95% of servers.

Please note that WinSCP has such option and it works on several servers I just checked.

daroc · August 19, 2013, 10:15am

Maybe you could introduce customizable FTP menus (with ability to create custom commands so one could add commands that work on servers he uses.

engineersthumb · August 21, 2013, 1:48am

Most of the time the additional compression for individual small files would prob not be worth the I/O bother.

It is better to gather all your small files into a single tar archive and transmit one large file.

If you have access to putty you can surely execute standard linux commands on the remote server, no?

e.g.
tar -cf smallfiles.tar *.txt

Use cwRsync or Deltacopy - rsync implementations on windows that allow compression on the fly
serverfault.com/questions/265171 ... -os-server
Use MobaXterm (a better putty), which allows you to use rsync, tar and other compression tools on windows and access the full linux command setavailable to the remote machine.

engineersthumb · August 21, 2013, 1:52am

Maybe you can use point 2 or 3 to create a script/plugin that you can add to the right-click menu? Don't know if you can do this for the ftp sites in the Dopus implementation though.

daroc · August 21, 2013, 11:20am

[quote="engineersthumb"]1) If you have access to putty you can surely execute standard linux commands on the remote server, no?

e.g.
tar -cf smallfiles.tar *.txt
[/quote]
I can do that, but opening Putty is yet another thing to do when deploying project.
On FTP/sFTP I can see "send raw command" in lister context menu. I thought it might be a way to send a command to remote shell but it seems that it doesn't work on sFTP sites. It's meant to send a command to FTP.

I will try 2 and 3.

Greg · August 21, 2013, 11:54pm

Yes. I don't believe this is possible. There's no actual shell to send commands. You can send manual commands via Opus' FTP connection but these will be limited to 'normal' FTP commands, not arbitary shell commands. While SFTP operates over an SSH tunnel, I don't 'believe' the remote FTP server will accept commands such as TAR etc. But I am prepared to be corrected. (Once only! )

engineersthumb · August 22, 2013, 12:32am

that is a good point.

@daroc: to clarify, do you only have access to just the sftp server on the remote machine or can you also actually ssh in via a shell, execute linux commands, etc?

If you have access to an actual prompt, your options are relatively large.

Greg · August 22, 2013, 1:02am

You don't have access to the shell.

engineersthumb · August 22, 2013, 1:10am

I meant: does he have access to a shell in addition to the sftp server on the remote machine, or is he limited to just sftp access.

daroc · August 22, 2013, 9:20am

I have full access to ssh. Nevertheless I'm used to DOpus and don't want to completely change the way I work. This is why I asked for a way to compress files using DOpus GUI.
Alternatively, I'd prefer a simple application that would compress selected files to .zip or .tar.gz, send the file over ssh/sftp to apt remote folder (selected in DOpus in dest lister) and automatically decompress the file.

Rsync seems to be much larger tool than necessary.

engineersthumb · August 22, 2013, 4:18pm

Allright. So a lot depends on your workflow and situation. You have many ways of accomplishing this.

Use a dedicated SFTP client which already has this functionality and don't bother getting Dopus involved
If you are constantly in the process of (e.g. incremental changes to website design) modifying and uploading files to a test server, install dropbox on your local machine, then install it on linux remote machine and select option to selectively sync one folder. This is the project folder you are working on. Place all your work relating to this remote machine in this folder and do any changes within it. All changes are incrementally modified, so you don't have to do big push uploads.

caveats: You will prob find it beneficial to pause dropbox sync while working on local machine, as changes to large files (i.e.10mb+) can cause problems if your upload speed is too low. Plus it's network inefficient to constantly sync as you work. Better to sync at end of day.

Also I believe dropbox is unencrypted and the files on the dropbox server are held in the US, somewhere not exactly known for protection of it's citizens privacy. So if you are working with sensitive docs, or just stuff that is private this is a concern. There are other tools like dropbox that possibly have encrypted features, but you will have to look into that and check if they are cross platform.

Install putty.exe, configure it, then design a sequence of commands that let you use it from within Dopus.

I've done this and will detail it in my next post. The commands are not complete, because you will want to introduce variables that automate the task in a better fashion. My knowledge of Dopus scripting is quite poor and I don't have that much time, but I'm sure you prob have a better idea than me on how to do this bit.

engineersthumb · August 22, 2013, 4:24pm

crappy guide to using putty with dopus:

1) download latest version of putty installer from:
the.earth.li/~sgtatham/putty/lat ... taller.exe

2) Add the location of the putty programs folder to the windows path:
etherealmind.com/putty-command-line/

3) Configure a Saved session in putty. In this test, I have created one called "015_test"

a) enter your remote machine hostname & port (usually 22). Select ssh as connection type

b) Go to Category --> Connection -->Data
enter your user id for the remote machine in the auto-login bit

Go to Category -->Connection --> SSH
Check "Enable compression"
Move Blowfish to the position in-between AES and Arcfour
save your settings

c) See the following links if you have problems with the key generation:

Start the program puttygen.exe to generate a public and a private key. Do not use any passphrase. Save this in your putty programs folder

In the putty key generator program, there is a grey box that says "Public key for pasting into openssh authorised_keys file". copy the key

Use putty to ssh into your remote machine and then execute the following linux commands:

[code]cd .ssh
nano authorized_keys # i can use nano generally. if you can't (in order of difficulty try gedit, nedit, joe, vim or vi.

at the end of authorized_keys, paste the public key you copied from the putty key generator program and save the file

exit your ssh session[/code]

d) Go back to the Putty Configuration Window
Go to Category -->Connection --> SSH --> Auth
Enter the path of the saved private key file (press browse). Leave other defaults as such.
Launch your saved session. In my case "015_test"
Enter your password. You should connect in.
Disconnect
Relaunch your saved session. You should not be prompted for a password now.

######################
4) Now you have to configure dopus. This bit can do with lots of improvement as I don't really know much about dopus scripting. this is the skeleton of a proper command. It does the following actions:

You need to create a file called iLikeCheese.txt for this example to work.
You also need to change the word username to your username as in the example I am uploading the file to my home directory which requires this. Ideally you would create a variable that allows you to specify the path. I think you need the absolute path as I couldn't get it working with ~ instead of /home/username/
the copy addtoarchive presents a dialogue for which I selected the format as 7zip-tar-gz, as pretty much all linux distros I know of have the tar command.
One issue is that which my (crappy) version of the Dopus command, you need to make sure the source and destination for the addtoarchive dialogue are the same, as it seems to default to src & dest.
note that the plink and pscp commands use the putty session shortcut we created earlier 015_test. You can replace this with username@remotemachineName in a variable if you want. This would mean that you can use this button for any remote machine. However, you will be prompted for user id. May have to modify pscp command to make this work though

@firstfileonly cd {sourcepath} copy addtoarchive=.tgz pscp iLikeCheese.tgz 015_test:/home/username/ plink -ssh 015_test tar -zxvf iLikeCheese.tgz

Hopefully you or others reading this may be able to improve this skeleton Dopus command and post it here.

All the best!

engineersthumb · August 22, 2013, 4:48pm

[quote="daroc"]
Alternatively, I'd prefer a simple application that would compress selected files to .zip or .tar.gz, send the file over ssh/sftp to apt remote folder (selected in DOpus in dest lister) and automatically decompress the file.

Rsync seems to be much larger tool than necessary.[/quote]

ps. I'm sure you can modify the skeleton command so that you can browse to the remotemachine-folder you wish to copy the file to.

then you will have to create a variable of the path: If I browse to a linux ftp folder, the path in dopus looks like:

ftp://CheeseFactory.somewhere.ac.uk//ho ... e/myfolder

you prob have to create a dest path variable that strips out the bits in bold:

ftp://CheeseFactory.somewhere.ac.uk//home/username/myfolder

so you get:

remoteDir_VAR="/home/username/myfolder"

then modify the skeleton command line to something like:

@firstfileonly cd {sourcepath} copy addtoarchive=.tgz pscp iLikeCheese.tgz 015_test:{remoteDir_VAR} plink -ssh 015_test tar -zxvf {remoteDir_VAR}/iLikeCheese.tgz

daroc · August 23, 2013, 11:44am

Thanks for pointing plink.exe out.

This seem to be simply enough to work with.

I open local filesystem and sFTP server in dual display mode.

Button to compress selected files and send it to sFTP. Created file will be named _copied.tgz.
I had to create file locally and then copy as it didn't work when I tried to create archives directly on remote server.
I had to use dopusrt.exe. It didn't work when I just used command (archive wasn't copied to the server).

<?xml version="1.0"?> <button backcol="none" display="both" textcol="none"> <label>Zip to dest</label> <icon1>#copy</icon1> <function type="normal"> <instruction>@firstfileonly</instruction> <instruction>@sync </instruction> <instruction>"/home/dopusrt.exe" /cmd Copy CREATEFOLDER _copied HERE ARCHIVE=.tgz</instruction> <instruction>"/home/dopusrt.exe" /cmd Copy Move _copied.tgz</instruction> </function> </button>

This button decompress _copied.tgz on sFTP.

<?xml version="1.0"?> <button backcol="none" display="both" textcol="none"> <label>Untar _copied.tgz</label> <icon1>#extract</icon1> <function type="normal"> <instruction>SELECT *</instruction> <instruction>@firstfileonly </instruction> <instruction>@nodeselect </instruction> <instruction>RENAME TO="*"</instruction> <instruction>SELECT NONE</instruction> <instruction>@script vbscript </instruction> <instruction>Option Explicit</instruction> <instruction>Sub Run(ByVal sFile)</instruction> <instruction>Dim shell</instruction> <instruction> Set shell = CreateObject("WScript.Shell")</instruction> <instruction> shell.Run Unescape(sFile), 1, false</instruction> <instruction> Set shell = Nothing</instruction> <instruction>End Sub</instruction> <instruction>Function Rename_GetNewName ( strFileName, strFilePath, fIsFolder, strOldName, ByRef strNewName )</instruction> <instruction> Dim pathArray</instruction> <instruction> Dim passwdArray</instruction> <instruction> Dim userArray</instruction> <instruction> Dim addrArray</instruction> <instruction> Dim user</instruction> <instruction> Dim passwd</instruction> <instruction> Dim addr</instruction> <instruction> pathArray = split(strFilePath, ":")</instruction> <instruction> passwdArray = split(pathArray(2), "@")</instruction> <instruction> userArray = split(pathArray(1), "?")</instruction> <instruction> passwd=passwdArray(0)</instruction> <instruction> user=userArray(1)</instruction> <instruction> addrArray = split(passwdArray(1), "/")</instruction> <instruction> addr=addrArray(0)</instruction> <instruction> Dim path</instruction> <instruction> path = Mid(passwdArray(1), InStr(passwdArray(1), "/"))</instruction> <instruction> Run """C:\Program Files (x86)\PuTTY\plink.exe"" -ssh -pw " & passwd & " " & user & "@" & addr & " cd """ & path & """ && tar -xvf _copied.tgz"</instruction> <instruction>End Function</instruction> </function> </button>

This button compress all files in current folder to _server.tar.gz.

<?xml version="1.0"?> <button backcol="none" display="both" textcol="none"> <label>Tar to _server.tar.gz</label> <icon1>#newzipfile</icon1> <function type="normal"> <instruction>SELECT *</instruction> <instruction>@firstfileonly </instruction> <instruction>@nodeselect </instruction> <instruction>RENAME TO="*"</instruction> <instruction>SELECT NONE</instruction> <instruction>@script vbscript </instruction> <instruction>Option Explicit</instruction> <instruction>Sub Run(ByVal sFile)</instruction> <instruction>Dim shell</instruction> <instruction> Set shell = CreateObject("WScript.Shell")</instruction> <instruction> shell.Run Unescape(sFile), 1, false</instruction> <instruction> Set shell = Nothing</instruction> <instruction>End Sub</instruction> <instruction>Function Rename_GetNewName ( strFileName, strFilePath, fIsFolder, strOldName, ByRef strNewName )</instruction> <instruction> Dim pathArray</instruction> <instruction> Dim passwdArray</instruction> <instruction> Dim userArray</instruction> <instruction> Dim addrArray</instruction> <instruction> Dim user</instruction> <instruction> Dim passwd</instruction> <instruction> Dim addr</instruction> <instruction> pathArray = split(strFilePath, ":")</instruction> <instruction> passwdArray = split(pathArray(2), "@")</instruction> <instruction> userArray = split(pathArray(1), "?")</instruction> <instruction> passwd=passwdArray(0)</instruction> <instruction> user=userArray(1)</instruction> <instruction> addrArray = split(passwdArray(1), "/")</instruction> <instruction> addr=addrArray(0)</instruction> <instruction> Dim path</instruction> <instruction> path = Mid(passwdArray(1), InStr(passwdArray(1), "/"))</instruction> <instruction> Run """C:\Program Files (x86)\PuTTY\plink.exe"" -ssh -pw " & passwd & " " & user & "@" & addr & " cd """ & path & """ && tar -zcvf _server.tar.gz *"</instruction> <instruction>End Function</instruction> </function> </button>

There might be need to include/exclude selected items as some folders might contain temporary or backup files. But for now I have no idea how to do it using plink.

engineersthumb · August 24, 2013, 12:11am

i'm going to have a play with your code over the next few days and see if i can learn something.

daroc · August 24, 2013, 11:50am

I've improved those codes a little bit.

First button executed commands asynchronously and didn't wait until archiving is complete.
And I've improved second and third button so they operate on selected archives or files/folders.

<?xml version="1.0"?> <button backcol="none" display="both" textcol="none"> <label>Zip to dest as _copied.tgz</label> <icon1>#copy</icon1> <function type="normal"> <instruction>@disablenosel </instruction> <instruction>@sync </instruction> <instruction>Copy CREATEFOLDER _copied HERE ARCHIVE=.tgz</instruction> <instruction>"/home/dopusrt.exe" /cmd Copy Move _copied.tgz</instruction> </function> </button>

<?xml version="1.0"?> <button backcol="none" display="both" textcol="none"> <label>Tar selected</label> <icon1>#newzipfile</icon1> <function type="normal"> <instruction>RENAME TO="*"</instruction> <instruction>@script vbscript </instruction> <instruction>Option Explicit</instruction> <instruction>Sub Run(ByVal sFile)</instruction> <instruction>Dim shell</instruction> <instruction> Set shell = CreateObject("WScript.Shell")</instruction> <instruction> shell.Run Unescape(sFile), 1, false</instruction> <instruction> Set shell = Nothing</instruction> <instruction>End Sub</instruction> <instruction>Function Rename_GetNewName ( strFileName, strFilePath, fIsFolder, strOldName, ByRef strNewName )</instruction> <instruction> Dim pathArray</instruction> <instruction> Dim passwdArray</instruction> <instruction> Dim userArray</instruction> <instruction> Dim addrArray</instruction> <instruction> Dim user</instruction> <instruction> Dim passwd</instruction> <instruction> Dim addr</instruction> <instruction> pathArray = split(strFilePath, ":")</instruction> <instruction> passwdArray = split(pathArray(2), "@")</instruction> <instruction> userArray = split(pathArray(1), "?")</instruction> <instruction> passwd=passwdArray(0)</instruction> <instruction> user=userArray(1)</instruction> <instruction> addrArray = split(passwdArray(1), "/")</instruction> <instruction> addr=addrArray(0)</instruction> <instruction> Dim path</instruction> <instruction> path = Mid(passwdArray(1), InStr(passwdArray(1), "/"))</instruction> <instruction> Run """C:\Program Files (x86)\PuTTY\plink.exe"" -ssh -pw " & passwd & " " & user & "@" & addr & " cd """ & path & """ && tar -zcvf ""'" & strFileName & ".tar.gz'"" ""'" & strFileName & "'"""</instruction> <instruction>End Function</instruction> </function> </button>

<?xml version="1.0"?> <button backcol="none" display="both" textcol="none"> <label>Untar</label> <icon1>#extract</icon1> <function type="normal"> <instruction>RENAME TO="*"</instruction> <instruction>@script vbscript </instruction> <instruction>Option Explicit</instruction> <instruction>Sub Run(ByVal sFile)</instruction> <instruction>Dim shell</instruction> <instruction> Set shell = CreateObject("WScript.Shell")</instruction> <instruction> shell.Run Unescape(sFile), 1, false</instruction> <instruction> Set shell = Nothing</instruction> <instruction>End Sub</instruction> <instruction>Function Rename_GetNewName ( strFileName, strFilePath, fIsFolder, strOldName, ByRef strNewName )</instruction> <instruction> Dim pathArray</instruction> <instruction> Dim passwdArray</instruction> <instruction> Dim userArray</instruction> <instruction> Dim addrArray</instruction> <instruction> Dim user</instruction> <instruction> Dim passwd</instruction> <instruction> Dim addr</instruction> <instruction> pathArray = split(strFilePath, ":")</instruction> <instruction> passwdArray = split(pathArray(2), "@")</instruction> <instruction> userArray = split(pathArray(1), "?")</instruction> <instruction> passwd=passwdArray(0)</instruction> <instruction> user=userArray(1)</instruction> <instruction> addrArray = split(passwdArray(1), "/")</instruction> <instruction> addr=addrArray(0)</instruction> <instruction> Dim path</instruction> <instruction> path = Mid(passwdArray(1), InStr(passwdArray(1), "/"))</instruction> <instruction> Run """C:\Program Files (x86)\PuTTY\plink.exe"" -ssh -pw " & passwd & " " & user & "@" & addr & " cd """ & path & """ && tar -xvf """ & strFileName & """"</instruction> <instruction>End Function</instruction> </function> </button>

Please note that Tar and Untar buttons will work correctly only on remote sFTP servers. If you try it on local files probably nothing wrong will happen, but it was not intended to be used on local files so use on your own risk.

engineersthumb · August 24, 2013, 11:54pm

Hi,

FYI Here are the commands to compress a file/folder on the remote server. Apologies if i am repeating info you already know

I could not get any test plink commands working using the Dopsus CLI, but they work fine if you use the command bar (thing similar to the filter bar. dunno exact name)

plink -ssh 015_test tar -zcvf /home/username/aiCarambaMoreCheese.tgz /home/username/SomeFile.txt

The anatomy of the command is:
015_test: The name of your saved session.

tar till end of code: the linux command you wish to run and it's associated parameters and options. If you search for man tar on google, you should find the command manual.

In the case of the above command -zcvf means create a compressed file with verbose output. Replacing c with x, makes it "extract", instead of "create". Inclusion or exclusion of -z equates to whether compression (with gzip, j=bzip) is "on" or "off" respectively.

/home/username/aiCarambaMoreCheese.tgz is the name of the file you wish to create.

/home/username/SomeFile.txt = Path to file. If you use absolute paths, your tar file will replicate the folder structure within it. So in order to get relative paths look at example below:

##################

plink command to create a .tgz file on the remote-machine that excludes certain file types:

I have created a test folder structure on the linux-remote machine located at /home/username/brie/test_tar/cheesychips . Contents are:

[ul]In /home/username/brie/test_tar:
cheesychips test10.txt test2.info test3.txt test5.info test6.txt test8.info test9.txt
--exclude=*.txt test1.info test2.txt test4.info test5.txt test7.info test8.txt
test10.info test1.txt test3.info test4.txt test6.info test7.txt test9.info[/ul]

[ul]In /home/username/brie/test_tar/cheesychips:
test20.info test21.txt test23.info test24.txt test26.info test27.txt test29.info test30.txt
test20.txt test22.info test23.txt test25.info test26.txt test28.info test29.txt
test21.info test22.txt test24.info test25.txt test27.info test28.txt test30.info[/ul]

the tar command is recursive by default, so the command:

plink -ssh 015_test cd /home/username/brie ; tar -zcvf /home/username/aiCarambaMoreCheese.tgz --exclude='*.txt' test_tar ; cd -

Will recursively compress everything apart from files with the .txt extension. So you should get only the .info files.

Anatomy of command (actually three linux commands)
a) plink -ssh 015_test :
use the plink command on your windows machine

b) cd /home/username/brie ;
change directory to where the folder you want to compress is. You can get the path of whatever directory you are in the Dopus dest panel and change "/home/username/brie" to ${Doupus-destPath-LinuxFormat}

c) tar -zcvf /home/username/aiCarambaMoreCheese.tgz --exclude='*.txt' test_tar ;

The actual tar command to create a compresed gzip archive. If you want Dopus to do this from any location the following variables need to be created and the command modified as follows:

Probably with path chopping like you have already done or passing of strings (with dopus dlgstring) for exclude file pattern

tar -zcvf ${tgz-save-path}/${tgz-filename}.tgz --exclude='${exclude-File-pattern}' ${tgz-foldername-ToBe-tgz'd}

d) cd -

moves back to previous directory
probably not needed if you are only cd'ing into 1 directory and don't plan to cd again. included in case you are going to do more complicated things.

If you want to exclude more than 1 kind of filename/type, just whack another --exclude='{exclude-File-pattern}' in the command. For more info on this feature of tar, see:

cyberciti.biz/faq/exclude-ce ... r-command/