How to speed up synchronization of GB's

Hi,

These days I'm facing a new challengge using Dopus.
I work half time at home and half time at the office. I work over 40-50 GB of data, and this data should be available at the office over LAN, so I must synchronize all this info when I get to the office.

As you can imagine, synchro of GB's of data is ... mortal! :blush: After more than 2 hours and a half the slider is only at 19% of file comparing on a 5200 rpm usb 2.0 drive over a pentium IV 3.0 Ghz and an 7200 rpm IDE drive ....

I guess a smarter way of doing this, is tracking changed paths and then synchro only over them. It's very tedious to write down manually those paths ... Maybe the history file of dopus can be used for this, but don't know if this can saved to a text file, if it has a maximun size, or it's organized by date ....

It would be nice to hear suggestions on how to use historic or maybe other alternatives to the syncrho job ...

Thanks in advance.

The Opus log wouldn't track changes made by other programs, which is presumably most of what needs synchronizing.

How are you synchronizing the data? Does it involve comparing the contents of the files, or just the dates? Does it take a long time due to the total size, or because there are so many small files (adding up to the total size) which are adding up to a large overhead?

If you're synchronizing changes made to huge files you might want to look for a specialised tool. There are diff/sync/patch tools which will, for example, only patch the changed parts of files, and that may speed things up (although only if the changes don't move around the other contents of the file, since you generally cannot insert data into a file using standard filesystems, only overwrite or append it). I don't know which are best for this sort of job but I know they exist. Then again, maybe those sorts of tools only make enough of a difference when going over the internet; they'll still have to read all of the data from both folders in order to work out what needs to change, and often that's about as slow as simply copying over everything based on date stamps...

Hi Nudel,

Well, in this case only changes that I make on my own (throw external programs) should be tracked; Curiously I have a way of working that always involves opening paths in dopus before operating with files, mainly because I want/need to visually check that everuthing went ok. That's why with dopus historical I think I can track most changes.

Also, I only need to synchro data files; nor systems or config files need to be identical in both machines.

In this case, both overhead due to maaaaany little files and the total amount of data slow down the job. I'm using date comparison because one or more days runs between changing from office to home and viceversa.

I don't have large file that should be updated, so in this case no need for "intra-file" synchro.

EDIT: Maybe it's faster to directly copy everything; in this way it's not neccesary the file comparing stage. :exclamation:

Well, what do you think?

I have to agree that the Directory Opus Synchronize Utility is not nearly as fast as the Copy command (or other competing synchronization utilities). Part of the performance hit is your network, but only part.

I have two FAT 32 hard drive partitions with 7-10 GB of data (software installation files), on a 3 GHz system with 2 GB RAM, with two different RAID 0 arrays (each on a different on-board RAID controller). It takes me about a 15-30 minutes to Synchronize them using DOpus (it varies with the degree of changes). It is also really annoying that while the operation completes, I am continuously interrupted by the status messages that grab the system focus (unlike those the Copy Raw command, which remain in the background, unless I click on them).

One of the biggest and most needless slow downs is that the Synchronize Utility gets in its own way, after each folder is copied (i.e. one folder out of many in the operation), trying to update its FlatView with the recent changes after the copy of that folder. This cannot do anything but unduly slow down the synchronization; updating the FlatView should wait until the copy operation is completed in its entirety, especially since that Lister that initiated the Synchronize cannot be used for any other purpose anyway, while the operation is running. (The Sync dialog interposes itself when you try to click on the underlying Lister).

However, despite its foibles, the Synchronize Utility is the only Directory Opus feature that can systematically delete files from the Destination that do not exist in the Source. In a one-way Synchronize, this is really the only feature, that distinguishes the Synchronize Utility from what can be accomplished with DOpus Raw commands. But, Synchronize also has no override to bypass the Recycle Bin, when deleting files. So this is another huge performance drain.

If you do not require deleting files from the Destination, you can try using these two buttons in-sequence with these commands:

Select All Copy WHENEXIST=skip and

Select All Copy UPDATEEXISTING=date FORCE

This will somewhat faster, requires two passes, and it does not delete files from the Destination that do not exist in the Source.

Microsoft recently released a Power Toy named Sync Toy. It's actually really good, for a free application. It allows you to save Folder Pairs, that become one-button synchronize operations. You can get it here. http://www.microsoft.com/windowsxp/using/digitalphotography/prophoto/synctoy.mspx

Synchronize is still new to DOpus (new in version 8.0), so it has some growing pains to go through still. Greg and Jon did a lot of hard work to get it in, and I'm sure they will be improving it based on feedback. (I'm still typing up my analysis to send them, and other have posted issues).
Make sure you log your issue on the official support site—the squeaky wheel gets the grease!

I would thinkk any speed difference should only depend on what criteria you select for the sync/compare. If you choose byte comparison, then it's naturally going to be waaaaaay slower than the copy sync raw commands - which ONLY check file metadata liek size and date.

Ken's point about the lister display having to update is well taken - but you can disable fields in a 'sync lister' format to cut down on the impact of any refresh lag - like description column should never be enabled in a sync tool lister display if you're concerned about speed... The flat view itself shouldn't be too bad, and the Hide unaffected files option miiiight cut down further on any flat view refresh speed impact... but Ken's idea for it to NOT even bother refreshing until the entire sync op is done is a GREAT suggestion.

Steje,
The Synchronize Utility is really slow in comparison to the Copy command, even using One-Way copy with a simple Date (newer) comparison. If I could turn off FlatView for Synchronize, I would. (But I do appreciate that Directory Opus goes into FlatView to review and confirm all file deletions and overwrites.)

Part of performance is also the order in which things occur. Many things are handled as a separate complete pass through the list of objects