On 11/18/07, Jay R. Ashworth <jra(a)baylink.com> wrote:
Well, I gather the new version of rsync is *much*
smarter that the old
versions were about rilly, rilly big files, so perhaps this is worth
revisiting.
Really big files aren't the issue, it's a really large number that's
the issue. rsync < 3.0 will first create a list, in memory, of all
files it's going to transfer. Only once it's made the list will it
start the transfer. When I moved my server and used rsync to copy the
entire contents of the old filesystem to the new server, it used
several hundred MB of memory before it even started transferring
files. The same happens for *image* files, since there are so many.
For a single large file, rsync's clever rolling diff algorithm might
or might not be entirely optimal, but I haven't heard of either
complaints against it or improvements in recent versions.