On Sun, Nov 18, 2007 at 02:11:49PM -0500, David A. Desrosiers wrote:
On Sun, 2007-11-18 at 13:53 -0500, Anthony wrote:
For the immediate future a way to restart a
broken dump is probably
the most important. Find the last ~900K segment of the bz2 file,
remove it, add the bzip2 end of file information, then concatenate the
rest of the dump? Sound reasonable?
I raised something similar about 3-4 years ago, regarding the
--rsyncable option of gzip, when the Wikipeda dump servers allowed us to
rsync the dumps, instead of straight http.
Perhaps something similar should be revisited on the server-side, while
preparing and compressing those dumps?
(waits for brion to counter with something valid to pull out the rug on
my idea =)
Well, I gather the new version of rsync is *much* smarter that the old
versions were about rilly, rilly big files, so perhaps this is worth
revisiting.
Cheers,
-- jra
--
Jay R. Ashworth Baylink jra(a)baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates
http://baylink.pitas.com '87 e24
St Petersburg FL USA
http://photo.imageinc.us +1 727 647 1274
Witty slogan redacted until AMPTP stop screwing WGA