Tony Sidaway wrote:
Brion Vibber said:
Brion Vibber wrote:
Update logs are still replaying, but we're up
to 42 minutes prior to
the crash on one machine and still going. I don't expect problems.
With two servers fully recovered we've got the wikis up for read-write
access; editing is open. Total time from crash to restoring edit
service was about 24 hours, 10 minutes. Sigh.
Some special pages (including contribs and watchlist) are off for the
moment to reduce server load until we have more machines up. Some
things remain a little wonky.
Interesting discussion on Slashdot about the relative recoverability of
Postgresql. If we stay with open source DBMS, perhaps at least some of
the database servers should be running alternative software.
Kudos to the developers for their heroic efforts in bringing everything
back from what threatened to be a serious data loss.
Regarding using different databases: I agree, diversity is good.
However, I should point out that I have destroyed a PostgreSQL database
on one occasion by power-cycling a machine (it was running VACUUM at the
time). This makes me sceptical about relying on software diversity
alone, particularly in the face of crude threats such as power loss,
fire, tornadoes and flood.
The value of the Wikipedia data is now big enough that it worth putting
a formal disaster recovery plan in place.
A good idea in the short term might be to keep a slave database or two
offsite, so that it they are unlikely to crash at the same time as the
central site. Note that online slaves are not a 100% solution to data
corruption, as they will faithfully mirror any corruption which
accumulates from causes other than database failure.
This emphasizes the importance of taking and saving snapshot dumps. At
the moment, keeping off-site dumps is done on an ad-hoc basis by
volunteers. This should certainly be formalized to include the
automatic creation and archiving of dumps off-site, in addition to
running offsite slave databases. At a data rate of only 10 Mbits/s, a
170GB offsite backup would take only 38 hours to move offsite. At these
sorts of rates, monthly backups could be lodged with any of a number of
mirror services, perhaps organizations such as universities, the UK
mirror service and Internet artchive might be interested in doing this?
The current worst case would be physical destruction of the servers at
the Florida colo; the data is both priceless and uninsurable, but is the
server farm insured against this sort of event?
-- Neil