We had a serious MySQL crash on suda with associated data corruption today
(6/6). There's a summary of the events leading up to the crash at
http://openfacts.berlios.de/index-en.phtml?title=Wikipedia_plans
Whether it was the kill -9 that led to the corruption or whether the
database was already corrupted and did therefore not respond we do not
know; in any case, there seem to have been no alternatives to killing it
(people on #mysql knew nothing either).
Shaihulud made a copy of the CUR tables from all wikis earlier today and
imported it on Ariel. We've switched the live wikis to readonly from
Ariel; readonly because Ariel doesn't have OLD and lots of other stuff,
because it's not sufficiently tested, and because we'd like to prevent any
data loss if possible.
Tim created a special "readonly" user on ariel for this purpose.
The following have been changed as long as we are in readonly mode:
1) Counters disabled on all wikis
2) Linkscc disabled
3) readonly file set to /home/wikipedia/common/readonly
4) user_newtalk disabled
5) $wgDatabaseServer and $wgDBuser changed to ariel
There will still be lots of error messages and because the OLD tables are
not on Ariel revision histories are missing etc. This is *only* to make
sure that people can read our articles.
The next step is to fix the data corruption on suda.
Regards,
Erik