[Xmldatadumps-admin-l] Degraded Raid Controller on dumps/snapshots storage node

Wed Jul 15 22:17:19 UTC 2009

I've been looking into really bad i/o times on the storage node behind 
the XML snapshots and have narrowed it down to a really degraded raid10 
disk set.

Unit     UnitType  Status         %RCmpl  %V/I/M  Port  Stripe  Size(GB)
------------------------------------------------------------------------
u0       RAID-10   DEGRADED       -       -       -     64K     3725.21

Which is all sorts of bad and scary. I'm going to send RobH in to check 
these drives over tomorrow. Fixing these drives should be non 
destructive but to capture the unhappy case of massive failure I'm going 
to start a backup of the two most recent snapshots for each wiki to one 
of our other storage nodes.

Any downtime will be reported on download.wikimedia.org and here.

Will update after Rob's work is done.

--tomasz