[Xmldatadumps-admin-l] Degraded Raid Controller on dumps/snapshots storage node

Thu Jul 16 17:29:22 UTC 2009

Tomasz Finc wrote:
> I've been looking into really bad i/o times on the storage node behind 
> the XML snapshots and have narrowed it down to a really degraded raid10 
> disk set.
> 
> Unit     UnitType  Status         %RCmpl  %V/I/M  Port  Stripe  Size(GB)
> ------------------------------------------------------------------------
> u0       RAID-10   DEGRADED       -       -       -     64K     3725.21
> 
> Which is all sorts of bad and scary. I'm going to send RobH in to check 
> these drives over tomorrow. Fixing these drives should be non 
> destructive but to capture the unhappy case of massive failure I'm going 
> to start a backup of the two most recent snapshots for each wiki to one 
> of our other storage nodes.
> 
> Any downtime will be reported on download.wikimedia.org and here.
> 
> Will update after Rob's work is done.

Looks like we aren't getting in the replacement drives until mon/tues of 
next week so the array will continue to be in degraded state until then. 
Thankfully it's still under warranty so the turn around wont be too bad. 
Tentatively putting the work to happen on Tuesday now.

--tomasz