[Xmldatadumps-admin-l] Degraded Raid Controller on dumps/snapshots storage node

Tue Jul 21 01:35:50 UTC 2009

Brion Vibber wrote:
> Tomasz Finc wrote:
>> Looks like we aren't getting in the replacement drives until mon/tues 
>> of next week so the array will continue to be in degraded state until 
>> then. Thankfully it's still under warranty so the turn around wont be 
>> too bad. Tentatively putting the work to happen on Tuesday now.
> 
> We were able to put in new disks today, but the raid array didn't fully 
> recover. We got lots of I/O errors, and have been unable to run JFS 
> recovery successfully so far.
> 
> In the meantime we're running http://download.wikimedia.org/ off the 
> copy of the last couple of dumps that had been copied to another server. 
> The dump _files_ are there but currently the index is not.
> 
> We're not 100% sure whether we'll be able to recover the earlier dumps 
> or not, but of course more will be made soon enough. :)
> 
> Some additional files such as the MediaWiki release download and DVD ISO 
> downloads are still in process of being restored.
> 
> -- brion

Thanks for the update Brion. I'll be checking in with Rob tomorrow to 
see how ready the new set of drives are and if we are set to start 
generating the snapshots anew.

--tomasz