[Xmldatadumps-admin-l] Degraded Raid Controller on dumps/snapshots storage node

Sat Jul 18 00:13:48 UTC 2009

Tomasz Finc wrote:
> Looks like we aren't getting in the replacement drives until mon/tues of 
> next week so the array will continue to be in degraded state until then. 
> Thankfully it's still under warranty so the turn around wont be too bad. 
> Tentatively putting the work to happen on Tuesday now.

We were able to put in new disks today, but the raid array didn't fully 
recover. We got lots of I/O errors, and have been unable to run JFS 
recovery successfully so far.

In the meantime we're running http://download.wikimedia.org/ off the 
copy of the last couple of dumps that had been copied to another server. 
The dump _files_ are there but currently the index is not.

We're not 100% sure whether we'll be able to recover the earlier dumps 
or not, but of course more will be made soon enough. :)

Some additional files such as the MediaWiki release download and DVD ISO 
downloads are still in process of being restored.

-- brion