[Labs-l] NFS outage in progress [UPDATE]

Thu Jun 18 12:55:49 UTC 2015

Toolslabs will be back soon?  When approximately?

> Date: Wed, 17 Jun 2015 23:09:11 -0400
> From: marc at uberbox.org
> To: abogott at wikimedia.org; labs-l at lists.wikimedia.org
> Subject: Re: [Labs-l] NFS outage in progress [UPDATE]
> 
> On 15-06-17 09:29 PM, Andrew Bogott wrote:
> > Coren is rebooting and fscking the system -- with luck it'll be up
> > again within the hour.
> 
> Fortune does not smile upon us; the fsck is taking a LONG time to
> progress (over a 40T filesystem), but the repair goes apace.
> 
> There is already indication of some lost data - though the failures are
> likely to date from a past crash and may have gone unnoticed for some time.
> 
> Either way, we have a fairly recent (8-9 days) backup of the entire
> filesystem so we may be able to recover lost or corrupted files if any
> turn up.
> 
> Worst case scenario is a catastrophic loss of the entire filesystem
> (which does not seem likely at this time), in which case we will restore
> the filesystem completely from the backup.
> 
> The repair process will take at least a few more hours, during which I
> will be keeping watch.  I will update this list as soon as I have a
> better assessment of the damage, if any.
> 
> -- Marc
> 
> 
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20150618/3f85418d/attachment.html>