[Labs-l] Short service interruption today

Wed Nov 27 03:22:38 UTC 2013

Hello,

Today around 22:30 UTC there was a failure of the file system underlying
the NFS service that caused an outage for the Labs NFS server.

The failure was apparently caused by an interaction between XFS and the
group quota system (which has since been disabled).  While no serious
damage was caused to the filesystem, correcting the mistake required
offlining it for a check, and restarting the NFS server.

The outage and repair lasted a little under an hour, and many tools were
impacted and may need restarting since the view of the file system may
have been incorrect for intervals during that period.

Things are back to normal, now, and the faulty subsystem will remain
disabled.

-- Marc