[Labs-l] Resolved: Labs and toollabs outage in progress

Tue Oct 7 23:32:33 UTC 2014

On 10/7/14 5:54 PM, Andrew Bogott wrote:
> One of the labs servers (virt1005) has just died.  Marc and I are 
> investigating, but for the moment roughly 10% of labs instances are 
> currently in a SHUTOFF state.  Please do not restart these instances 
> until I send an 'all clear' message to the list.
Virt1005 is back up and seems to be OK.  I'm now booting all instances 
on that box -- they should be up and running in a few minutes, but will 
show signs of an unceremonious reboot so you'll want to make sure your 
services are all still running properly.

This crash may be related to overprovisioning on virt1005... we're in 
the process of purchasing new hardware to expand capacity and avoid such 
issues in the future.

Thank you again for your patience!

-Andrew