[Labs-l] Resolved: Labs and toollabs outage in progress

Tue Oct 7 23:50:36 UTC 2014

Any details on what parts of toolslab went down? Ie services running on
that virt?

On Tuesday, October 7, 2014, Andrew Bogott <abogott at wikimedia.org> wrote:

> On 10/7/14 5:54 PM, Andrew Bogott wrote:
>
>> One of the labs servers (virt1005) has just died.  Marc and I are
>> investigating, but for the moment roughly 10% of labs instances are
>> currently in a SHUTOFF state.  Please do not restart these instances until
>> I send an 'all clear' message to the list.
>>
> Virt1005 is back up and seems to be OK.  I'm now booting all instances on
> that box -- they should be up and running in a few minutes, but will show
> signs of an unceremonious reboot so you'll want to make sure your services
> are all still running properly.
>
> This crash may be related to overprovisioning on virt1005... we're in the
> process of purchasing new hardware to expand capacity and avoid such issues
> in the future.
>
> Thank you again for your patience!
>
> -Andrew
>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20141007/90ca5108/attachment.html>