[Labs-l] Rolling reboots today (finished)

Thu Jan 29 17:39:52 UTC 2015

On 1/29/15 8:10 AM, Federico Leva (Nemo) wrote:
> Thanks for the massive update job! One of my instances, 
> ttmserver-salt01 (i-00000773.eqiad.wmflabs), was reportedly in "reboot 
> pending" state: I rebooted it manually. If that's related to the 
> security updates, perhaps it's worth checking whether reboots actually 
> completed in all instances.
Yeah, there are a few stragglers.  The current theory is that a few of 
our extra-busy virt hosts (virt1002 and 1009, mostly) were swamped by 
the effort of booting so many instances at once and dropped a few back 
to the SHUTDOWN state.  I've hunted down most of these already, but 
anyone with a stopped-and-unbootable instance should contact me; it's 
pretty easy for me to get them up again.

-Andrew

>
> Nemo
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l