[Labs-l] IMPORTANT: Many instances slated for reboot and downtime this weekend

Tue Sep 16 16:04:05 UTC 2014

Hi Andrew,

I just wanted to let you know that rebooting should not be a problem for

   math: mws

Best
Moritz

On Tue, Sep 16, 2014 at 11:45 AM, Andrew Bogott <abogott at wikimedia.org> wrote:
> -- Executive Summary:
>
> Many instances will be rebooted at some point this weekend or next week.
> The total list of instances subject to reboot is here:
>
> https://wikitech.wikimedia.org/wiki/Virt1006_rebuild
>
> Tools and Beta users can ignore this email.
>
>
> -- The full story:
>
> Sorry about sending two different IMPORTANT emails this week; we generally
> try to keep labs crises to a minimum.  Indeed, this email is about avoiding
> a potential crisis.
>
> The labs server known as 'virt1006' has been acting poorly lately. Several
> times in the last month we've seen instances that live on virt1006 get into
> inconsistent states during reboot... they reboot and never come back up, or
> they stay in a perpetual 'rebooting' state.
>
> So far we've been able to rescue such instances, but the misbehavior of a
> Labs server is very disconcerting.  Rather than wait for a full collapse
> (and resulting sudden death of 50+ VMs) we've decided to migrate all
> instances instances off of virt1006 and then either rebuild the system or
> discard the hardware.  Moving an instance off of a server is fairly
> painless, but it does require a few minutes of downtime and a reboot.
>
> I've spoken to a few of you directly about the reboots; the affected Tools
> and Deployment-prep instances have already been handled. There are a lot
> more to go, though.  If your instance is stable and has its init scripts set
> up properly and a reboot is no big deal, then, congratulations!  Otherwise,
> please take whatever steps you need to take to batten down the hatches and
> get ready for a reboot.
>
> If you need the reboot to happen at a scheduled time while you are standing
> by, that's totally fine.  In that case please schedule a reboot window on
> this page:
>
> https://wikitech.wikimedia.org/wiki/Virt1006_rebuild
>
> Thanks for your cooperation.
>
> -Andrew
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l

-- 
Mit freundlichen Grüßen
Moritz Schubotz

  Telefon (Büro):  +49 30 314 22784
  Telefon (Privat):+49 30 488 27330
  E-Mail: schubotz at itp.physik.tu-berlin.de
  Web: http://www.physikerwelt.de
  Skype: Schubi87
  ICQ: 200302764
  Msn: Moritz at Schubotz.de