[Labs-l] [Labs-announce] Possible reboots and/or outages -- please read

Fri May 20 15:45:38 UTC 2016

Does this unusual behavior, also cause erroneous fingerprints being given during authentication.  Yesterday when I was SSHing into Cyberbot-exec-01 via bastion I got a fingerprint mismatch with the exec node.  This was using WinSCP.

To clarify, it got into Bastion just fine but not into my node, because I aborted the authentication process.  I waited 5 minutes and tried again and it worked just fine.

Cyberpower678
English Wikipedia Account Creation Team
ACC Mailing List Moderator
Global User Renamer

> On May 20, 2016, at 11:10, Andrew Bogott <abogott at wikimedia.org> wrote:
> 
> Note:  Tools users can ignore this message
> 
>    We are seeing some unusual behavior on labvirt1003, which hosts a large number of labs instances.  The problem is not yet diagnosed, but it is likely a hardware problem that will require reboots or downtime.  Here is a complete list of labs instances currently living on labvirt1003:
> 
> https://phabricator.wikimedia.org/P3159
> 
>    If you have any hosts on that box that cannot survive a reboot, please either let me know, or take steps to minimize the damage.  I've removed labvirt1003 from the scheduler, so if you want to build a new instance and migrate services to it you can be assured that the new instance will be isolated from the coming chaos.
> 
>    A simple reboot shouldn't produce more than 5-10 minutes of downtime.  If a major outage seems likely, I'll follow up with additional warning.
> 
> -Andrew
> 
> 
> _______________________________________________
> Labs-announce mailing list
> Labs-announce at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-announce
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l