[Labs-l] All Clear: Labs network work Wednesday 2015-08-19 21:00 UTC

Thu Aug 20 15:04:31 UTC 2015

On 8/19/15 5:04 PM, Andrew Bogott wrote:
> This is done and everything should be back to normal.  Let me know if 
> you encounter irregularities!
A few followup details:

- During the update window, there was a general network outage of about 
15 minutes.  This was because, predictably, nova-network didn't behave 
as we expected.

- Due to a puppet bug (https://phabricator.wikimedia.org/T109711), 
network performance was subpar for 18 hours or so after the switch. That 
resulted in a lot of spurious Diamond alerts and some puppet failures.  
This should be resolved now.

- The good news is:  We're now running a slightly-more-modern (and more 
upgradeable) network host.  We also learned a lot during the switch so 
should be able to arrange for a much shorter outage time during future 
upgrades.  In addition, we're a few days away from having a fully 
functioning hot spare for our network node.

-Andrew