[Labs-l] [Labs-announce] Instance creation and security group handling issues in Liberty

Thu Aug 11 01:10:26 UTC 2016

The issues discussed in this email are now resolved.

Instance creation on Horizon and Wikitech has been re-enabled. After 
some discussion we've decided to enable security group editing on 
Horizon but leave it disabled on Wikitech -- the Horizon interface is 
generally nicer, more feature-rich, and more reliable.  Please go to 
Horizon for any future security group needs.

There were two bugs that triggered this incident.  One of them[1] 
prevented enforcement of firewall rules in certain cases, and the 
other[2] enforced rules but updated them very haphazardly.  Both issues 
are now well understood, with patches in place and proper long-term 
solutions underway.

We have not yet written a full incident report, but when we do it will 
most likely be here: 
https://wikitech.wikimedia.org/wiki/Incident_documentation/20160805-LabsSecurityGroups

Sorry for the inconvenience!

-Andrew

[1]  https://phabricator.wikimedia.org/T142388

[2]  https://phabricator.wikimedia.org/T142165

On 8/5/16 3:21 PM, Chase Pettet wrote:
> Currently running instances within Labs are fine.
>
> This week we upgraded to Openstack Liberty[1][2].  Thursday (8/4) we 
> had reports of issues involving new instances[3].  We have now 
> determined there is errant behavior with Liberty managing source 
> groups.  We use this to allow instances within the same project to 
> communicate with each other.  Attempts to resolve this behavior for 
> the Tool Labs project resulted in a short issue today[4].  Requests 
> via the web proxy were failing to connect.  Tools and bots within Tool 
> Labs were still running.
>
> Currently:
> * Newly created instances are not being integrated into their security 
> groups appropriately
> * We have disabled the self-serve options for instance creation 
> temporarily
> * Modifying security groups can result in existing instances 
> experiencing issues
> * We have disabled the self-serve options for security group 
> management temporarily as well
>
> We'll update the task[3] as we have more information.  An incident 
> report will be filed as well.   As always, we can be found at labs-l 
> or on IRC in #wikimedia-labs.
>
> Thanks,
>
> Chase Pettet (on behalf of the Labs team)
>
>
> [1] https://www.openstack.org/software/liberty/
> [2] https://lists.wikimedia.org/pipermail/labs-l/2016-July/004564.html
> [3] https://phabricator.wikimedia.org/T142165
> [4] https://lists.wikimedia.org/pipermail/labs-l/2016-August/004575.html
>
>
> _______________________________________________
> Labs-announce mailing list
> Labs-announce at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-announce

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20160810/1f0e42a1/attachment.html>