[Labs-l] ToolLabs back up

Fri Jan 30 23:14:32 UTC 2015

Oh, sorry, I was aware of the security hole and the Ops team's awesome (as
usual) efforts to patch it.  And I know this particular outage was due to
the security hole, and I don't think anyone could reasonably expect it to
have gone much better.  But, in general, things like the updates to NFS
have caused problems for people who value stability over improvements.
Sorry if I brought this up in the middle of a different thread though.

As always, team Analytics appreciates all the great work that's done to
keep Labs up.  A lot of our tools run in Labs and that's great for many
reasons.

On Fri, Jan 30, 2015 at 4:20 PM, John <phoenixoverride at gmail.com> wrote:

> This wasnt a case of improvements. This was an emergency patching of a
> security hole. All WMF servers had to be updated and rebooted, the only
> noticeable reboots where in labs where there isnt redundancy/standby
> services that can be swapped into place to prevent down time.
>
> This is not to say that our process went without issues. I will be
> drafting a postmortem from the user's perspective with several ideas and
> suggestions this weekend.
>
> On Fri, Jan 30, 2015 at 4:11 PM, Dan Andreescu <dandreescu at wikimedia.org>
> wrote:
>
>> FYI: some of us in Analytics are talking about a "graduate labs" kind of
>> environment.  Which would be more stable and treated a bit less
>> experimentally.  It's not something we're going to push for yet, but I was
>> wondering how people here felt about it.  It feels like our current labs
>> environment is improving at an amazing rate, but sometimes we might want to
>> prioritize stability over improvement.
>>
>> On Fri, Jan 30, 2015 at 4:05 PM, Harry Burt <jarry1250 at gmail.com> wrote:
>>
>>> Okay, gonna do something about my tools, they fall over every time this
>>> happens.
>>>
>>> Is it just a case of echo "webservice" > ".bigbrotherrc" ? Or does the
>>> daemon have to be enabled?
>>>
>>> Harry
>>>
>>> --
>>> Harry Burt (User:Jarry1250)
>>>
>>>
>>> _______________________________________________
>>> Labs-l mailing list
>>> Labs-l at lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>
>>>
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20150130/13f11d69/attachment.html>