[Labs-admin] RECOVERY alert - ToolLabs/ToolLabs Home Page is OK

Mon Nov 28 14:07:33 UTC 2016

Possibly related to https://phabricator.wikimedia.org/T142932?

On Sun, Nov 27, 2016 at 1:11 AM, Andrew Bogott <abogott at wikimedia.org>
wrote:

> I poked at this, but I'm pretty sure it recovered on its own.  When I
> tried to restart the service by hand, I got this:
>
> Traceback (most recent call last):
>   File "/usr/bin/webservice-runner", line 27, in <module>
>     webservice.run(port)
>   File "/usr/lib/python2.7/dist-packages/toollabs/webservice/services/lighttpdwebservice.py",
> line 108, in run
>     with open(config_path, 'w') as f:
> IOError: [Errno 13] Permission denied: '/var/run/lighttpd/admin'
>
> It's late and this is half-baked (and my attempts to fix the problem
> destroyed the evidence) but my speculation is that in some situations we
> are 'leaking' read-only /var/run/lighttpd/admin files.  Once one of them is
> out there, each time the webservice restarts it's just the luck of the draw
> whether we hit an exec host that has or doesn't have a read-only file, so
> the failure is intermittent.
>
> For now, I've explicitly removed that file on all trusty lighttpd hosts.
> When/if this problem recurs we should check the writeability of the
> complaining file before doing anything else.
>
> -A
>
>
> On 11/27/16 1:05 AM, shinken wrote:
>
>> Notification Type: RECOVERY
>>
>> Service: ToolLabs Home Page
>> Host: ToolLabs
>> Address: tools.wmflabs.org
>> State: OK
>>
>> Date/Time: Sun 27 Nov 07:05:01 UTC 2016
>>
>> Additional Info:
>>
>> HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.041 second response time
>>
>
>
>
> _______________________________________________
> Labs-admin mailing list
> Labs-admin at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-admin
>

-- 
Chase Pettet
Engineering Manager -- Labs
chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and
IRC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-admin/attachments/20161128/b6c94fce/attachment.html>

[Labs-admin] ** RECOVERY alert - ToolLabs/ToolLabs Home Page is OK **

[Labs-admin] RECOVERY alert - ToolLabs/ToolLabs Home Page is OK