[Labs-l] Labs down?

Tue May 20 15:40:28 UTC 2014

Hello!

nginx barfed this time because it ran out of open file descriptors.
I've prepared (and Andrew has merged) a patch that has raised this
limit, and also a bunch of other tuneups that should make it perform
better. Also moved off nginx 1.7.0 to a supposedly more 'stable'
1.6.0, so that might help as well.

Apologies for the intermittent downtime, which don't seem to be happening now.

There was not much load on the box throughout, so I don't think a new
box will help much. In the future we might have two VMs load balanced
via DNS, and I think that's an easier / better solution than getting a
full on dedicated box.

Root cause was probably organic traffic increases and the upgrade to
nginx 1.7.0 - it let us stop using a homebuilt package and had SPDY/3
support. 1.6.0 has it too, and seems more stable, so have switched to
1.6.0.

I'll add some external monitoring to alert me if the proxy goes down
and should be able to respond faster, if it happens again.

Thanks!

-- 
Yuvi Panda T
http://yuvi.in/blog