[Engineering] mira/tin mediawiki-staging incident report

Thu Feb 4 15:28:46 UTC 2016

Here's the full incident report from Tuesday:

https://wikitech.wikimedia.org/wiki/Incident_documentation/20160202-deployment-server-loss

The tldr: a server (tin) was pooled was a deployment co-master before it
was totally ready. It hosed
the sane master (mira) trying to sync the two. Lots of actionables.

Thanks to Joe, Jaime and Antoine for spotting so quick and staying up
basically all day long to help
fix it. Tyler, Ori, Daniel Z, Chase all helped along the way too :)

As always: it's an incident report on a wiki, please feel free to chime in
and add extra details if you
see something missing.

-Chad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/engineering/attachments/20160204/e8c8691c/attachment.html>