Hello all,
This is a quick email to give you an update on the DevOps Sprint[0].
== What we focused on ==
As you can see on the project page, there were three main areas of
(attempted) focus: Monitoring, Cache improvements, and Deployment. That
may sound like a lot, and it was. It was too much, in fact. There were
also items under those categories that weren't as well defined as they
should have been (see below).
== What we did ==
* We (Tim and Brad) did fix a couple long standing annoying cache
improvement bugs ([1] and [2]).
* We (Ori) improved scap a bit
** IRC logging of commits
** Reporting time to graphite - This is useful in that it gives us a
real baseline to judge any future improvements against.
** Made it more atomic by having rsync defer updating of files until
they've all been transmitted to the server, thus minimizing the
amount of time code is in an inconsistent state
** Made rsync use --compress which helped in over scap time, but we
don't know exactly how much (the 'time reporting' thing above
happened after)
* We (Aaron, Antoine, Bryan, Ori) setup Logstash
** This is still ongoing, but it is close to being setup in production.
** There is also a labs instance[3] you can look at that uses log info
from the Beta Cluster.
* We (Ori, Bryan, Aaron) wrote an RFC for adding structured logging to
MediaWiki [4]
== Things we didn't get to ==
Much of the deployments category. This is mostly due to the above
mentioned issues (too much, too broad).
Specifically:
* Trebuchet (aka: git-deploy)
** Ryan put a lot of work into Trebuchet during this sprint which
brought it to a great position
** We (Platform) weren't able to devote the requisite time to it during
the high level of churn (which was good).
** We (Platform) hope to work on this in the coming months from the
deployer's experience end (ie: not the backend of transferring files
around, specifically): more on this later[5].
Best,
Greg and the DevOps Sprint team.
[0]
https://www.mediawiki.org/wiki/DevOps_Sprint_2013
[1]
https://bugzilla.wikimedia.org/show_bug.cgi?id=5382
[2]
https://bugzilla.wikimedia.org/show_bug.cgi?id=27935
[3]
http://logstash.instance-proxy.wmflabs.org/#/dashboard
user/password:
https://office.wikimedia.org/wiki/User:BDavis_%28WMF%29/logstash
[4]
https://www.mediawiki.org/wiki/Requests_for_comment/Structured_logging
[5] Basically, there will be a documentation sprint to produce a
overview of the current dev and deploy process followed by a "where
we want to go" high level thing (text or flowchart, something). From
that we can create a pretty solid design doc for deployment 2.0.
--
| Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |