[Engineering] 1.27.0-wmf.23 rolled back from Wikipedias

Fri May 6 06:43:20 UTC 2016

The roll-out of 1.27.0-wmf.23 to all Wikipedias coincided with a
significant (~100-150ms) increase in first paint time for Wikimedia page
views. First paint time is the time it takes the user to see anything on
the screen after clicking on a link. Keeping it low is crucial to good user
experience, and this is a severe regression, so I rolled back wmf.23, in
coordination with Chad and Tyler from Release Engineering.

In the graph below, the green line shows median first paint time in
milliseconds over the course of the past 24 hours. The blue line shows what
the value was that time the day before. You can see they diverge shortly
after 19:00 UTC and converge around 4:20 UTC.

I have not investigated any further than that yet, but the train is blocked
on isolating and fixing this regression (or disproving it, if it is a bug
in the instrumentation). This is tracked in Phabricator as
https://phabricator.wikimedia.org/T134553. You are encouraged to help out.
:)

Ori
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/engineering/attachments/20160505/0e61e289/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: first-paint-2016-05-05.png
Type: image/png
Size: 108339 bytes
Desc: not available
URL: <https://lists.wikimedia.org/pipermail/engineering/attachments/20160505/0e61e289/attachment-0001.png>