[Engineering] 1.27.0-wmf.23 rolled back from Wikipedias

Tue May 10 07:30:14 UTC 2016

My theory seems to have been confirmed by the backport of the
NavigationTiming change to wmf.22, 2 hours before wmf.23 was redeployed.
The backport caused the same kind of 100ms-ish spike on coal.firstPaint as
the one we saw during the initial wmf.23 deployment.

Independently of that, the wmf.23 redeploy didn't seem to cause any
additional (apparent or real) firstPaint regression.

Going forward, the new coal.firstPaint figure is more accurate, as it
doesn't include junk negative values anymore (which we believe came from
user agents that don't comply to the NavigationTiming standard properly).

Someone from releng might correct me if this is no longer true, but given
that wmf.23 was redeployed yesterday, we should be back on track for the
regular deploy train schedule this week.

On Mon, May 9, 2016 at 7:04 PM, Antoine Musso <amusso at wikimedia.org> wrote:

> Le 09/05/2016 15:17, Gilles Dubuc a écrit :
> > Investigation and likely explanation on
> > https://phabricator.wikimedia.org/T134553
> >
> > tl;dr: the source of the spike seems to be in the instrumentation and
> > the newer higher values should be closer to reality.
> >
> > It is safe to roll out 1.27-wmf.23 ASAP. Releng, please let me know when
> > that will be, so that I can keep an eye on things to confirm my findings.
> >
> > Sorry about the inconvenience!
>
> It is better safe than sorry :-}
>
> I am very happy we detect performance regression and end up rollbacking
> a deployment because of that.  That is an indication we have largely
> improved over the last few years.
>
>
>
> --
> Antoine "hashar" Musso
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/engineering/attachments/20160510/e6b34e50/attachment-0001.html>