[Labs-l] wiki replication stopped since September

Thu Nov 27 00:49:19 UTC 2014

On Wed, Nov 26, 2014 at 10:58 PM, Krinkle <krinklemail at gmail.com> wrote:

> Seems to be at least 62 days behind on commonswiki
>
> tin.eqiad.wmnet$ sql commonswiki
> rc_id: 142360816
> rc_timestamp: 2014-11-26 12:55:31
>
> tools-login$ sql commonswiki
> rc_id: 136903866
> rc_timestamp: 2014-09-24 12:51:42
>
> See also https://tools.wmflabs.org/unpatrollededitstats/commonswiki/
>

As mentioned in this thread and elsewhere, the labs replicas are being
resynced and an upgraded package with a replication bug fix prepared
(upstream release occurred Tuesday). Presently s1 and s2 are done, s4 and
s5 are in progress, and s3 s6 s7 will follow up.

However, the above example demonstrates a recent client-side configuration
change that I was not aware of: "sql commonswiki" is apparently hitting
labsdb1003 where it used to hit labsdb1002 (the canonical s4.labsdb replica
since forever). For a number of reasons this change breaks some assumptions
made in the resync and is now exposing an old commonswiki snapshot.

Eg, on labsdb1002:

labsdb1002 commonswiki> select max(rc_timestamp) from recentchanges;

| 20141127004237    |

Coren is helping to chase this down right now.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20141127/3196aa00/attachment.html>