[Labs-l] replication lag

Mon Nov 23 16:54:44 UTC 2015

Labsdb databases are still suffering from corruption. I am trying to repair
the tables affected. Updates will be at: <
https://phabricator.wikimedia.org/T119315> Expect some downtime in order to
fix the issues.

On Mon, Nov 23, 2015 at 2:01 PM, Magnus Manske <magnusmanske at googlemail.com>
wrote:

> Ah, didn't see this, created a bug report:
> https://phabricator.wikimedia.org/T119382
>
> On Mon, Nov 23, 2015 at 12:30 PM Yetkin Sakal <superyetkin at yahoo.com>
> wrote:
>
>> That is good news for us all, jcrespo.
>>
>> Cheers!
>>
>>
>>
>> On Monday, November 23, 2015 2:00 PM, Jaime Crespo <jcrespo at wikimedia.org>
>> wrote:
>>
>>
>> It seems that 2 out of 3 labs replica servers had crashed, creating
>> corruption in the process. Replica lag is now going down.
>>
>> Lag measuring (with 1 second resolution) is coming soon, will be
>> available as a table, but I hit a bug and it is not being displayed
>> correctly, so it is not yet public.
>> I will announce it as soon as it is fixed.
>>
>>
>> On Sun, Nov 22, 2015 at 10:06 PM, Yetkin Sakal <superyetkin at yahoo.com>
>> wrote:
>>
>> Thanks.
>>
>>
>>
>> On Sunday, November 22, 2015 10:52 PM, John <phoenixoverride at gmail.com>
>> wrote:
>>
>>
>> I have http://tools.wmflabs.org/betacommand-dev/cgi-bin/replag which
>> gives a rough estimate, plus we have
>> https://phabricator.wikimedia.org/T119315
>>
>> On Sun, Nov 22, 2015 at 3:26 PM, Yetkin Sakal <superyetkin at yahoo.com>
>> wrote:
>>
>> Is there anywhere I can keep track of the status pertaining to the
>> replication lag on s3? My tools are showing data from nearly a day ago.
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>>
>>
>>
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>>
>>
>>
>> --
>> Jaime Crespo
>> <http://wikimedia.org>
>>
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>

-- 
Jaime Crespo
<http://wikimedia.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20151123/6c4024c6/attachment-0001.html>