Good morning Dan,
Thanks very much for the explanation. Is there a Phabricator task we
can upvote (award a token?) to make this issue more visible?
As always, we really appreciate your help with this.
Best,
Amanda
On Tue, Feb 17, 2015 at 7:20 AM, Dan Andreescu <dandreescu(a)wikimedia.org
Sorry for the trouble, Amanda. The problem is
solely with the
underlying database, which we don't maintain. It's a sanitized replica of
all the changes being made to all the wikis so it's a fairly complicated
piece of infrastructure that sometimes has problems. The folks who
maintain it are aware of the issues, but we'll continue representing them
until they're solved.
On Mon, Feb 16, 2015 at 3:49 PM, Amanda Bittaker <
abittaker(a)wikimedia.org> wrote:
> Oop, thanks for the ping, Nuria. Wikimetrics seems to be working
> better now. I still get failures, especially when running three or four
> reports in one batch, but the reports work if you rerun them (sometimes a
> couple times.)
>
> I'm still getting "PENDING"s that turn into "FAILURE"s
sometimes,
> which I just noticed for the first time last Thursday. Also, sometimes the
> "FAILURE"s change position in the Current Report Inbox list, moving up or
> down a spot. Not sure if that helps diagnose what might be happening...
>
> In any case, Wikimetrics is mostly functioning but seems to be having
> recurring troubles that sometimes blow up to freeze the whole tool. It
> would be great to resolve the troubles before the next explosion--is there
> anything I can do to help? Dan H and I still have plenty of reports to
> run, we can keep you updated on the reports ran and failure rate while you
> are fixing, if that would be useful.
>
> Many thanks,
> Amanda
>
>
> On Mon, Feb 16, 2015 at 10:15 AM, Nuria Ruiz <nuria(a)wikimedia.org>
> wrote:
>
>> Ping ....
>>
>> On Fri, Feb 13, 2015 at 2:19 PM, Nuria Ruiz <nuria(a)wikimedia.org>
>> wrote:
>>
>>> Amanda,
>>>
>>> Looks like wikimetrics was able to run automatic reports last night
>>> w/o big issues, are your reports still failing?
>>>
>>> Thanks,
>>>
>>> Nuria
>>>
>>> On Thu, Feb 12, 2015 at 1:42 PM, Amanda Bittaker <
>>> abittaker(a)wikimedia.org> wrote:
>>>
>>>> Alright, thanks so much for your help once again, Nuria.
>>>>
>>>> If there's anything I can do or any information I can contribute,
>>>> please don't hesitate to ping me.
>>>>
>>>> Best,
>>>> Amanda
>>>>
>>>>
>>>>
>>>> On Thu, Feb 12, 2015 at 1:36 PM, Nuria Ruiz <nuria(a)wikimedia.org>
>>>> wrote:
>>>>
>>>>> DB connections in labs look to be failing, unfortunately I think
>>>>> besides asking for help on the labs list there is not much we can do
there.
>>>>> I will start a thread on this regard.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Nuria
>>>>>
>>>>> On Thu, Feb 12, 2015 at 1:32 PM, Amanda Bittaker <
>>>>> abittaker(a)wikimedia.org> wrote:
>>>>>
>>>>>> Thanks so much for the quick response, Nuria.
>>>>>>
>>>>>> I ran the exact same reports on the same cohort as one of the
>>>>>> last batches that were failing. Last time 2/4 of the reports
failed, when
>>>>>> I reran the individually they succeeded. (But they don't
always, I reran
>>>>>> one report 3 times this morning before it worked.) This time,
my failure
>>>>>> rate got worse: 4/4 failed, although they said
"PENDING" for a few seconds
>>>>>> first, which is new.
>>>>>>
>>>>>> Is that useful information? Please do let me know what else I
>>>>>> can do to help solve this.
>>>>>>
>>>>>> Thanks again,
>>>>>> Amanda
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 12, 2015 at 1:09 PM, Jonathan Morgan <
>>>>>> jmorgan(a)wikimedia.org> wrote:
>>>>>>
>>>>>>> Thanks Nuria!
>>>>>>>
>>>>>>> On Thu, Feb 12, 2015 at 12:57 PM, Nuria Ruiz <
>>>>>>> nuria(a)wikimedia.org> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> If so a cohort + report to repro will be most useful.
>>>>>>>>
>>>>>>>
>>>>>>> Translation:* try to run the exact same reports on the same
>>>>>>> cohort again, to see if the same metrics fail. Let us know
what you find. ;)
>>>>>>>
>>>>>>> Same goes for anyone else who experiences these issues: the
more
>>>>>>> details we (users) can provide the engineers, the more
effective they can
>>>>>>> be at diagnosing and addressing the problems.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> - J
>>>>>>>
>>>>>>> *for anyone who is not 100% familiar with that hip, new
software
>>>>>>> engineering lingo
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Nuria
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Feb 12, 2015 at 12:35 PM, Dan Andreescu <
>>>>>>>> dandreescu(a)wikimedia.org> wrote:
>>>>>>>>
>>>>>>>>> Recently there was a restart of the labsdb cluster.
I'm sorry
>>>>>>>>> but I don't have time to check on it, but I bet
that's the problem. I'm
>>>>>>>>> off tomorrow unfortunately but I'll try to check
tomorrow night :( I hope
>>>>>>>>> someone else beats me to it.
>>>>>>>>>
>>>>>>>>> On Thu, Feb 12, 2015 at 3:20 PM, Jonathan Morgan
<
>>>>>>>>> jmorgan(a)wikimedia.org> wrote:
>>>>>>>>>
>>>>>>>>>> (ping Kevin and Dan A.)
>>>>>>>>>>
>>>>>>>>>> Hi Amanda, I've had some problems with report
failures
>>>>>>>>>> recently when I ran a few test cohorts. On the
same cohort, when I ran
>>>>>>>>>> multiple concurrent reports (say, bytes added,
edits, and pages created),
>>>>>>>>>> some would fail and others succeed. It wasn't
clear what the issue was.
>>>>>>>>>>
>>>>>>>>>> - J
>>>>>>>>>>
>>>>>>>>>> On Thu, Feb 12, 2015 at 12:16 PM, Amanda Bittaker
<
>>>>>>>>>> abittaker(a)wikimedia.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello all,
>>>>>>>>>>>
>>>>>>>>>>> I am getting failures again, both when
uploading cohorts and
>>>>>>>>>>> running reports. Strangely, it seems the
more reports you try to run in
>>>>>>>>>>> one batch the less likely it is any report
will succeed.
>>>>>>>>>>>
>>>>>>>>>>> Is anyone else having these problems again?
Wonderful
>>>>>>>>>>> Analytics people, could you please work your
magic again?
>>>>>>>>>>>
>>>>>>>>>>> Many thanks,
>>>>>>>>>>> Amanda
>>>>>>>>>>>
>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>> Wikimetrics mailing list
>>>>>>>>>>> Wikimetrics(a)lists.wikimedia.org
>>>>>>>>>>>
https://lists.wikimedia.org/mailman/listinfo/wikimetrics
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Jonathan T. Morgan
>>>>>>>>>> Community Research Lead
>>>>>>>>>> Wikimedia Foundation
>>>>>>>>>> User:Jmorgan (WMF)
>>>>>>>>>>
<https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)>
>>>>>>>>>> jmorgan(a)wikimedia.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Wikimetrics mailing list
>>>>>>>>> Wikimetrics(a)lists.wikimedia.org
>>>>>>>>>
https://lists.wikimedia.org/mailman/listinfo/wikimetrics
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Jonathan T. Morgan
>>>>>>> Community Research Lead
>>>>>>> Wikimedia Foundation
>>>>>>> User:Jmorgan (WMF)
>>>>>>> <https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)>
>>>>>>> jmorgan(a)wikimedia.org
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Wikimetrics mailing list
>>>>>>> Wikimetrics(a)lists.wikimedia.org
>>>>>>>
https://lists.wikimedia.org/mailman/listinfo/wikimetrics
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
> _______________________________________________
> Wikimetrics mailing list
> Wikimetrics(a)lists.wikimedia.org
>
https://lists.wikimedia.org/mailman/listinfo/wikimetrics
>
>
_______________________________________________
Wikimetrics mailing list
Wikimetrics(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimetrics
_______________________________________________
Wikimetrics mailing list
Wikimetrics(a)lists.wikimedia.org