There's this graph: https://graphite.wikimedia.org/render/?width=586&height=308&_salt=1… <https://graphite.wikimedia.org/render/?width=586&height=308&_salt=1418343627.977&from=-1weeks&target=movingMedian(diffSeries(eventlogging.overall.raw.rate%2Ceventlogging.overall.valid.rate)%2C20)> The key is 'diffSeries(eventlogging.overall.raw.rate,eventlogging.overall.valid.rate)', which gets you the rate of invalid events per second. It is not broken down by schema, though.

We can't write invalid events to a database -- at least not the same way we write well-formed events. The table schema is derived from the event schema, so an invalid event would violate the constraints of the table as well.

It's possible (and easy) to set something up that watches invalid events in real-time and does something with them. The question is: what? E-mail an alert? Produce a daily report? Generate a graph? If you describe how you’d like to consume the data, I can try to hash out an an implementation with Nuria and Christian.

Besides the ability of testing in beta labs and the monitoring that ori highlited the incoming raw stream of events is available in 1003/1002 on port 8600.

From 1002 or 1003 you can run: zsub vanadium.eqiad.wmnet:8600 and see the incoming stream.

But I see that meanwhile a Phabricator task got added, and I guess I am alone with my judgement :-)

Hi Dario, On Thu, Dec 11, 2014 at 04:11:49PM -0800, Dario Taraborelli wrote: Thanks! Full ACK. However, that sounds like we're only talking about schemas where the collection code got tested using Vagrant or beta, and is known to work on the relevant portion of the traffic. And since you say that it's on browsers/platforms that we don't necessarily test for internally, I assume we're actually talking only about a small fraction of the traffic. I assume that scope for the rest of the reply. * Urgent, ad-hoc needs For urgent, ad-hoc needs, (which should happen really seldom, given the scope), ping us in IRC in #wikimedia-analytics. At least qchris, milimetric, and nuria should be able to ssh into vanadium and can take a look right away. If none of them are around, Ops of course have access to the relevant files on vanadium [1]. And since we're in the case of urgent, ad-hoc needs, I am sure they'd help out. * Not so urgent needs For not so urgent needs, since it's only a small fraction of the traffic, I am not sure real-time need is worth it. Sure it would be nice to provide near real-time access to those files, but we should also get the cluster into a more reliable state, implement UDFs for researches to make their lives easier, and get the data-warehouse up and running ;-) But I see that meanwhile a Phabricator task got added, and I guess I am alone with my judgement :-) Have fun, Christian [1] Either /srv/log/eventlogging/client-side-events.log or /srv/log/eventlogging/server-side-events.log depending on the kind of event you're looking for. -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ --------------------------------------------------------------- _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

I closed the Phabricator task with a links to this thread and the wikitech doc for testing on beta cluster.

I share Christian's concerns - Dario/Leila - can you comment based on your recent experiences with WikiGrok? Thanks -Toby christian(a)quelltextlich.at&gt; wrote: wikitech https://lists.wikimedia.org/pipermail/analytics/2014-December/002884.html _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

QA in beta labs is good but not enough. We still need to do QA when a

On Mon, Dec 15, 2014 at 10:06 AM, Toby Negrin &lt;tnegrin(a)wikimedia.org&gt; wrote: I agree with Christian. QA in beta labs is good but not enough. We still need to do QA when a feature goes to production and currently, it's very hard to figure out if there's a problem with logging. An example: While testing WikiGrok in production, we learned that after some point tests from Firefox browser from my machine were not logged. We did not get any errors for this. I found out about this because I was trying to manually make a trace of activities and see if I can stitch them together and make sense of them. We eventually figured out what was going on in that case [1], but it concerns me that there may be other important events that we don't log in the DB and we never know that we're not logging. Leila [1] https://lists.wikimedia.org/pipermail/analytics/2014-December/002864.html _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

I'd like to move this to a video conference call between analytics developers and analytics engineering to come to a mutual understanding of what the current pain points are and what's the biggest priority.

On Mon, Dec 15, 2014 at 4:37 PM, Nuria Ruiz &lt;nuria(a)wikimedia.org <javascript:_e(%7B%7D,'cvml','nuria@wikimedia.org');>> wrote: > > >QA in beta labs is good but not enough. We still need to do QA when a > feature goes to production and currently > This is true but at the same time, I do not see anything in the > description of your FF events that could not be tested on beta-labs. If we > are talking add-block that can be tested even earlier, vagrant will be a > fine venue. All the issues related to the client (browser) not emitting > events can be tested on the development environment with ease. > > > > On Mon, Dec 15, 2014 at 4:18 PM, Leila Zia &lt;leila(a)wikimedia.org > <javascript:_e(%7B%7D,'cvml','leila@wikimedia.org');>> wrote: >> >> >> On Mon, Dec 15, 2014 at 10:06 AM, Toby Negrin &lt;tnegrin(a)wikimedia.org >> <javascript:_e(%7B%7D,'cvml','tnegrin@wikimedia.org');>> wrote: >>> >>> I share Christian's concerns - >>> >>> Dario/Leila - can you comment based on your recent experiences with >>> WikiGrok? >>> >> >> I agree with Christian. >> >> QA in beta labs is good but not enough. We still need to do QA when a >> feature goes to production and currently, it's very hard to figure out if >> there's a problem with logging. An example: >> >> While testing WikiGrok in production, we learned that after some point >> tests from Firefox browser from my machine were not logged. We did not get >> any errors for this. I found out about this because I was trying to >> manually make a trace of activities and see if I can stitch them together >> and make sense of them. We eventually figured out what was going on in that >> case [1], but it concerns me that there may be other important events that >> we don't log in the DB and we never know that we're not logging. >> >> Leila >> [1] >> https://lists.wikimedia.org/pipermail/analytics/2014-December/002864.html >> >> >>> >>> Thanks >>> >>> -Toby >>> >>> >>> > On Dec 15, 2014, at 9:42 AM, Christian Aistleitner < >>> christian(a)quelltextlich.at >>> <javascript:_e(%7B%7D,'cvml','christian@quelltextlich.at');>> wrote: >>> > >>> > Hi, >>> > >>> >> On Mon, Dec 15, 2014 at 08:34:39AM -0800, Kevin Leduc wrote: >>> >> I closed the Phabricator task with a links to this thread and the >>> wikitech >>> >> doc for testing on beta cluster. >>> > >>> > I am fine with keeping the task closed. >>> > >>> > But I am somewhat surprised to see beta mentioned in the >>> > resolution. Note that Dario's request set scope as [1] >>> > >>> > However, there are types of data quality issues that we only >>> > discover when collecting data at scale and in the wild (on >>> > browsers/platforms that we don’t necessarily test for internally). >>> > >>> > . That's a valid scope, but from my point of view, beta does not match >>> > that scope. >>> > >>> > Neither is beta large scale, nor is it hammered on with crazy devices. >>> > >>> > Beta is just a halfing the distance between EventLogging's devserver >>> > (Vagrant!) and production. >>> > >>> > Have fun, >>> > Christian >>> > >>> > >>> > >>> > [1] >>> https://lists.wikimedia.org/pipermail/analytics/2014-December/002884.html >>> > >>> > >>> > >>> > -- >>> > ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- >>> > Companies' registry: 360296y in Linz >>> > Christian Aistleitner >>> > Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at >>> <javascript:_e(%7B%7D,'cvml','christian@quelltextlich.at');> >>> > 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 >>> > Fax: +43 7946 / 20 5 81 >>> > Homepage: http://quelltextlich.at/ >>> > --------------------------------------------------------------- >>> > _______________________________________________ >>> > Analytics mailing list >>> > Analytics(a)lists.wikimedia.org >>> <javascript:_e(%7B%7D,'cvml','Analytics@lists.wikimedia.org');> >>> > https://lists.wikimedia.org/mailman/listinfo/analytics >>> >>> _______________________________________________ >>> Analytics mailing list >>> Analytics(a)lists.wikimedia.org >>> <javascript:_e(%7B%7D,'cvml','Analytics@lists.wikimedia.org');> >>> https://lists.wikimedia.org/mailman/listinfo/analytics >>> >> >> _______________________________________________ >> Analytics mailing list >> Analytics(a)lists.wikimedia.org >> <javascript:_e(%7B%7D,'cvml','Analytics@lists.wikimedia.org');> >> https://lists.wikimedia.org/mailman/listinfo/analytics >> >> > _______________________________________________ > Analytics mailing list > Analytics(a)lists.wikimedia.org > <javascript:_e(%7B%7D,'cvml','Analytics@lists.wikimedia.org');> > https://lists.wikimedia.org/mailman/listinfo/analytics > >