[QA] Notifying people when integration tests

Fri Mar 7 15:23:21 UTC 2014

This looks like a good time to fork this conversation as this is a good
problem to fix. How can we notify developers when they break other things
in the stack and how?
On 7 Mar 2014 05:36, "Bartosz Dziewoński" <matma.rex at gmail.com> wrote:

> (continued, about the browser testing)
>
> (tl;dr where are the tests and how do I know they fail?)
>
> So, we have some slick browser tests. Awesome! But what's not good is
> that the tests run off-site, the results are not reported back to
> gerrit nor Bugzilla (unless someone manually files a bug, usually
> Chris) not IRC nor anywhere else, and are generally non-discoverable
> until someone shouts at you for breaking them. (As Tim guessed, I did
> not know about any failures until Jon told me.)
>
> In fact, I still have no idea what exactly the tests encompass (I've
> heard about some browser tests for VE because I lurk a lot, never
> heard of any for core) or where to find them or how to run them.
> Either I'm slow or we have a serious documentation failure here.
>
> Can something be done about it? Can we have the results reported
> somewhere visible – preferably to gerrit, as jenkins already reports
> some post-merge checks there? Or maybe we can have automatically filed
> bug reports if the build breaks? A bot reporting test status on
> #wikimedia-dev? Anything?
>
> (I understand that the tests take too long to run them in the
> pre-merge checks.)
>
> (Jon proposed reverting problematic changes outright, but to me that
> seems like a bit of an overreaction – bugs in tests and false
> positives happen, let's not make a huge fuss out of that.)
>
> (to be continued: about the deployment)
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140307/48585f52/attachment.html>