[QA] [WikimediaMobile] Visual regression testing

Tue Apr 22 15:58:09 UTC 2014

Yes, but I think it might be possible to share the underlying mechanism of
taking screen shots among all the projects that want to do that, for
whatever purposes they have.

-Chris

On Tue, Apr 22, 2014 at 8:53 AM, Jon Robson <jrobson at wikimedia.org> wrote:

> This wouldn't just be when a test fails. It would be used to detect
> regressions in appearance at given steps....
> e.g. Given I go to the login page When I trigger an error Then I see
> an error message And the login page looks exactly like it was a week
> ago.
>
>
> On Tue, Apr 22, 2014 at 7:20 AM, Chris McMahon <cmcmahon at wikimedia.org>
> wrote:
> >
> >
> > Language people also working on screen shots:
> > https://gerrit.wikimedia.org/r/#/c/127899/
> >
> >
> > On Mon, Apr 21, 2014 at 2:25 PM, Juliusz Gonera <jgonera at wikimedia.org>
> > wrote:
> >>
> >> Nope, it's an early prototype. It basically only makes an updated
> Cucumber
> >> test fail if new screenshot differs from reference screenshot by more
> than
> >> X%. Adding visual comparison of two screenshots would be the next step
> and
> >> would not be too hard. I might work on it one evening this week.
> >>
> >>
> >>
> >> On 04/21/2014 11:42 AM, Tomasz Finc wrote:
> >>>
> >>> Great. Eager to see us test with this.
> >>>
> >>> Do we have any of its early screenshot comparisons up?
> >>>
> >>> --tomasz
> >>>
> >>> On Fri, Apr 18, 2014 at 4:33 PM, Juliusz Gonera <jgonera at wikimedia.org
> >
> >>> wrote:
> >>>>
> >>>> I conducted a little bit of research on visual regression testing,
> >>>> mainly
> >>>> for the mobile team for now, but it could be easily reused for other
> >>>> teams.
> >>>> I had a look at three existing solutions that seem somewhat popular
> and
> >>>> are
> >>>> actively developed:
> >>>>
> >>>> * Wraith (https://github.com/BBC-News/wraith)
> >>>> * PhantomCSS (https://github.com/Huddle/PhantomCSS)
> >>>> * Huxley (https://github.com/facebook/huxley)
> >>>>
> >>>> They all have their own pros and cons, but in my opinion they share
> one
> >>>> important disadvantage: they can't be easily integrated with our
> current
> >>>> browser testing setup. For all the aforementioned tools we would have
> to
> >>>> create from scratch a completely separate set of tests just for visual
> >>>> regression testing instead of extending our existing browser tests.
> >>>>
> >>>> I spent a few hours in my spare time tinkering with an alternative
> idea
> >>>> which would enable us to add visual regression testing to our
> >>>> Cucumber/Watir
> >>>> tests. The result is a small prototype available at
> >>>> https://github.com/jgonera/photographer. There is no docs or anything
> >>>> yet,
> >>>> but I prepared a simple demo patch for MobileFrontend:
> >>>> https://gerrit.wikimedia.org/r/#/c/126878/.
> >>>>
> >>>> The idea is to add a new method for Cucumber steps (snap) that takes a
> >>>> screenshot of the current browser state and compares it with a
> >>>> screenshot
> >>>> taken in one of the previous test runs. To update screenshots that are
> >>>> used
> >>>> as a reference you run tests with env var PHOTOGRAPHER=update. If
> newly
> >>>> taken screenshot differs by too many pixels from an old one, the test
> >>>> will
> >>>> fail.
> >>>>
> >>>> It's still only an early prototype, but I'd appreciate any comments
> >>>> about
> >>>> this idea.
> >>>>
> >>>> --
> >>>> Juliusz
> >>>>
> >>>> _______________________________________________
> >>>> Mobile-l mailing list
> >>>> Mobile-l at lists.wikimedia.org
> >>>> https://lists.wikimedia.org/mailman/listinfo/mobile-l
> >>
> >>
> >>
> >> _______________________________________________
> >> QA mailing list
> >> QA at lists.wikimedia.org
> >> https://lists.wikimedia.org/mailman/listinfo/qa
> >
> >
> >
> > _______________________________________________
> > QA mailing list
> > QA at lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/qa
> >
>
> _______________________________________________
> QA mailing list
> QA at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/qa
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140422/77f91dee/attachment.html>