[QA] Visual regression testing

Fri Apr 18 23:33:43 UTC 2014

I conducted a little bit of research on visual regression testing, 
mainly for the mobile team for now, but it could be easily reused for 
other teams. I had a look at three existing solutions that seem somewhat 
popular and are actively developed:

* Wraith (https://github.com/BBC-News/wraith)
* PhantomCSS (https://github.com/Huddle/PhantomCSS)
* Huxley (https://github.com/facebook/huxley)

They all have their own pros and cons, but in my opinion they share one 
important disadvantage: they can't be easily integrated with our current 
browser testing setup. For all the aforementioned tools we would have to 
create from scratch a completely separate set of tests just for visual 
regression testing instead of extending our existing browser tests.

I spent a few hours in my spare time tinkering with an alternative idea 
which would enable us to add visual regression testing to our 
Cucumber/Watir tests. The result is a small prototype available at 
https://github.com/jgonera/photographer. There is no docs or anything 
yet, but I prepared a simple demo patch for MobileFrontend: 
https://gerrit.wikimedia.org/r/#/c/126878/.

The idea is to add a new method for Cucumber steps (snap) that takes a 
screenshot of the current browser state and compares it with a 
screenshot taken in one of the previous test runs. To update screenshots 
that are used as a reference you run tests with env var 
PHOTOGRAPHER=update. If newly taken screenshot differs by too many 
pixels from an old one, the test will fail.

It's still only an early prototype, but I'd appreciate any comments 
about this idea.

-- 
Juliusz