[QA] Browser tests coverage in MobileFrontend stable (Result of spike 1687)

Thu Apr 10 16:02:05 UTC 2014

Update. It's been a bit of a rollercoaster but we are now in a much
better state. Also hopefully with the merging of
https://gerrit.wikimedia.org/r/125204 and
https://gerrit.wikimedia.org/r/125202 all our Jenkins tests builds
will pass again.

Going back to my original mail, here's a few updates on each of the
problems identified.

> Our coverage is actually not too bad, but there is definitely room
> from improvement. That said I have a big concern is around the
> organisation of the tests and how they are written and what is written
> - many of the tests could do with being reworded and a lot of them
> should probably actually be deleted. There is a lot of code
> duplication.

There is now a lot less code duplication. These browser tests have
been massively refactored.

>
> When auditing I found tests scattered all over the place. This
> suggests that we could benefit from reorganising the file structure to
> be more logical, in particular features that relate to special pages
> should have their own folder (this is particularly useful for
> clarifying what tests the watch star and what tests the actual
> watchlist page - Zeljko / Chris is it possible to have subfolders in
> the features directory that contain features?).

This is better but subfolders would still be useful. Željko maybe we
could explore this now?

> I would also suggest the following actions for improving our test coverage:> * Add tests for this page has issues
Provided in https://gerrit.wikimedia.org/r/125211 - a review needed

> * Improve tests for key editing and upload workflows
Our browser tests actually go through the entire workflow now - see
Tests merged in https://gerrit.wikimedia.org/r/#/c/118104/
Our lead photo upload workflow could do with a few more tests though,
although the fact we test uploads on the uploads page which uses the
same code means we are mostly covered.

> * Add tests for full text search support
https://gerrit.wikimedia.org/r/124742 provides this - please
> * Improve the existing watch star tests so they actually check the end result
Merged in https://gerrit.wikimedia.org/r/#/c/123679/

> * Add tests for reference overlay
Merged in Ied8533d42d6b005909347b6c6f7e05bb8cb31364

> * Add tests for toggling
Merged in Iab4449e2cffba61a3934b73cb430c0b2c847146a

> * Address hygiene issues on
> https://www.mediawiki.org/wiki/Mobile/QA_Test_Audit_March_2014
These issues have now been addressed.

This leaves the following features untested - the value of which is debatable:
> * Add tests for error handling on login / account creation
> * Add tests for Language variant support
> * Add tests for logout
> * Add test to check the user can close the left navigation menu
(this one should be trivial to cover)
> * Add tests for Nearby in skins other than Minerva (mobile skin)