On Wed, Jul 22, 2015 at 2:51 PM, Dan Duvall <dduvall(a)wikimedia.org> wrote:
On Wed, Jul 22, 2015 at 10:43 AM, Brian Gerstle
<bgerstle(a)wikimedia.org>
wrote:
Also, one neat thing I saw done at Spotify was a "Test Data Service."
This service's job was to abstract away concrete test data from its
qualities. This was mostly used to acquire test users, thus allowing
testers to declaratively acquire test data. For example, you could say "I
want a free user with X flags" or "I want a premium user who has tracks in
their starred playlist." The service's other job was to lock users to
prevent parallel tests from corrupting a shared test user's state (also
resulting in a false negative).
That's really interesting. Do you have a link to further reading/docs?
Nothing about the the Test Data Service (referred to commonly as TDS)
specifically. They're main drum to beat was Model Based Testing
<http://www.cs.tut.fi/tapahtumat/testaus12/kalvot/Karl_ta.pdf> (mostly
driven by Kristian Karl). Towards the end of my time there, I was talking
to people about creating users on-demand to side-step locking and
hand-curating test users, which it seems like you're nudging us towards.
I'd be happy to reach out and see if they'd be willing to talk to us more
about what they're doing now.
I had thought of exploring a simple rotating queue of users for Beta
Cluster a while back to improve isolation, but given the variance of
setup/teardown operations across different test suites, it seemed like it
still might be difficult to maintain a clean state after a while.
Deadlocks (or missed unlocks) were occasionally a problem, which is why
locks were always given a specific life span (e.g. lock for 30
minutes—hopefully your tests finish first!!).
Maybe once we have a proper staging cluster and the
liberty to wipe the
database clean periodically, this type of setup would be worth exploring
further.
For now, I think making strides toward more isolated environments and
using the API for creation of initial users/pages seems to be the right
approach. It's relatively inexpensive—probably the least expensive aspect
of browser-driven tests anyway—and can be easily extended in the framework
or in test helpers for specific test cases, e.g. creating Flow topics, etc.
This is where Spotify was going around the time I left. They were very SOA,
and trying to push adoption of docker for testing and production. There
were some promising results of orchestrating an isolated ecosystem of
services on demand, testing them, then tearing it all down. Combine this
with pre-baked Docker containers and I think you see where it could lead.
Keeping the provisioning of preconditions in code, closer to the test
implementation, also makes tests easier to reason about—you known that test
case A depends on the precondition that the user that has 3 edits because
the test case creates the user and makes 3 edits, instead of having to
inspect a preexisting account or rely on comments that may not exist.
As usual, all about trade-offs. If you want to keep your tests short
(ideally you have a lot of them), side-loading or pre-baking service
images/containers/etc. with data can be a big time saver.
We might want something similar for retrieving
and locking test articles
for editing, but this might not be necessary if our vagrant/virtualization
infrastructure is mature enough to ensure tests are isolated.
The environment that's setup for CI browser tests (per patch tests, not
the daily runs) is completely isolated for a single project/patchset.[1]
I'd like to push this further though, by isolating pages to unique
prefixes/namespaces per scenario, so that we might run scenarios in
parallel. Another possibility would be to use a multiwiki setup with *n* wikis and
simply isolate each subset of scenarios to one of them*.* Of course you'd
have to be using our CI infrastructure to benefit from the latter. :P
(Couldn't resist ... in all seriousness we could figure out a way to
generalize CI-type MW setup as libraries/tools.)
[1]
https://www.mediawiki.org/wiki/Continuous_integration/Browser_tests
--
Dan Duvall
Automation Engineer
Wikimedia Foundation <http://wikimediafoundation.org>
--
EN Wikipedia user page:
https://en.wikipedia.org/wiki/User:Brian.gerstle
IRC: bgerstle