[QA] [WikimediaMobile] Qualifiers for selecting test articles for vagrant role

Wed Jul 22 19:27:24 UTC 2015

On Wed, Jul 22, 2015 at 2:51 PM, Dan Duvall <dduvall at wikimedia.org> wrote:

> On Wed, Jul 22, 2015 at 10:43 AM, Brian Gerstle <bgerstle at wikimedia.org>
> wrote:
>>
>>
>> Also, one neat thing I saw done at Spotify was a "Test Data Service."
>> This service's job was to abstract away concrete test data from its
>> qualities.  This was mostly used to acquire test users, thus allowing
>> testers to declaratively acquire test data.  For example, you could say "I
>> want a free user with X flags" or "I want a premium user who has tracks in
>> their starred playlist."  The service's other job was to lock users to
>> prevent parallel tests from corrupting a shared test user's state (also
>> resulting in a false negative).
>>
>
> That's really interesting. Do you have a link to further reading/docs?
>

Nothing about the the Test Data Service (referred to commonly as TDS)
specifically. They're main drum to beat was Model Based Testing
<http://www.cs.tut.fi/tapahtumat/testaus12/kalvot/Karl_ta.pdf> (mostly
driven by Kristian Karl).  Towards the end of my time there, I was talking
to people about creating users on-demand to side-step locking and
hand-curating test users, which it seems like you're nudging us towards.
I'd be happy to reach out and see if they'd be willing to talk to us more
about what they're doing now.

>
> I had thought of exploring a simple rotating queue of users for Beta
> Cluster a while back to improve isolation, but given the variance of
> setup/teardown operations across different test suites, it seemed like it
> still might be difficult to maintain a clean state after a while.
>

Deadlocks (or missed unlocks) were occasionally a problem, which is why
locks were always given a specific life span (e.g. lock for 30
minutes—hopefully your tests finish first!!).

> Maybe once we have a proper staging cluster and the liberty to wipe the
> database clean periodically, this type of setup would be worth exploring
> further.
>
> For now, I think making strides toward more isolated environments and
> using the API for creation of initial users/pages seems to be the right
> approach. It's relatively inexpensive—probably the least expensive aspect
> of browser-driven tests anyway—and can be easily extended in the framework
> or in test helpers for specific test cases, e.g. creating Flow topics, etc.
>

This is where Spotify was going around the time I left. They were very SOA,
and trying to push adoption of docker for testing and production.  There
were some promising results of orchestrating an isolated ecosystem of
services on demand, testing them, then tearing it all down.  Combine this
with pre-baked Docker containers and I think you see where it could lead.

>
> Keeping the provisioning of preconditions in code, closer to the test
> implementation, also makes tests easier to reason about—you known that test
> case A depends on the precondition that the user that has 3 edits because
> the test case creates the user and makes 3 edits, instead of having to
> inspect a preexisting account or rely on comments that may not exist.
>

As usual, all about trade-offs.  If you want to keep your tests short
(ideally you have a lot of them), side-loading or pre-baking service
images/containers/etc. with data can be a big time saver.

>
>
>> We might want something similar for retrieving and locking test articles
>> for editing, but this might not be necessary if our vagrant/virtualization
>> infrastructure is mature enough to ensure tests are isolated.
>>
>
> The environment that's setup for CI browser tests (per patch tests, not
> the daily runs) is completely isolated for a single project/patchset.[1]
> I'd like to push this further though, by isolating pages to unique
> prefixes/namespaces per scenario, so that we might run scenarios in
> parallel. Another possibility would be to use a multiwiki setup with *n* wikis and
> simply isolate each subset of scenarios to one of them*.* Of course you'd
> have to be using our CI infrastructure to benefit from the latter. :P
> (Couldn't resist ... in all seriousness we could figure out a way to
> generalize CI-type MW setup as libraries/tools.)
>
> [1] https://www.mediawiki.org/wiki/Continuous_integration/Browser_tests
>
> --
> Dan Duvall
> Automation Engineer
> Wikimedia Foundation <http://wikimediafoundation.org>
>

-- 
EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
IRC: bgerstle
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20150722/f3b7f00d/attachment-0001.html>