[QA] Failing MobileFrontend browser tests

Wed Jul 16 21:53:25 UTC 2014

On Wed, Jul 16, 2014 at 2:37 PM, S Page <spage at wikimedia.org> wrote:

So I filed  *Bug 68125*
> <https://bugzilla.wikimedia.org/show_bug.cgi?id=68125> - browser tests
> failing with "getaddrinfo: Name or service not known (SocketError)"
>

This seems to be in every case some test not sending any command to
SauceLabs in 90 seconds.  After 90 seconds with no activity Sauce deletes
the VM and the test has nothing to talk to.

This is likely a problem with Jenkins.  However, I have also seen this with
poorly-considered API calls that take longer than 90 seconds, so do check
to make sure the test is not doing something that takes a really long time.

> [Sage manager] suggested
>
>> can we change the wait to 60 seconds and call it good?  How was 5 seconds
>> arrived at as the time for an automated test to fail?
>>
>
> The other Firefox test failure on that run was adding a topic took 6
> seconds, thus triggering
>
> timed out after 5 seconds, Element still visible after 5 seconds (Watir::Wait::TimeoutError)
>
> Flow tests often fail with these timeouts yet the expected result appears
> is in the screencast or ends up on the test page. So yes, increasing the
> wait timeout to 10 seconds would cut down our false failures.
>

Of course, it would be better if we did not have any operations that take
over 5 seconds to complete, that seems a long time, even on beta labs.  For
the operations that we know take longer, the convention so far has been to
use an argument in seconds to "when_present" in individual test steps, for
example

"on(FooPage).my_element.when_present(10).click"  to poll for my_element for
10 seconds and I have even used
"on(FooPage).my_element.when_present(15).click" for some tedious VE commands

The vast, vast majority of our test steps across all repos take less than 5
seconds to complete.  I would oppose a global 60-second timeout.  I might
entertain a global 10-second timeout.  But I would rather know about
chronically poor performance on particular aspects of features than not
know.

> QA folk, is there a way to "grep" all browser tests for gettaddrinfo and
> "timed out after 5 seconds" to see if there's a pattern to when and how
> often they occur?
>

That's a good question.  I would assume Jenkins has some raw output stored
somewhere that might lend itself to this kind of thing, Antoine might know
better.
-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140716/ea43dd45/attachment-0001.html>