[QA] Browser test are now throttled

Mon Aug 25 15:20:52 UTC 2014

On Fri, Aug 22, 2014 at 12:19 PM, Antoine Musso <hashar+wmf at free.fr> wrote:

> Hello,
>
> The browser tests running on Jenkins are often failing due to some
> mysterious errors (timeout, network access, machine overloaded, race
> condition or whatever).
>

It's not all that mysterious:  most of these failures are because the Sauce
VM does not see any instruction in a 90-second window.  And we see the
90-second timeouts mostly when we are running 10 simultaneous Selenium
tests at busy times (e.g. during the US work day) less often at slow times
(overnight US time) and rarely when running single test suites at times
when Jenkins is not busy.

> Earlier today I have deployed a change that restrict one build per
> Jenkins slave.  That might help at the expense of slowing down the time
> it takes for all jobs to complete.
>
> We will see.

With Cloudbees we had a limit of two Jenkins executors at one time.  We
should be able to run ten Selenium tests(*) at once with WMF Jenkins.
 While throttling might help diagnosis of performance problems, ultimately
we to support Jenkins (and beta labs) with enough CPU/disk/memory to do
what we need done, that is, to not wait 90 seconds for instructions to
complete.

-Chris
(*) SauceLabs gives us a maximum of 10 VMs regardless of the number of
Jenkins builds in progress. In practice I see that sometimes we have all
ten in use, but on average we'll see about 8 tests running at once, +- 2 or
so.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20140825/eb1f76c2/attachment.html>