[Labs-l] large bots

Thu Jan 10 19:07:34 UTC 2013

On Thu, Jan 10, 2013 at 11:02 AM, Tim Landscheidt <tim at tim-landscheidt.de>wrote:

> Petr Bena <benapetr at gmail.com> wrote:
>
> > in order to save system resources on bots project I decided to create one
> > huge application instance instead of ton of small instances.
>
> > If we consider the usage of operating system for disk storage, and RAM,
> you
> > easily find out that 10 small instances eat about 1800MB RAM and  20GB of
> > hdd storage more than it would be needed in case we had only one OS
> working
> > with multiple processes.
>
> > I know that this bring out some extra problems, so I would rather
> consider
> > it experiment for now. But given that we install some process killers
> > handling out broken bots that eat too much ram and such, it should work
> > (though it would be more like toolserver used to be). Of course, some
> extra
> > bots can still live on their own dedicated servers.
>
> > So in case you want to help me test this new experiment and you have some
> > bots running on bots-2 - bots-4 you can move them to bots-bnr1 (big non
> > root = bnr)
>
> On Toolserver, bots can use SGE to distribute the load more
> evenly over the cluster
> (cf. https://wiki.toolserver.org/view/Job_scheduling).  I
> had asked at
>
> http://www.mediawiki.org/wiki/Wikimedia_Labs/Toolserver_features_wanted_in_Tool_Labs
> if there is a "standard" way of job scheduling in Ubuntu
> shops - so is there? :-)
>
>
I don't believe so, no. These kinds of things are always custom built or
off-the-shelf schedulers are used.

- Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/labs-l/attachments/20130110/eea52757/attachment.html>