"Scot Wilcoxon" <scot(a)wilcoxon.org> wrote:
The way I have my bots avoid irritating the servers is to back off very
quickly, and slowly speed back up. Starting with a minimum of 5 seconds
between updates... If there is no reply after 30 seconds, the bot
sleeps for 60 seconds before retrying. Another failure increases the
sleep time to 120 seconds. A successful update decreases the sleep time
by 10 seconds, until the minimum 5 seconds is reached. So a slow period
can slow the bot to several minutes between attempts, and it then takes
many minutes to return to normal.
On Thu, 2011-03-03 at 23:44 -0500, richardcavell(a)mail.com wrote:
> Hi all and TIA.
>
> I usually run my bot with maxlag = 5 seconds.
>
> When it attempts a
> communication with Wikipedia, if it receives Error:maxlag or HTTP 503 ,
> it retries up to 5 times with 10 seconds between each try. If it still
> can't get through, it aborts. At the time I'm writing this, about 3pm
> Friday afternoon in Australia, my bot is routinely aborting due to 5
> maxlag or 503s in a row.
>
The algorithm I've used in working on pywikipediabot is, upon receiving a
maxlag error, to have the bot sleep for min(5, lag / 2) seconds, where 'lag'
is the lag reported in the error message. If there is a long replication
lag, then it is going to take longer for the database slaves to catch up
than with a short lag, so there is no point in rechecking every 5 seconds.
Aborting after 5 maxlag errors is purely a design choice; if you don't mind
having your bot wait minutes or hours for the lag to subside, you can do
that too.
Russ