I ran some aggregate live profiling on en, and noticed that wfMsg was
responsible for 70% of request time. Each wfMsg() call was leading to a
database query. Investigating, I noticed that the memcached key for messages
was set to "error" -- this indicates that the script attempted to load
messages from the database and save them to memcached, but setting the large
value failed. This error is typical of the old slabs reassignment problem.
However that wasn't the problem in this case, none of the memcached servers
were at their memory quota.
In 1.2, only the internal messages from the MediaWiki namespace were cached.
In 1.3, custom messages are moved to the Template namespace and the whole
namespace is cached. But since I only just started running the move script,
we've been attempting to cache the whole namespace, including all the
templates, for the last week. Presumably the size of the namespace was
larger than memcached's value size limit, so it failed.
To restore site performance adversely affected by loading the messages, I
temporarily switched off $wgUseDatabaseMessages. Then I ran the move script
on en. It is still running as I type. When it is finished, I will clear the
error value from memcached and re-enable $wgUseDatabaseMessages. The first
web request after that should then cache the namespace successfully.
-- Tim Starling