On Wed, 25 Feb 2004, Nick Reinking wrote: E23 allegedly has been working on profiling code, but I've not played with it in a long time so can't say what results look like. Last time we did profiling, the Prime #1 Enemy was OutputPage::replaceInternalLinks(). The rest of the page parsing was zippy fast even with a hojillion passes. This requires splitting up the page, identifying links, normalizing titles, checking lookup tables for page existence, and generation of output links. Make it fast and you win a prize! That's what the next revision will concentrate on: cleaning up the schema, moving the large data blobs out of the most-queried tables, etc. Note that page parsing time is largely separate from (though can be slowed down by locking or disk churning due to) database slowness. But an overloaded database does slow everything down and makes people very unhappy. -- brion vibber (brion @ pobox.com)

Brion Vibber wrote: My guess is that the slowest part of it is checking whether a page exists, and if it does, checking its size (if the user has set the preference that shows stubs in a different colour), because both of this requires a database query. These two things are therefore perfect candidates for memcaching (hint, hint). Notice that we need to check if a page exists in several other places too (hint, hint). Timwi

Brion Vibber wrote: Seconded. In my opinion, performance is perfectly fine right now. It could be better, of course, I mean, anything could always be better, right? And we're running with a suboptimal db server. suda has a load average of 3-4 fairly often, while geoffrin (when running) had a load average under 1.0 m ost of the time. When Geoffrin is healed and here (just over a week from now, I think), then we'll have 2 db servers, and we can do some simple load balancing on those. And our current architecture is now very reasonably scalable, *after* we come to some understanding of where bottlenecks might be. If we need more webservers, or more RAM in some squids, or more database servers, we will find it fairly easy to just add them. I'm personally still a lot more focussed on *reliability*. We want to work hard to make sure that wikipedia always answers every request. I've already made my campaign against worrying about arcane possibilities like datacenter failure and whatnot. Those are good to think about in the abstract, but we have other issues *right now*. (Although nothing so serious as what we faced over the holidays, of course.) --Jimbo

Nick Reinking wrote: I totally agree. ...or memcache more. :-) Timwi

Neil Harris wrote: This isn't actually quite so bad an idea; this could be done for about a day to gather the necessary data. However, it *only* tells us how much of the performance is eaten up by the parser. It doesn't really tell us anything else. Our biggest problem is most likely not the parser. Timwi