Wikimedia developers <wikitech-l(a)Wikipedia.org> writes: I suspect there is a lot that can be done without going to C/C++. With some performance tuning you probably could get a doubling to an order of magnitude performance out of the current code base. PHP is the right size language for this product. At worst you might have to write some C/C++ helper programs. Typically, if a product hasn't been written with performance in mind and never tuned, 90% of the CPU cycles occur in 5-10% of the code. I've been writing C/C++ programs for over 20 years. It is not a great language for working with string data. Unless you code carefully, you'll end up worse off. In terms of features/programmer hour, PHP is probably several times more productive than C/C++. Speaking on a purely theoretical basis, an important thing to writing or tuning a program written in something like PHP is to understand what operations cost the most when executed in the particular language. Also in almost any language, repeatedly scanning, copying and concatinating large strings, like 100,000 byte articles is really costly. Most string based programs spend 90% of their time doing storage allocation, so *looking* is much better than *touching*. Another thing to avoid are operations that go up by the square of the number of or size of objects. linear to 'n * Log n' are what you want to strive for. Database turning is quite important as well, though I get feeling you've been on top of that. Has much work been done in seeing what takes the most time when running mediawiki? I am still quite new to it and PHP as well. Nick

On Wed, 25 Feb 2004, Timwi wrote: Well, this is easy to settle. If a developer posts somewhere the load averages of the DB server, web servers and squids, and can see what's most taxed. Alfio

Nick Reinking wrote: I doubt whether you'd get an order of magnitude performance enhancement. Nick is right. The 90-10 law applies here: 90% (or more) of the time is spent in 10% (or less) of the code. The problem is that often you don't know which 10%. For example, there's not much point in making the system 10 times faster at the CPU level if it's I/O-bound: it will just spend 90% of the time waiting for I/O without running any faster. [[Amdahl's law]] applies to software as well as hardware. There's also not much point in rewriting the whole thing if you could have got almost the same gain by just rewriting a few percent of the code, and not made every part of the system far less maintainable by rewriting it in a low-level language. What makes sense is measuring hot-spots using diagnostic tools, looking at algorithms to see if algorithmic complexity can be reduced, finding places where parallelism can be increased, or faster hardware deployed, or _very selective_ low-level acceleration code added. More MRTG monitoring of stats from the various servers, and the public availability of the low-level SMTP stats files, would be very useful indeed in locating the hot-spots, allowing better decisions to be made about how to spend resources of all or any of these kinds. -- Neil

On Wed, Feb 25, 2004 at 11:19:45AM -0700, Kelly Anderson wrote: No, what you would do is extended PHP itself C/C++. Essentially, you'd be adding a new Wikipedia module into PHP, and then calling those functions. -- Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN

Wikimedia developers <wikitech-l(a)Wikipedia.org> writes: Yes, excellent idea! In fact I am proposing doing just such an operation in our company for one of our products. In that case, I need to blow apart an .html file much like you would do for XML, change some .html tags, which may involve pruning the tree, then walk the tree to generate a new .html file. It didn't occur to me that it might be worthwhile for this application. Parsing it to a tree first may not be that costly as you have probably have to make two passes anyway. For example, you can't generate the Table of Contents in the front until you've process the whole file. There may be some advantage to processing all the internal links at once. For the second pass you'd really like to work with some intermediate form, rather than working with the original or modified text string. Nick Pisarro

Tomasz Wegrzanowski wrote: I had to chuckle at your claim of something being "100% secure".