Edward Z. Yang wrote: The performance of our extension system is poor, currently about 20% of CPU time on the cluster is used by extension registration of various kinds. Extensions in PHP need to be arranged so that on a normal startup, no code needs to be executed from any of the extensions. It should be a matter of checking the file modification times, and loading a pre-merged message array from a cache. The input text could indeed be decentralized while maintaining good performance, with careful design. Whether or not to do this in any given application should probably be at the discretion of the application designer. They're very rarely out of sync. I think most of us have found that it's convenient to have all the English messages in the same place. Revising the English text can be done without reference to the code, just like translation can. Programmers sometimes make very poor choices for the default text. The translators encourage us to avoid reuse. Just because two concepts can be expressed with the same word in English doesn't mean they can be expressed with the same word in every language. "OK" is a good example: we use it for a generic button label, but in some languages, they prefer to use a button label related to the operation which will be performed by the button. An easy way to duplicate messages across all languages would reduce the programmer's need to reuse messages. Preferably you would have a reference rather than a full copy, to reduce maintenance. We didn't have plural support for a long time, I tend to think of it as eye candy. English needs plurals just like any other language, but programmers routinely abuse the language for our convenience. We now do have plurals, which I suppose makes for a nicer-looking product. For example: 'undelete_short' => 'Undelete {{PLURAL:$1|one edit|$1 edits}}', Grammatical transformations for agglutinative languages came a bit earlier. I implemented them for Finnish, when I was told that it was absolutely necessary if we wished to make our language files site-independent, i.e. to remove the Wikipedia references. In Finnish, "about Wikipedia" becomes "Tietoja Wikipediasta" and "you can upload it to Wikipedia" becomes "Voit tallentaa tiedoston Wikipediaan". Suffixes are added depending on how the word is used, plus minor modifications to the base. There is a long list of exceptions, but since we only needed to translate a few words, such as the site name, we didn't need to include it. We now have grammatical transformation functions for 18 languages. Some of these are just dictionaries for Wikimedia site names, but many have proper algorithms. Even before we had arbitrary grammatical transformation, we had a nominative/genitive distinction for month names. This distinction is absolutely necessary if you wish to substitute month names into sentences. The other (much simpler) issue with parameter substitution is HTML escaping. Despite being much simpler, MediaWiki does a pretty poor job of it. We have a plethora of poorly-named wfMsg*() functions, including the multitasking wfMsgExt(), with lots of ways to slip up and let through unescaped user input. I'd like to do some work in that area at some stage. In another other post: There's also the MessageCache class, which handles input of text via the MediaWiki namespace. And there's the wfMsg*() functions in GlobalFunctions.php. When I wrote MessageCache.php, I intended the wfMsg*() functions to remain as simple shortcuts to functions in the MessageCache class. That was not how it turned out; I wasn't paying enough attention. Now we have large amounts of message retrieval code in GlobalFunctions.php. The manual recompilation model was a mistake, and I intend to remove it at the earliest opportunity. It's inconvenient for site administrators. Caching is much more versatile, especially when you add dependency checking. The only problem is that you need to have a data store which is both fast to read and writable by the webserver. Such a store is not always available. -- Tim Starling

Edward Z. Yang wrote: Sure, I would have a naming convention. We've already more or less standardised on using prefixes for extension messages. Namespaces for core messages would probably be useful too. My main bugbear with the current system is punctuation style: Lee used strings of unbroken lowercase letters, which I think is hard to read. They're always quoted, so why not have spaces between words? Or underscores, if that's too progressive. If you have an algorithm, then you only have to enter the site name once, not six times. That's very convenient, so I would encourage people to implement algorithms rather than dictionaries. Even a comprehensive dictionary isn't going to be much good for site names which are invented words. Most conversion functions respect $wgGrammarForms, which is an exception list for grammatical transforms that can be customised from LocalSettings.php, just like the site name. -- Tim Starling

On 24/11/06, Simetrical <Simetrical+wikitech(a)gmail.com> wrote: That's bollocks, spaces in message names are fine and there's no reason we can't use them. We have, I think, a few messages with spaces in their names at present as it is. Rob Church

Rob Church wrote: I think I used ucfirst instead of Title::newFromText() to convert from message names to DB keys, for efficiency reasons. That could be changed, of course. The efficiency of the message cache system has been completely blown to hell anyway, by the invention of parsed messages. Here I am worrying about a few more lines of PHP, and then someone adds a shell out to Tidy. -- Tim Starling

Here is my analysis of MediaWiki's I18N system: == Structure == First, you have a Language object. This object contains all the localisable message strings, as well as other important language-specific settings and custom behavior (uppercasing, lowercasing, printing dates, formatting numbers, etc.) The object is constructed from two sources: subclassed versions of itself (classes) and Message files (messages). == General use == You load a language object by calling the Language::factory() function. This function the class file for the object (taking in account fallback languages by using the fallback langauge's object but overloading the language key) and returns that object. Nothing else happens. When a message/etc is requested, a lazy load initializor is called. Now the real work starts. We're first going to take the scenario that the language is not cached. The system loads the Messages file by: require( $filename ); $cache = compact( self::$mLocalisationKeys ); ...where self::$mLocalisationKeys is the name of variables that could be used in the localization file. This lets you use things like: $fallback = false; $rtl = false; ...and easily siphon them into arrays. Then, we load the $fallback language (if not set, English) to fill in the gaps in the messages. There is specialized behavior for certain keys, as they can be mergeable maps, lists or alias lists (not sure what the last one is). == Caching == MediaWiki has lots of caching mechanisms built in, which make the code somewhat more difficult to understand. Before doing any loading, MediaWiki will check the following places to see if we can be lazy: 1. $mLocalisationCache[$code] - just a variable where it may have been stashed 2. serialized/$code.ser - compiled serialized language file 3. Memcached version of file (with expiration checking) Expiration checking consists of by ensuring all dependencies have filemtime that match the ones bundled with the cached copy. Similar checking could be implemented for serialized versions, as it seems that they are not updated until manually recompiled. == Behavior == Things that are localizable: - Weekdays (and abbrev) - Months (and abbrev) - Bookstores - Skin names - Math names - Date preferences - Date format - Default date format - Date preference migration map - Default user option overrides - Language names - Timezones - Character encoding conversion via iconv - UpperLowerCase first (needs casemaps for some) - UpperLowerCase - Uppercase words - Uppercase word breaks - Case folding - Strip punctuation for MySQL search - Get first character - Alternate encoding - Recoding for edit (and then recode input) - RTL - Direction mark character depending on RTL - Arrow depending on RTL - Languages where italics cannot be used - Number formatting (commafy, transform digits, transform separators) - Truncate (multibyte) - Grammar conversions for inflected languages - Plural transformations - Formatting expiry times - Segmenting for diffs (Chinese) - Convert to variants of language - Language specific user preference options - Link trails [[foo]]bar - Language code (RFC 3066) Neat functionality: - I18N sprintfDate - Roman numeral formatting