MediaWiki supports both ways. Wikipedia still uses the mysql 4 compatible options, and since mysql chars only support the bmp, it isn't likely to change. It all depends on what you choose on install. Currently, the preselected choice is 'MySQL 4.1/5.0 binary'.

No. The BMP problem is only present with the "MySQL 4.1/5.0 UTF-8" option.

It seems 5.0 and 5.1 support for UTF-8 are the same http://dev.mysql.com/doc/refman/5.0/en/charset-unicode.html http://dev.mysql.com/doc/refman/5.1/en/charset-unicode.html

wrote:

wrote:

Hello, On Jun 8, 2010, at 11:22 PM, Gerard Meijssen wrote: It does not, it sorts according to the partial UCA implementation. We have discussed CLDR in the past - it is a huge collection of distinct collations, and even though it is possible to use LDMLs from CLDR project, it is PITA, due to both partial UCA support and continuous effort to rebuild indexing, resolve conflicts, and hit all sorts of obscure "linguists are not computer scientists" problems :)

Hi! it Well, I agree that it might be good idea to have language-specific ordering, just costs are quite high and there're not too many people eager to do engineering part of such project. CLDR isn't panacea, it is constantly evolving project, with inaccurate stable versions (even for well established languages like mine, heheh), and various proposed/testing versions. So, to pick CLDR based flow, and do it properly, it would consist of infinite loop of: 1. Understanding which languages need a separate collation 2. Evaluating all available collations for a language, attracting input from local communities and standardization bodies 3. Evaluating the algorithmic implications of chosen collation - then either approaching standards bodies to change it, or simplifying it internally (and forking), or implementing algorithms in software (though that sometimes is impossible to do in efficient way) 4. Porting (3) into a backend of choice 5. Provide upgrade path and conflict resolution method for existing content 6. Provide framework to do full index rebuilds and switchover between different collations (ok, this probably is one-time engineering project, albeit quite complex, as it has to have (4) and (5) in mind) 7. Monitor for new versions of collations :) Multiply all that by number of languages we have, and do note that there're multiple sorting variants per language too (e.g. dictionary vs phonebook ordering in Germany). So yes, it would be fantastic to have that kind of functionality, but you'd need quite some engineering capacity to pull it off. And if we get to implementation specifics - ordering rules are same as equality rules, causing quite some confusion in some cases (and some people will definitely want to have same sorted but not equal terms.. :) Of course, we can use community driven sortkey hacks for some features ;-) I guess it isn't absolutely tragic for others, as otherwise we wouldn't see projects in other languages at all. Now thats a benchmark! ;-) Domas _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Currently it is possible to define the sortkeys using {{DEFAULTSORT:Sortkey}} and in lots of places this "sortkey" coincide with the value of magic words like {{PAGENAME}} and {{SUBPAGENAME}}, so we don't need to update them manually. The (annoying) exception is when the value returned by these magic words starts, for example, with "Á" and we would like to have the page sorted in "A": in this cases we are forced to type the sortkey manually. Would it be possible to have some command for language-dependent word conversions, like {{collation:<langcode>|<string>}}, whose result should be the given <string> with character "Á" changed to "A" (and so on for other characters), according to the <langcode>? This could be used in situations like {{DEFAULTSORT: {{collation: pt | {{SUBPAGENAME}} }} }} or even to create a template {{Simplified sortkey}} with the code above. What do you think? Helder

varbinary() otoh is varchar with 'binary' character set (if you define default server charset to be binary, as we do on our 5.x boxes, all varchar creation will be varbinary).