Wikihackers, You may remember me from last year, pushing for UTF-8 on the English wikipedia. There was more to it than I thought at the time, so I retreated, checking the site's encoding every few weeks to see if anyone had made a breakthrough. Alas, here we are, still chugging along in trusty old ISO-8859-1. Knowing that shutting down the site for a few days is unacceptable, I'm working on a technique to keep the site up, already in UTF-8 mode, while 'cur' and 'old' are converted. The site itself would be doing the bulk of the conversion. The idea is to intercept calls to these tables and convert the encoding before they are sent to the user. If a conversion is made, save it to the database so we don't have to convert it again. Here's an outline: 1. Take site offline. :( 2. Convert smaller, essential tables. (user? what else?) 3. Apply code changes to intercept retrievals from cur and old: Change A: For _title queries that don't return anything, try converting querystring to Latin1. Change B: Check the _text (and _user_text ?) strings to see if they are valid UTF-8. If not, convert and update the database. Change C: $wgUseLatin1 = false ! 4. Take site back online. :D 5. When the dust settles, start an external script to run through the database and update any unpopular entries that haven't been converted. I've written some code for the cur_text intercept to make sure that it will work. It's running well for me on a test DB I made, a simple English DB converted to Latin1. It's only for the cur_text field right now, but it should work the others if expanded. I haven't looked at the problem with titles (Change A), but I think it could be similarly tackled. Before I continue, I wonder if anyone sees catastrophic problems with this approach? My familiarity with the codebase is still pretty spotty. Here's the patch: Index: Database.php =================================================================== RCS file: /cvsroot/wikipedia/phase3/includes/Database.php,v retrieving revision 1.71.2.7 diff --unified=3 -w -B -r1.71.2.7 Database.php --- Database.php 14 Jan 2005 13:04:04 -0000 1.71.2.7 +++ Database.php 8 Feb 2005 20:10:37 -0000 @@ -681,11 +681,37 @@ } $obj = $this->fetchObject( $res ); $this->freeResult( $res ); + + if ($conds['cur_id']) + $this->checkEncoding($conds['cur_id'], $obj); + return $obj; } /** + * Checks if the object's encoding is UTF-8. If it is not, convert to UTF-8 and update + * the database record as well. + * + * @param $id the cur_id or old_id + * @param $obj object whose fields will be verified and possibly updated + */ + function checkEncoding($id, $obj) { + + // If a conversion from UTF-8 to UTF-8 does not result in the same string, it is + // not a valid UTF-8 string. + if ($obj->cur_text != iconv("UTF-8", "UTF-8", $obj->cur_text)) { + + // Assume it is Windows-1252 (Latin1 + em dashes etc., then convert. + $obj->cur_text = iconv("Windows-1252", "UTF-8", $obj->cur_text); + + //save converted string to db + $cur_text = mysql_real_escape_string($obj->cur_text); + $this->query("UPDATE cur SET cur_text = '$cur_text' WHERE cur_id = $id"); + } + } + + /** * Removes most variables from an SQL query and replaces them with X or N for numbers. * It's only slightly flawed. Don't use for anything important. *