Hi Everyone,
What started out as an attempt to derive useful confidence measures for
language identification (with TextCat
<https://www.mediawiki.org/wiki/TextCat>) turned into a generalized
improvement effort. We still don't have useful external confidence
measures—though there's a little work yet to be done there (T149323
<https://phabricator.wikimedia.org/T149323>, T155670
<https://phabricator.wikimedia.org/T155670>). However, I did get a sizable
improvement to the F0.5 <https://en.wikipedia.org/wiki/F1_score> accuracy
scores by improving TextCat internals that don't really generalize to
externally useful measures. The result was a mean improvement of just under
5% across the corpora from nine Wikipedias. The two worst performing
corpora, enwiki and nlwiki, each went up around 10%! All nine are now above
90% F0.5 score.
You can read the final summary and recommendations
<https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/TextCat_Improvements#Final_Summary_.26_Recommendations>,
or read the rest of the page, too, if you want to know more about the whole
odyssey, or if you have trouble sleeping. ; )
Next steps for language identification are to get these changes deployed,
and then to look at other measures of confidence, and/or extend language
identification to more wikis, though the latter two may take a backseat to
working on new and improved language analyzers
<https://phabricator.wikimedia.org/T154511> for the rest of this quarter.
—Trey
Trey Jones
Software Engineer, Discovery
Wikimedia Foundation