Hello, This is our first weekly update being posted in this mailing list
New Developments
- Now you can abandon tasks you don't want to review in Wikilabels
(T105521)
- We collect user-agents in ORES requests (T113754)
- Precaching in ORES will be a daemon and more selective (T106638)
Progress in supporting new languages
- Russian reverted, damaging, and goodfaith models are built. They look
good and will be deployed this week.
- Hungarian reverted model is built, will be deployed this week.
Campaign for goodfaith and damaging is loaded in Wikilabels.
- Japanese reverted model are built, but there are still some issues to
work out. (T133405)
Active Labeling campaigns
- Edit quality (damaging and good faith)
- Wikipedias: Arabic, Azerbaijani, Dutch, German, French, Hebrew,
Hungarian, Indonesian, Italian, Japanese, Norwegian, Persian (v2),
Polish, Spanish, Ukrainian, Urdu, Vietnamese
- Wikidata
- Edit type
- English Wikipedia
Sincerely,
The Revision Scoring team.
<https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service#Team>
Hey folks, we have a couple of announcements for you today. First is that
ORES has a large set of new functionality that you might like to take
advantage of. We'll also want to talk about a *BREAKING CHANGE on April
7th.*
Don't know what ORES is? See
http://blog.wikimedia.org/2015/11/30/artificial-intelligence-x-ray-specs/
*New functionality*
*Scoring UI*
Sometimes you just want to score a few revisions in ORES and remembering
the URL structure is hard. So, we've build a simple scoring user-interface
<https://ores.wmflabs.org/ui/> that will allow you to more easily score a
set of edits.
*New API version*
We've been consistently getting requests to include more information in
ORES' responses. In order to make space for this new information, we needed
to change the structure of responses. But we wanted to do this without
breaking the tools that are already using ORES. So, we've developed a
versioning scheme that will allow you to take advantage of new
functionality when you are ready. The same old API will continue to be
available at https://ores.wmflabs.org/scores/, but we've added two
additional paths on top of this.
- https://ores.wmflabs.org/v1/scores/ is a mirror of the old scoring API
which will henceforth be referred to as "v1"
- https://ores.wmflabs.org/v2/scores/ implements a new response format
that is consistent between all sub-paths and adds some new functionality
*Swagger documentation*
Curious about the new functionality available in "v2" or maybe what the
change was from "v1"? We've implemented a structured description of both
versions of the scoring API using swagger -- which is becoming a defacto
stanard for this sort of thing. Visit https://ores.wmflabs.org/v1/ or
https://ores.wmflabs.org/v2/ to see the Swagger user-interface.
Visithttps://ores.wmflabs.org/v1/spec/ or https://ores.wmflabs.org/v2/spec/
to get the specification in a machine-readable format.
*Feature values & injection*
Have you wondered what ORES uses to make it's predictions? You can now ask
ORES to show you the list of "feature" statistics it uses to score
revisions. For example,
https://ores.wmflabs.org/v2/scores/enwiki/wp10/34567892/?features will
return the score with a mapping of feature values used by the "wp10"
article quality model in English Wikipedia to score oldid=34567892
<https://en.wikipedia.org/wiki/Special:Diff/34567892>. You can also
"inject" features into the scoring process to see how that affects the
prediction. E.g.,
https://ores.wmflabs.org/v2/scores/enwiki/wp10/34567892?features&feature.wi…
*Breaking change -- new models*
We've been experimenting with new learning algorithms to make ORES work
better and we've found that we get better results with gradient boosting
<https://en.wikipedia.org/wiki/Gradient_boosting> and random forest
<https://en.wikipedia.org/wiki/Random_forest> strategies than we do with
the current linear svc
<https://en.wikipedia.org/wiki/Support_vector_machine> models. We'd like to
get these new, better models deployed as soon as possible, but with the new
algorithm comes a change in the range of probabilities returned by the
model. So, when we deploy this change, any tools that uses hard-coded
thresholds on ORES' prediction probabilities will suddenly start behaving
strangely. Regretfully, we haven't found a way around this problem, so
we're announcing the change now and we plan to deploy this *BREAKING CHANGE
on April 7th*. Please subscribe to the AI mailinglist
<https://lists.wikimedia.org/mailman/listinfo/ai> or watch our project page
[[:m:ORES <https://meta.wikimedia.org/wiki/ORES>]] to catch announcements
of future changes and new functionality.
In order to make sure we don't end up in the same situation the next time
we want to change an algorithm, we've included a suite of evaluation
statistics with each model. The filter_rate_at_recall(0.9),
filter_rate_at_recall(0.75), and recall_at_fpr(0.1) thresholds represent
three critical thresholds (should review, needs review, and definitely
damaging -- respectively) that can be used to automatically configure your
wiki tool. You can find out these thresholds for your model of choice by
adding the ?model_info parameter to requests. So, come breaking change, we
strongly recommend basing your thresholds on these statistics in the
future. We'll be working to submit patches to tools that use ORES in the
next week to implement this flexibility. Hopefully, all you'll need to do
is worth with us on those.
-halfak & The Revision Scoring team
<https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service>
Fun story, the logs of #wikimedia-dev suggest that I performed 218 actions
during the grooming of this backlog. :)
On Wed, Mar 30, 2016 at 2:01 PM, Grace Gellerman <ggellerman(a)wikimedia.org>
wrote:
> Your micro-victory looks great- well done!
>
> On Wed, Mar 30, 2016 at 10:46 AM, Aaron Halfaker <ahalfaker(a)wikimedia.org>
> wrote:
>
>> Hey folks,
>>
>> I had a micro victory today, so I thought I'd share. The revision
>> scoring project has a huge backlog. I just spent a long time resolving old
>> tasks and organizing the *live* tasks by the projects they affect and what
>> they need me and my team to do.
>>
>> See
>> https://phabricator.wikimedia.org/tag/revision-scoring-as-a-service-backlog/
>>
>> Why you're looking (assuming you care to look at another project's
>> backlog), feel free to file new tasks in the "Backlog" or the appropriate
>> column.
>>
>> Note that the columns are not sorted in a nice way. Next time I have
>> time for this, I'll be doing prioritization.
>>
>> -Aaron
>>
>> _______________________________________________
>> Research-Internal mailing list
>> Research-Internal(a)lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/research-internal
>>
>>
>
> _______________________________________________
> Research-Internal mailing list
> Research-Internal(a)lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/research-internal
>
>
Hey folks,
I had a micro victory today, so I thought I'd share. The revision scoring
project has a huge backlog. I just spent a long time resolving old tasks
and organizing the *live* tasks by the projects they affect and what they
need me and my team to do.
See
https://phabricator.wikimedia.org/tag/revision-scoring-as-a-service-backlog/
Why you're looking (assuming you care to look at another project's
backlog), feel free to file new tasks in the "Backlog" or the appropriate
column.
Note that the columns are not sorted in a nice way. Next time I have time
for this, I'll be doing prioritization.
-Aaron
Hey folks,
As part of setting up production servers, we need to do a change that will
affect our hosts in labs (via puppet). This will require a brief downtime
event that should last less than 15 minutes.
We'll do this at 1400 UTC tomorrow, Tuesday March, 23rd. I'll post an
update once we're done.
-Aaron
Hey all,
We are working on deployment of new version of ORES. This new version
includes new version of revscoring, API v2 updates, and new models for
arwiki and plwiki.
The maintenance is taking more than we anticipated so you might experience
some issues or delays requesting scores. We apologize for any inconvenience.
We let you know once it's done.
Thank you for your patience.
Best
Forwarding the campaign announcement, and ideas from Aaron Halfaker.
Pine
---------- Forwarded message ----------
From: "Aaron Halfaker" <ahalfaker(a)wikimedia.org>
Date: Feb 29, 2016 16:00
Subject: Re: [Wikimedia-l] [Wmfall] Inspire Campaign on content curation &
review launches today!
To: "Chris Jethro Schilling" <cschilling(a)wikimedia.org>
Cc: "Staff All" <wmfall(a)lists.wikimedia.org>, <glam(a)lists.wikimedia.org>,
"Wikimedia Mailing List" <wikimedia-l(a)lists.wikimedia.org>
I just finished submitting two ideas that I'd like to advise.
https://meta.wikimedia.org/wiki/Grants:IdeaLab/Automated_good-faith_newcome…
Build and deploy a machine learning model for flagging newcomers who are
editing in good-faith. This has the potential to mitigate some of the
secondary, demotivational effects when good-faith newcomers' work passes
through curation/review processes.
https://meta.wikimedia.org/wiki/Grants:IdeaLab/Fast_and_slow_new_article_re…
Concerns about the introduction of spam into Wikipedia has lead Wikipedians
towards implementing high speed new article review/curation processes. The
speed at which editors tag articles for deletion via these processes is
great for dealing with spam, but it might also be faster that good-faith
new article creators can build their articles. We could build a machine
learning classifier that is tuned to detect spammy article drafts. This
would allow the new pages queue to be split into a high-speed spammy
article review, and a low-speed article review that allows creators time to
make a better first draft.
I'll submit some more when I can. :)
On Sun, Feb 28, 2016 at 4:56 PM, Chris "Jethro" Schilling <
cschilling(a)wikimedia.org> wrote:
> Hi everyone,
>
> I am pleased to announce the launch of the second Inspire Campaign for
> IdeaLab.[1] The theme of this campaign is focused on improving tasks
> related to content curation & review in our projects:
>
> <https://meta.wikimedia.org/wiki/Grants:IdeaLab/Inspire>
>
> Reviewing and organizing tasks are fundamental to all WIkimedia projects,
> and these efforts maintain and directly improve the quality of our
projects
> in addition to increasing the visibility of their content. We invite
> everyone to participate by sharing your ideas and proposals on how to
> enhance these efforts. Constructive feedback and collaboration on ideas is
> encouraged - your skills and advice can elevate a project into action. The
> campaign runs until 29 March.
>
> All proposals are welcome - research projects, technical solutions,
> community organizing and outreach, or something completely new! Grants are
> available from the Wikimedia Foundation for projects developed during this
> campaign that need financial support.[2] Google Hangout sessions are
> available in March if you'd like to have a conversation about your
ideas.[3]
>
> Join the Inspire Campaign and let’s work together to improve review and
> curation tasks so that we can make our content more meaningful and
> accessible.
>
> With thanks,
>
> Jethro
>
> [1] You can learn more about the results of the first Inspire Campaign
> here: <
> https://meta.wikimedia.org/wiki/Research:Spring_2015_Inspire_campaign>
> [2] <https://meta.wikimedia.org/wiki/Grants:Start>
> [3] <https://meta.wikimedia.org/wiki/Grants:IdeaLab/Events> (Note: If
> another time would work better for you, feel free to e-mail me or ping me
> on-wiki).
>
> ---
> Chris "Jethro" Schilling
> I JethroBT (WMF) <https://meta.wikimedia.org/wiki/User:I_JethroBT_(WMF)>
> Community Organizer, Wikimedia Foundation
> <https://wikimediafoundation.org/wiki/Home>
>
> _______________________________________________
> Wmfall mailing list
> Wmfall(a)lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wmfall
>
>
_______________________________________________
Wikimedia-l mailing list, guidelines at:
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
New messages to: Wikimedia-l(a)lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
<mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe>
Hey all,
TLDR: ORES extension [1] which is an extension that integrates ORES service
[2] with Wikipedia to make fighting vandalism easier and more efficient is
in the progress of deployment. You can test it in
https://mw-revscoring.wmflabs.org (Enable it in your preferences first)
You probably know ORES. It's an API service that gives probably of an edit
being vandalism, it also does other AI-related stuff like guessing the
quality of articles in Wikipedia. We have a nice blog post in Wikimedia
Blog [3] and media paid some attention to it [4]. Thanks to Aaron Halfaker
and others [5] for their work in building this service. There are several
tools using ORES to highlight possibly vandalism edits. Huggle, gadgets
like ScoredRevisions, etc. But an extension does this job much more
efficiently.
The extension which is being developed by Adam Wight, Kunal Mehta and me
highlights unpatrolled edits in recentchanges, watchlists, related changes
and in future, user contributions if ORES score of those edits pass a
certain threshold. GUI design is made by May Galloway. ORES API (
ores.wmflabs.org) only gives you a score between 0 and 1. Zero means it's
not vandalism at all and one means it's vandalism for sure. You can test
its simple GUI in https://ores.wmflabs.org/ui/. It's possible to change the
threshold in your preferences in the recent changes tab (you have options
instead of numbers because we thought numbers are not very intuitive).
Also, we enabled it in a test wiki so you test it:
https://mw-revscoring.wmflabs.org. You need to make an account (use a dummy
password) and then enable it in beta features tab. Note that building AI
tool to detect vandalism in a test wiki sounds a little bit silly ;) so we
set up a dummy model that probability of an edit being vandalism is
backward of the last two digits (e.g. diff id:12345 = score:54%). In a more
technical aspect, we store these scores in ores_classification table so we
can do a lot more analysis with them once the extension is deployed. Fun
use cases such as the average score of a certain page or contributions of a
user or members of a category, etc.
We passed security review and we have consensus to enable it in Persian
Wikipedia. We are only blocked on ORES moving from Labs to production
(T106867 [6]). The next wiki is Wikidata, we are good to go once the
community finishes labeling edits so we can build the "damaging" model. We
can enable it Portuguese and Turkish Wikipedia after March because s2 and
s3 have database storage issues right now. For other Wikis, you need to
check if ORES supports the Wiki and if community finished labeling edits
for ORES (check out the table at [2])
If you want to report bugs or add feature requests you can find it in here
[7].
[1]: https://www.mediawiki.org/wiki/Extension:ORES
[2]: https://meta.wikimedia.org/wiki/Objective_Revision_Evaluation_Service
[3]:
https://blog.wikimedia.org/2015/11/30/artificial-intelligence-x-ray-specs/
[4]:
https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service/Media
[5]:
https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service#Team
[6]: https://phabricator.wikimedia.org/T106867
[7]: https://phabricator.wikimedia.org/tag/mediawiki-extensions-ores/
Best
Hey folks,
I just wanted to share a quick writeup that Andrew Lih did about using ORES
<https://meta.wikimedia.org/wiki/Objective_Revision_Evaluation_Service>'
article quality prediction model in the classroom to help students see the
impact of their work in Wikipedia. See
https://en.wikipedia.org/wiki/User:Fuzheado/ORES_experiment
Some quotes from the writeup:
“Yay!"
>
> “Alright!"
>
> "I feel like we’re part of a secret computer club!"
>
> [...] having students excited and experiencing that instant dopamine hit
> when editing articles is a breath of fresh air when pleasurable
> interactions seem to be less frequent today than in Wikipedia’s early days.
>
One important note is that I had *no idea* ORES would be used in this way.
In a lot of ways, that's the point of building infrastructure like this --
to empower others to enact their own values.
-Aaron
OK, thanks for the response.
On another note, I recently learned that the purpose of the beta features
extension is to allow extensions to be tested before they can be pushed to
production. This means that after some time (I think, 6 months) we will
have to promote to stable or dismantle the WikiLabels extension. I don't
think promoting is an option as not everyone needs this feature. So I was
wondering if we still want to move the functionality from the gadget to an
extension.
Baha
On Mon, 18 Jan 2016 11:09:38 -0500, Aaron Halfaker
<aaron.halfaker(a)gmail.com> wrote:
> Right now, we build some of that JS based on configuration parameters
> that
> are needed by the WikiLabels service back-end. It seems that it would be
> easiest to generate those files and cache them so that we can serve with
> them extension code.
>
> I'm not sure how much sense this makes long-term. If the answer is "no
> sense -- at all", then I think our next step would be getting a good
> description of which parts of that JS are configured and then working on
> a
> proposal for how we'd change things within the extension. I can do the
> description if we get that far.
>
> -Aaron
>
> On Mon, Jan 18, 2016 at 6:55 AM, Bahodir Mansurov
> <bmansurov(a)wikimedia.org>
> wrote:
>
>> Hello,
>>
>> As I was working on the WikiLabels extension, I had to copy over a lot
>> of
>> CSS and JavaScript files to the extension folder. I suppose it's fine by
>> itself, but I was wondering if those files will continue to be used
>> outside
>> the extension? If so, we should keep them where they are and load them
>> dynamically in the extension (as it's done in the gadget). What do you
>> think?
>>
>> --
>> Baha
>>
>> _______________________________________________
>> AI mailing list
>> AI(a)lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/ai
>>
--
Baha