Hey,
This is the 18th weekly update from revision scoring team that we have sent
to this mailing list.
*Communications:*
- Aaron presented on how user-feedback has been helping us address some
sneaky biases in ORES' models. [1, 2, 3]
*New development:*
- We included 'autoreview' and 'patroller' groups in Turkish wiki models
to get a fitness boost. [4]
- We added some basic uwsgi metrics to grafana[5] and added a response
timing metric from Change Propagation so that we can track any performance
issues. [6]
*Maintenance and robustness:*
- We increased the number of workers per node in production for a 66%
increase in total capacity for ORES[7]
- We updated all of our edit quality models with the new version of
revscoring [8] and sent an email out to wikitech-l and ai-l about the
implications for tool developers. [9]
- We decided not to make specialized models for ORES in beta labs. [10]
Instead, we'll use the production models so that issues with them will be
caught in beta.
1. https://phabricator.wikimedia.org/T143275 -- Present on user-feedback
stories at Research Showcase
2. https://www.youtube.com/watch?v=rsFmqYxtt9w#t=29m00s -- Video of ORES
user-feedback talk
3.
https://www.mediawiki.org/wiki/File:Deploying_and_maintaining_AI_in_a_socio…
4. https://phabricator.wikimedia.org/T140474 -- Include specific user
groups in the trwiki edit quality model
5. https://phabricator.wikimedia.org/T143081 -- Add uwsgi-related metrics
to grafana
6. https://phabricator.wikimedia.org/T143568 -- Add median, 75% and 95%
response time to ORES dashboard
7. https://phabricator.wikimedia.org/T143105 -- Increase celery workers to
40 per scb node
8. https://phabricator.wikimedia.org/T143125 -- Update editquality models
with new version of revscoring
9. https://lists.wikimedia.org/pipermail/ai/2016-August/000068.html --
"[AI] New models coming to ORES & notes"
10. https://phabricator.wikimedia.org/T141980 -- Should we make a model for
ores in beta?
Sincerely,
Aaron from the Revision Scoring team
Forwarding, since the subjects may be of interest to people on the
Wikitech, AI, and Research lists.
I'm unqualified to evaluate Damon's comments and the FB exec's comments
about AI, so please refrain from shooting the messenger if these aren't
helpful or interesting to those of you who do know enough about AI to make
well-educated assessments.
Regards,
Pine
---------- Forwarded message ----------
From: "Damon Sicore" <damon(a)sicore.com>
Date: Aug 18, 2016 21:35
Subject: [Wikimedia-l] Facebook CTO on strategy, Internet access, Wikipedia
To: "Wikimedia Mailing List" <wikimedia-l(a)lists.wikimedia.org>
Cc:
Hi,
I usually don't recommend these things, but this interview with Schrep [1]
[2] is interesting and insightful. I recommend listening to it instead of
reading. He discusses FB's ten year plan, AI, VR, Internet access for
all, mentions Wikipedia several times, confirms their insatiable hunger for
structured data, and reveals several details on their innovation approach.
Trigger Warning: Corporate Speak
Make no mistake, I've nothing but contempt and spite for Facebook, but
having worked with Mike I also know he demonstrates formidable intellect
and is a decent person. He's incredibly capable in building amazing teams
and predicting (more like sniffing out) the future of tech. I watch his
moves closely to stay sharp.
He's right about how papers are coming out constantly which augment current
AI tech in interesting new ways. I believe we're living in interesting
times for computer science and mathematics--computational linguistics and
probabilistic search in particular. A person can't read the CS and math
papers fast enough in order to keep up with the innovation. A lot of it is
trivial, sure, but some is quite startling in impact as they combine a few
smaller things which seemed previously innocuous yet when used together
they solve key problems.
When looking into tech and strategy for WMF and the engineers it supports,
I'd be very interested in the direction Facebook is going and the
technologies they plan on investing in, so passing it along.
Yours faithfully,
Damon
[1] http://www.metisstrategy.com/interview/mike-schroepfer/
[2] https://en.wikipedia.org/wiki/Mike_Schroepfer
Damon Sicore
512 963 5126
https://damon.sicore.com
6E98 FBFB
D192 D325
B85D D4FF
FD2A 20ED
DC1D 3975
_______________________________________________
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
wiki/Mailing_lists/Guidelines
New messages to: Wikimedia-l(a)lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
<mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe>
Hey all,
Despite working and getting multiple people to review the code for adding
WikiLabels as a beta feature [1], I haven't been able to see it to
completion because of other commitments. I also haven't been able to keep
up with the developments in ORES. Yesterday I realized that there is
another extension that's doing the same job. So I am thinking of abandoning
my work and having the WikiLabels extension removed from gerrit. Let me
know if anyone wants the extension not deleted.
Thanks,
Baha
[1] https://gerrit.wikimedia.org/r/#/c/264415/
Hey,
We are fixing a rather urgent bug [1] introduced in the refactor so we will
have an unscheduled deployment today. Also we need to clear ORES redis
cache meaning it'll be a little slower than usual for a while but since
ORES review tool has its own caching and this bug didn't effect that tool,
it won't affect users of the extension.
1. https://phabricator.wikimedia.org/T142857
Sorry for any inconvinece
Best
Hey folks,
This is the 16th weekly update from revision scoring team that we have sent
to this mailing list.
New developments:
- We created dashboards for the ORES service in the Beta cluster[1] and
created panes for tracking failed jobs[2].
- We extended the documentation for the ORES review tool[3,4]
Maintenance:
- We did some work to make the Beta cluster look more like production so
that we can do better testing before the next deployment
- We set up a password on the Beta redis server[5]
- We configured the Beta ORES extension to actually use the Beta ORES
service[6]
- We also prepared a set of puppet changes for the deployment of a
refactored version of ORES to production[7]
Issues in WMFLabs
- We investigated a series of timeout errors that were appearing in the
logs[8]
- We investigated a periodic redis-related error that shower up when
scoring edits[9]
- We fixed our "05" web node that was periodically running out of
memory[10]
Estimating future resource needs
- In preparation for buying new hardware, we measured our past memory
usage and extrapolated forward two years to estimate what hardware
requirements we'll have[11]
1. https://phabricator.wikimedia.org/T142294 - Dashboard or pane for
ORES service in beta
1. https://phabricator.wikimedia.org/T142119 - Dashboard or pane for
ORES failed jobs on beta
1. https://phabricator.wikimedia.org/T140150 - Make user-centered
documentation for review tool
1. https://www.mediawiki.org/wiki/ORES_review_tool
1. https://phabricator.wikimedia.org/T141823 - Set up password on ORES
Beta redis server
1. https://phabricator.wikimedia.org/T141825 - Config beta ORES
extension to use the beta ORES service
1. https://phabricator.wikimedia.org/T141575 - Puppet config changes for
ORES refactor
1. https://phabricator.wikimedia.org/T141368 - [Investigate] ORES time
out errors in logs
1. https://phabricator.wikimedia.org/T141946 - [Investigate] Periodic
redis related errors in wmflabs
1. https://phabricator.wikimedia.org/T141523 - [Investigate] web-05
downtime
1. https://phabricator.wikimedia.org/T142046 - Extrapolate memory usage
per worker forward 2 years
Sincerely,
Aaron from the Revision Scoring team
This means ORES in labs will be down at period of time since our redis will
be restarted.
Best
---------- Forwarded message ---------
From: Andrew Bogott <abogott(a)wikimedia.org>
Date: Sat, Aug 6, 2016 at 8:58 PM
Subject: [Labs-l] [Labs-announce] Some Labs instances rebooting TODAY,
19:00 UTC
To: <labs-announce(a)lists.wikimedia.org>
Hello!
We have discovered a surprisingly terrible bug in the kernel
that's running on two of the nova-compute hosts. To remedy this, we
will be downgrading and rebooting both hosts in a few hours, at high
noon San Francisco time, 19:00UTC.
We will shuffle things around to so there is no impact on Tools.
VMs on these hosts only (listed below) will experience a single reboot
and accompanied downtime.
Sorry for the short notice... it's worth it, believe me. We'll
work on getting a post-mortem incident report written, but that may not
happen until Monday. In the meantime, here is a complete list of
instances that will be affected:
| 330f16d6-374d-44a8-bf96-53e763d5dd3a | captcha-apiproxy-02
| privpol-captcha
| 00422b7c-9711-4a92-bba4-73db0efb5889 | deployment-db01
| deployment-prep
| 1597e37a-99e6-431e-8117-3901e4ac9858 | encoding02
| video
| 01518800-406c-4689-a289-9e3e33fd387b | kafka501
| analytics
| 6a5289e9-1b69-48a3-a6fd-5de63c2ee285 | labstore-test-05
| testlabs
| 37fc9d21-6614-4442-9b45-a7ae349b6d09 | secgroup-server-labvirt1012
| testlabs
| d35fedde-9f0f-42b2-a22d-780cb1477a17 | spice-test-102
| admin
| 21706518-ccd4-45b1-9853-37f463d02393 | striker-uwsgi01
| striker
| c4bc63f8-cbd7-4384-b349-54b115e91a5c | util-abogott
| testlabs
| 3d3288cd-3523-449c-bf48-1bd48cac3e5d |
captcha-proxypostgres-01 | privpol-captcha
| 7ebb2447-7c74-4fd1-aa5e-19c48011e39d |
deployment-depurate01 | deployment-prep
| 9336405f-cf57-4234-9c73-0a539d97580e |
deployment-kafka05 | deployment-prep
| 60ece1f0-327b-49b2-aa1e-a3df241f70fb |
encoding03 | video
| 8b90e688-aba7-4b6c-9b6b-bbb13ec84da5 |
gerrit-test3 | git
| 833bb2a0-a442-445e-8210-ac77e950ead7 |
integration-slave-jessie-1003 | integration
| a64e731e-c987-4b84-85b9-cf450275d26d |
integration-slave-jessie-android | integration
| a18beafb-2279-49fa-8b88-049b3f55a2f5 |
kafka601 | analytics
| fe12b525-5a64-48ff-a3a3-e69c65043e26 |
labstore-test-01 | testlabs
| 9b45a5c1-c6a5-4a59-8305-9d4542a4c27a |
labstore-test-02 | testlabs
| f5bcb8c1-51b2-4820-bda3-a7a3155b0a6f |
mwdiffstuff | catgraph
| 59664505-be72-4294-b639-b1ea2218a44b |
ores-redis-02 | ores
| d5b434ce-2cae-4a56-9b95-4459913516e9 |
pole | wikidata-query
| fdc47372-b60c-4c61-ab27-9231a5fd2d4e |
secgroup-server-labvirt1013 | testlabs
| c8e73c70-9d13-480f-a6a9-01c46328a83d |
striker-build | striker
| 55e6d403-c916-44f9-881f-23dec8968111 |
striker-deploy03 | striker
| d54ea2da-f3d5-4d8b-aa47-41837accb285 |
utrs-secondary | utrs
-Andrew
_______________________________________________
Labs-announce mailing list
Labs-announce(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
_______________________________________________
Labs-l mailing list
Labs-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l
Hey,
This is the 15th weekly update from revision scoring team that we have sent
to this mailing list.
*New developments:*
- We'll no longer unnecessarily load the models into memory on the web
workers[1].
- We can now score multiple models against the same revision ID for
(essentially) free[2].
- Our precaching system will take advantage of this to drop load by
about 3X[3].
- Update wmflabs deploy repo for new version of ORES[4].
*Documentation & maintenance:*
- We completed deployment and maintenance docs for Wiki labels[5], which
means we've now got complete docs for our systems[6].
- We implemented basic continuous integration tests for the ORES
extension[7].
*Downtime:*
- We had a 1 hour long downtime while trying to deploy new code to
ores.wikimedia.org[8]. We've filed two critical tasks for making sure
we don't make the mistake again[9,10].
1. https://phabricator.wikimedia.org/T134606 - Score multiple models with
the same cached dependencies
2. https://phabricator.wikimedia.org/T139407 - Don't load models into
memory of web workers
3. https://phabricator.wikimedia.org/T141376 - Update precached to group
requests by model
4. https://phabricator.wikimedia.org/T141377 - Update wmflabs deploy repo
for new version of ORES
5. https://phabricator.wikimedia.org/T131768 - Wikilabels deployment docs
6. https://phabricator.wikimedia.org/T106271 - Document maintenance tasks
7. https://phabricator.wikimedia.org/T140455 - CI test for ORES extension
8. https://wikitech.wikimedia.org/wiki/Incident_documentation/20160801-ORES
9. https://phabricator.wikimedia.org/T141823 - Set up password on ORES Beta
redis server
10. https://phabricator.wikimedia.org/T141825 - Config beta ORES extension
to use the beta ORES service
Sincerely,
Aaron from the Revision Scoring team