All,
The Beta Cluster[0] is currently not being updated with new code[1].
That normally happens every ~10 minutes.
The issue is that we were not prepared for the php5.3 -> 5.5 transition;
the deployment bastion that runs the code update ('scap') is a Precise
image. We (Tyler) are currently working on rebuilding it with a Trusty
image to solve this problem.
I know how frustrating this is for everyone (believe me) and I apologize
on behalf of Release Engineering for the second fallout of the php5.3 ->
5.5 transition (the first being yesterday's CI resource exhaustion).
I will update the list when Beta Cluster is back to normal.
Greg
[0] https://www.mediawiki.org/wiki/Beta_Cluster
[1] https://phabricator.wikimedia.org/T126573 with a blocker on
https://phabricator.wikimedia.org/T126568
--
| Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |
Hello all,
I want to apologize for the current slowness in our CI stack. Everything
is working fine in the sense that nothing is broken.
We simply did not adequately prepare for the switch to php5.5 being the
new minimum version for MediaWiki. This resulted in a large influx of
jobs on a set of our Jenkins executers that was not a large enough pool.
Timo and Antoine have created more executers to help mitigate [0].
More explicitly: due to package version availability in Ubuntu, our
php5.3 jobs run on our pool of Ubuntu Precis hosts, while the
php5.5/hhvm jobs run on our pool of Ubuntu Trusty hosts.
Both sets of jobs are in our pipeline to migrate to the new Nodepool
system that allows burstable availability of executers.
Again, apologies for the delays but know that the jobs will complete
just not as fast as you are accustomed to.
Greg
[0] https://phabricator.wikimedia.org/T126423
--
| Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |
Hi everyone,
I am a third year Computer Science undergraduate from India. I am
interested in contributing to Wikimedia Foundation.Can anyone help me get
started?
I am currently working on making a Wikipedia Search Engine for ~50 GB of
Wikipedia data . The project involves applying various indexing and
retrieving techniques to make the engine scalable and efficient.
Thank you.
Regards,
Arushi Dogra
It is possible to download extensions in ZIP format from the WMF repo
using a link like this:
http://git.wikimedia.org/zip/?r=mediawiki/extensions/FooBar.git&h=someTag&f…
However, this will produce a package with the extension's files in its
root folder. An unsuspecting user will probably simply extract this
package into the .../extensions folder with all its files ending up
there instead of in the extensions subfolder.
The ExtensionDistributor will provide correctly built packages, but it
is apparently only working on MW release tags, e.g. REL1_26.
Is there any way to download correctly built packages with tags other
than the MW relases?
Stephan
correcting date in subj line
On Wed, Feb 10, 2016 at 2:57 PM, Grace Gellerman <ggellerman(a)wikimedia.org>
wrote:
> https://www.mediawiki.org/wiki/Scrum_of_scrums/2016-02-10
>
> = 2015-02-10 =
>
> == Technology ==
>
> === Analytics ===
> * '''*Blocking'''*: Nobody we know of
> * '''*Blocked'''*: By nothing we're aware of
> * '''*Updates*'''*:*
>
> - ** organizing work to replace limn dashboards with dashiki
>
>
> - ** wikimetrics program global metrics deployed
>
>
> - ** moved wikimetrics from vagrant to docker-compose and it was a
> pleasant experience
>
>
> - ** working on cleaning up / organizing analytics files on
> dumps.wikimedia.org/other
>
>
> - ** estimating uniques with last access cookies is in the final
> stages of being productionized
>
>
> - ** Joseph's sanitization of pageview data (for long term retention
> and privacy) is up for review by halfak and anyone else interested
>
>
> === Architecture ===
> * '''*Blocking'''*: ???
> * '''*Blocked'''*: ???
> * '''*Updates*'''*:*
> ** ???
>
> === Performance ===
> * '''*Blocking'''*: ???
> * '''*Blocked'''*: ???
> * '''*Updates*'''*:*
> ** ???
>
> === Release Engineering ===
> * '''*Blocking'''*:
> ** https://phabricator.wikimedia.org/T125003 - move CI slaves in Beta to
> Jessie and Node 4.2
> * '''*Blocked'''*: none
> * '''*Updates*'''*:*
> ** Train deploy rolling out wmf.13 this week
> ** CI had some trouble due to lack of precise executioners on Monday,
> should be fixed now
> ** Lots of scap changes in puppet getting lined up: provider,
> scap::target, service::node
>
> === Research ===
> * '''*Blocking'''*:
> * '''*Blocked'''*:
> * '''*Updates*'''*:*
> ** Reader survey will go out (most probably, if all the technical
> components work smoothly) as part of the deployment train tomorrow
> (Thursday)
> *** Working with the mobile web team
> *** Survey will run for a week in enwiki, using QuickSurveys and we expect
> to collect between 200-500K responses
> *** Sampling rates will be finalized today
> *** Documentation:
> https://meta.wikimedia.org/wiki/Research:Characterizing_Wikipedia_Reader_Be…
>
> === Security ===
> * '''*Blocking'''*:
> ** Maps: Security review for PHP schema composer module -
> https://gerrit.wikimedia.org/r/#/c/269325/
> ** Graphs: Security review for SVG sanitization -
> https://github.com/cure53/DOMPurify
> * '''*Blocked'''*: Waiting to see how SessionManger goes before we deploy
> a bunch of patches
> * '''*Updates*'''*:*
> ** 2FA for Keystone done, working on Horizon integration next
>
> === Services ===
> * '''*Blocking'''*: ???
> * '''*Blocked'''*:
> ** https://phabricator.wikimedia.org/T125003 - move CI slaves in Beta to
> Jessie and Node 4.2
> ** https://phabricator.wikimedia.org/T107307 - move CXServer to SCB
> ** https://phabricator.wikimedia.org/T107306 - Apertium pkgs for Jessie
> * '''*Updates*'''*:*
> ** separated out the framework part of RESTBase into
> https://github.com/wikimedia/hyperswitch
> ** Testing Brotli compression for Cassandra in staging, seeing OOMs
> ** restbase1007 bootstrapping
> *** rb1001 and rb1002 might run out of disk space because of it
> ** separate AQS off of RESTBase -
> https://phabricator.wikimedia.org/T126294
>
> === Technical Operations ===
> * '''*Blocking'''*: ORES
> * '''*Blocked'''*: none known
> * '''*Updates*'''*:*
> ** OTRS upgrade to 5.0.6
> ** MultiDC work ongoing with the rest of Technology
>
> == Product ==
> === Community Tech ===
> * '''*Blocking'''*: none
> * '''*Blocked'''*: none
> * '''*Updates*'''*:*
> ** Continuing work on Gadgets 2.0 after recieving further feedback
> ** Continuing work on PageAssessments extension
>
> === Discovery ===
> * '''*Blocking'''*: none
> * '''*Blocked'''*:
> ** Maps: Security review for PHP schema composer module -
> https://gerrit.wikimedia.org/r/#/c/269325/
> ** Graphs: Security review for SVG sanitization -
> https://github.com/cure53/DOMPurify
> * '''*Updates*'''*:*
> ** Completion suggester changes done, probably not deployed yet due to
> SessionManager things
> ** Tuning suggester & morelike performance
> ** TextCat language detection merged, will work on making A/B test with it
> next
> ** Working on next year plans/budgets
> ** WDQS Blazegraph 2.0 update rolled back due to serious bug,
> investigating the bug
> ** Maps: blocked on security to deploy Kartographer extension
> ** Graphs: disabled http / https protocols, use custom protocols now
> ** Data: begining data storage planning (Yuri will add task # here :)
>
> === Editing ===
> ==== Collaboration ====
> * '''*Blocking'''*:
> ** Dry run patch for external store migration is merged. Now we need to
> set External Store up on Beta, then test the dry run patch there:
> https://phabricator.wikimedia.org/T119567
> * '''*Blocked'''*:
> ** Flow dump generation on dumps.wikimedia.org:
> https://phabricator.wikimedia.org/T119511
> * '''*Updates*'''*:*
> ** Human-readable name patch is almost done:
> https://phabricator.wikimedia.org/T121936
> ** Cross-wiki notifications will soon be rolled out to "real" wikis
> (currently in production it is only on test and test2):
> *** 'testwikidatawiki'
> *** 'wikidatawiki'
> *** 'mediawikiwiki'
> *** 'commonswiki'
> *** 'hewiki'
> *** 'hewikibooks'
> *** 'hewikinews'
> *** 'hewikiquote'
> *** 'hewikisource'
> *** 'hewikivoyage'
> *** 'hewiktionary'
> *** 'frwiki'
> *** 'frwikibooks'
> *** 'frwikinews'
> *** 'frwikiquote'
> *** 'frwikisource'
> *** 'frwikiversity'
> *** 'frwikivoyage'
> *** 'frwiktionary'
>
> ==== Language ====
> * '''*Blocking'''*:
> ** I have got some pings outside SoS, but don't know if those are blocking
> ** Packages for Jessie requested to speed up
> * '''*Blocked'''*: nope
> * '''*Updates*'''*:*
> **
>
> ==== Multimedia ====
>
> * '''*Blocking'''*: Not that I know of.
> * '''*Blocked'''*: *Metrics work* currently paused pending some burning
> questions for *halfak*, MarkTraceur to sit down with him this week. *Look
> out, Security*, we're hoping to get security review on a *new extension* (see
> below) in the next few months.
> * '''*Updates*'''*:*
> ** Labs instance with *ImageTweaks* (name WIP) going up today, *email to
> (wikitech|commons|multimedia)-l* later tonight
> ** Some work on *NewFiles*, in particular *filtering by tag*, but it's
> not going super. MatmaRex wanted some extra eyes on it, but I think he's
> contacted someone already.
>
>
> ==== Parsing ====
> * '''*Blocking'''*: ???
> * '''*Blocked'''*: ???
> * '''*Updates*'''*:*
> ** The templatedata-based serialization patch (
> https://gerrit.wikimedia.org/r/#/c/264043/ ) didn't go out this week
> after all -- yet to be reviewed. Possible it will get reviewed this week.
> ** Parsoid testing services (parsoid rt-testing; parsoid <--> php-parser
> html visual diff testing) fully puppetized and operational on ruthenium.
> Work ongoing to enable visual diff testing between production mediawiki and
> a non-production mediawiki (to test for example changes to parsing) using
> labs hardware ( https://phabricator.wikimedia.org/T120345 ).
> *** Would appreciate help from anyone familiar with hiera (to configure
> labs hardware to use existing puppet roles, and making any additional
> tweaks).
> ** This week, Arlo will start Parsoid-side work to separate out data-mw
> into its own RESTBase bucket -- deployment is still some ways away. We need
> to finalize REST API versioning policy (
> https://phabricator.wikimedia.org/T124365 ) and also have all the various
> Parsoid clients updated. We will create phab tickets for dependent projects
> once T124365 is resolved.
> *** Last week, Flow, Services, and us had a meeting and figured out a
> strategy for Flow since they use VE, but also talk with Parsoid (and not
> RESTBase).
> ** Parsoid native implementation for <gallery> is underway (heads up: CX,
> VE, Flow, Reading)
> *** https://phabricator.wikimedia.org/P2506 is the spec for Parsoid's
> output if you want to provide feedback
> *** https://gerrit.wikimedia.org/r/#/c/264026/ is the WIP patch
> ** Tim working to collect source location information in the PHP
> preprocessor for errors / warnings and a way to expose them. Useful in
> general, but, also specifically in the context of work related to
> replacying Tidy.
>
> ==== VisualEditor ====
> * '''*Blocking'''*: ???
> * '''*Blocked'''*: ???
> * '''*Updates*'''*:*
> ** ???
>
> === Fundraising Tech ===
> No blockers, most work is continued from last week
> * updates for new CiviCRM financial tracking
> * fixes and enhancements for backup credit card processor
> * prep for Latin America fundraising expansion
> * debugging email clickthrough tracking
>
> === Reading ===
>
> ==== Android ====
> * Updates: beta v2.1.140 released but will not be promoted due Wikipedia
> Zero incompatibilities. v2.1.141 coming soon.
> * Blocked: the options API was not designed for storing collection-like
> data. Gather collections don't support inter-wiki pages.
>
> ==== Reading Infrastructure ====
> * SessionManager is coming back this week. Please let Brad, Bryan, and
> Gergő know if you spot any problems. Preferably before Friday afternoon.
>
>
>
>
https://www.mediawiki.org/wiki/Scrum_of_scrums/2016-02-10
= 2015-02-10 =
== Technology ==
=== Analytics ===
* '''*Blocking'''*: Nobody we know of
* '''*Blocked'''*: By nothing we're aware of
* '''*Updates*'''*:*
- ** organizing work to replace limn dashboards with dashiki
- ** wikimetrics program global metrics deployed
- ** moved wikimetrics from vagrant to docker-compose and it was a
pleasant experience
- ** working on cleaning up / organizing analytics files on
dumps.wikimedia.org/other
- ** estimating uniques with last access cookies is in the final stages
of being productionized
- ** Joseph's sanitization of pageview data (for long term retention and
privacy) is up for review by halfak and anyone else interested
=== Architecture ===
* '''*Blocking'''*: ???
* '''*Blocked'''*: ???
* '''*Updates*'''*:*
** ???
=== Performance ===
* '''*Blocking'''*: ???
* '''*Blocked'''*: ???
* '''*Updates*'''*:*
** ???
=== Release Engineering ===
* '''*Blocking'''*:
** https://phabricator.wikimedia.org/T125003 - move CI slaves in Beta to
Jessie and Node 4.2
* '''*Blocked'''*: none
* '''*Updates*'''*:*
** Train deploy rolling out wmf.13 this week
** CI had some trouble due to lack of precise executioners on Monday,
should be fixed now
** Lots of scap changes in puppet getting lined up: provider, scap::target,
service::node
=== Research ===
* '''*Blocking'''*:
* '''*Blocked'''*:
* '''*Updates*'''*:*
** Reader survey will go out (most probably, if all the technical
components work smoothly) as part of the deployment train tomorrow
(Thursday)
*** Working with the mobile web team
*** Survey will run for a week in enwiki, using QuickSurveys and we expect
to collect between 200-500K responses
*** Sampling rates will be finalized today
*** Documentation:
https://meta.wikimedia.org/wiki/Research:Characterizing_Wikipedia_Reader_Be…
=== Security ===
* '''*Blocking'''*:
** Maps: Security review for PHP schema composer module -
https://gerrit.wikimedia.org/r/#/c/269325/
** Graphs: Security review for SVG sanitization -
https://github.com/cure53/DOMPurify
* '''*Blocked'''*: Waiting to see how SessionManger goes before we deploy a
bunch of patches
* '''*Updates*'''*:*
** 2FA for Keystone done, working on Horizon integration next
=== Services ===
* '''*Blocking'''*: ???
* '''*Blocked'''*:
** https://phabricator.wikimedia.org/T125003 - move CI slaves in Beta to
Jessie and Node 4.2
** https://phabricator.wikimedia.org/T107307 - move CXServer to SCB
** https://phabricator.wikimedia.org/T107306 - Apertium pkgs for Jessie
* '''*Updates*'''*:*
** separated out the framework part of RESTBase into
https://github.com/wikimedia/hyperswitch
** Testing Brotli compression for Cassandra in staging, seeing OOMs
** restbase1007 bootstrapping
*** rb1001 and rb1002 might run out of disk space because of it
** separate AQS off of RESTBase - https://phabricator.wikimedia.org/T126294
=== Technical Operations ===
* '''*Blocking'''*: ORES
* '''*Blocked'''*: none known
* '''*Updates*'''*:*
** OTRS upgrade to 5.0.6
** MultiDC work ongoing with the rest of Technology
== Product ==
=== Community Tech ===
* '''*Blocking'''*: none
* '''*Blocked'''*: none
* '''*Updates*'''*:*
** Continuing work on Gadgets 2.0 after recieving further feedback
** Continuing work on PageAssessments extension
=== Discovery ===
* '''*Blocking'''*: none
* '''*Blocked'''*:
** Maps: Security review for PHP schema composer module -
https://gerrit.wikimedia.org/r/#/c/269325/
** Graphs: Security review for SVG sanitization -
https://github.com/cure53/DOMPurify
* '''*Updates*'''*:*
** Completion suggester changes done, probably not deployed yet due to
SessionManager things
** Tuning suggester & morelike performance
** TextCat language detection merged, will work on making A/B test with it
next
** Working on next year plans/budgets
** WDQS Blazegraph 2.0 update rolled back due to serious bug, investigating
the bug
** Maps: blocked on security to deploy Kartographer extension
** Graphs: disabled http / https protocols, use custom protocols now
** Data: begining data storage planning (Yuri will add task # here :)
=== Editing ===
==== Collaboration ====
* '''*Blocking'''*:
** Dry run patch for external store migration is merged. Now we need to
set External Store up on Beta, then test the dry run patch there:
https://phabricator.wikimedia.org/T119567
* '''*Blocked'''*:
** Flow dump generation on dumps.wikimedia.org:
https://phabricator.wikimedia.org/T119511
* '''*Updates*'''*:*
** Human-readable name patch is almost done:
https://phabricator.wikimedia.org/T121936
** Cross-wiki notifications will soon be rolled out to "real" wikis
(currently in production it is only on test and test2):
*** 'testwikidatawiki'
*** 'wikidatawiki'
*** 'mediawikiwiki'
*** 'commonswiki'
*** 'hewiki'
*** 'hewikibooks'
*** 'hewikinews'
*** 'hewikiquote'
*** 'hewikisource'
*** 'hewikivoyage'
*** 'hewiktionary'
*** 'frwiki'
*** 'frwikibooks'
*** 'frwikinews'
*** 'frwikiquote'
*** 'frwikisource'
*** 'frwikiversity'
*** 'frwikivoyage'
*** 'frwiktionary'
==== Language ====
* '''*Blocking'''*:
** I have got some pings outside SoS, but don't know if those are blocking
** Packages for Jessie requested to speed up
* '''*Blocked'''*: nope
* '''*Updates*'''*:*
**
==== Multimedia ====
* '''*Blocking'''*: Not that I know of.
* '''*Blocked'''*: *Metrics work* currently paused pending some burning
questions for *halfak*, MarkTraceur to sit down with him this week. *Look
out, Security*, we're hoping to get security review on a *new extension* (see
below) in the next few months.
* '''*Updates*'''*:*
** Labs instance with *ImageTweaks* (name WIP) going up today, *email to
(wikitech|commons|multimedia)-l* later tonight
** Some work on *NewFiles*, in particular *filtering by tag*, but it's not
going super. MatmaRex wanted some extra eyes on it, but I think he's
contacted someone already.
==== Parsing ====
* '''*Blocking'''*: ???
* '''*Blocked'''*: ???
* '''*Updates*'''*:*
** The templatedata-based serialization patch (
https://gerrit.wikimedia.org/r/#/c/264043/ ) didn't go out this week after
all -- yet to be reviewed. Possible it will get reviewed this week.
** Parsoid testing services (parsoid rt-testing; parsoid <--> php-parser
html visual diff testing) fully puppetized and operational on ruthenium.
Work ongoing to enable visual diff testing between production mediawiki and
a non-production mediawiki (to test for example changes to parsing) using
labs hardware ( https://phabricator.wikimedia.org/T120345 ).
*** Would appreciate help from anyone familiar with hiera (to configure
labs hardware to use existing puppet roles, and making any additional
tweaks).
** This week, Arlo will start Parsoid-side work to separate out data-mw
into its own RESTBase bucket -- deployment is still some ways away. We need
to finalize REST API versioning policy (
https://phabricator.wikimedia.org/T124365 ) and also have all the various
Parsoid clients updated. We will create phab tickets for dependent projects
once T124365 is resolved.
*** Last week, Flow, Services, and us had a meeting and figured out a
strategy for Flow since they use VE, but also talk with Parsoid (and not
RESTBase).
** Parsoid native implementation for <gallery> is underway (heads up: CX,
VE, Flow, Reading)
*** https://phabricator.wikimedia.org/P2506 is the spec for Parsoid's
output if you want to provide feedback
*** https://gerrit.wikimedia.org/r/#/c/264026/ is the WIP patch
** Tim working to collect source location information in the PHP
preprocessor for errors / warnings and a way to expose them. Useful in
general, but, also specifically in the context of work related to
replacying Tidy.
==== VisualEditor ====
* '''*Blocking'''*: ???
* '''*Blocked'''*: ???
* '''*Updates*'''*:*
** ???
=== Fundraising Tech ===
No blockers, most work is continued from last week
* updates for new CiviCRM financial tracking
* fixes and enhancements for backup credit card processor
* prep for Latin America fundraising expansion
* debugging email clickthrough tracking
=== Reading ===
==== Android ====
* Updates: beta v2.1.140 released but will not be promoted due Wikipedia
Zero incompatibilities. v2.1.141 coming soon.
* Blocked: the options API was not designed for storing collection-like
data. Gather collections don't support inter-wiki pages.
==== Reading Infrastructure ====
* SessionManager is coming back this week. Please let Brad, Bryan, and
Gergő know if you spot any problems. Preferably before Friday afternoon.
Hi,
<https://gerrit.wikimedia.org/r/266931> has been merged, so MediaWiki
core now requires PHP 5.5.9 or higher to run. We had to make some (read:
a lot) of CI changes for that to happen, here's a quick summary:
* php53 jobs are only triggered for REL1_2[3-6] branches
* php55 jobs are only triggered for branches that are not REL1_2[3-6]
* All extensions that previously had php53 tests now run them under both
hhvm and php55
* composer related jobs were renamed to standardize with other CI jobs
Since this is a large change for extensions, I went ahead and ran jobs
for all extensions that have unittests, and collected the results:
<https://www.mediawiki.org/wiki/User:Legoktm/PHP_5.5/Extensions>. Please
take a look to see if your extension(s) are failing, and fix or file
bugs as needed! Some of the failures aren't even related to HHVM/PHP5.5,
the repos have just been broken.
And if you run into any CI related issues, please file a bug in the
#CI-Config Phab project!
-- Legoktm
I am pleased to announce that Chris Koerner has joined the Wikimedia
Foundation as Community Liaison on the Technical Collaboration team
<https://meta.wikimedia.org/wiki/Technical_Collaboration>. His main
responsibility is to support community engagement activities for projects
led by the Discovery team
<https://www.mediawiki.org/wiki/Wikimedia_Discovery>.
https://meta.wikimedia.org/wiki/User:CKoerner_(WMF)
Chris started his approach to the Wikimedia movement as administrator of a
third party MediaWiki’s. He will be working remotely from Saint Louis, MO
(USA). Here is an introduction in his own words:
"I've been a MediaWiki fan since 2011, having used it inside a large
healthcare organization to share knowledge among co-workers. From there I
fell into the the larger Wikimedia rabbit hole and haven't looked back. I'm
active in the MediaWiki Stakeholders' Group
<https://www.mediawiki.org/wiki/MediaWiki_Stakeholders%27_Group>, a user
group of third-party MediaWiki administrators, developers, and users who
advocate for using MediaWiki. I also edit from time to time on the English
Wikipedia.
I've always loved technology and people. There's nothing like the feeling
of sitting next to someone as they discover how they can use a bit of
technology to solve a need. I've worked in desktop support, web
development, marketing, and healthcare IT throughout my career. All along
the way I've valued communication and transparency and love to encourage
others to get involved.
I'm an enthusiast photographer, a person who enjoys board and video games,
and a total nerd who rides a motorcycle. Meaning, the bike looks cool, but
not me."
Please join me in welcoming Chris!
--
Quim Gil
Engineering Community Manager @ Wikimedia Foundation
http://www.mediawiki.org/wiki/User:Qgil