Hello services-list,
since my very first contact with JSON, I was asking myself: Why can't
people simply use XML?
In the meantime I got aware of the advantages, especially for fast
prototyping and high performance applications.
However, when applications get larger, more complex and more mature
the absence of schema information is problematic.
For example, I found writing the parser for the WikiData dump [1]
quite exhausting. Alternatives like Json-lib work well for testing,
but I was quite worried about stability after hitting a log tail bug
[2]. Moreover, in the PHP Math extension it's often uncomfortable to
figure out which JSON properties are set under certain circumstances
[3]. Yesterday, I discovered another problem related to a missing JSON
schema [4] which finally motivated me to start this effort to discuss
about JSON schema options.
For the communication between services, we use spec files. This is a
great thing. But would it not be even better to use a JSON schema even
within services. So one could throw exceptions right at the place
where the problem occurs. I'm aware that there are approaches for a
JSON schema like [5], but I'm not sure if that is convenient to use in
practice.
To keep the discussion focused, we could use "how HTTP errors are
supposed to look" [6] as a running example to discuss how JSON schema
definition and validation could work.
Best
Physikerwelt
PS: This is my first post the services-list. I hope it fits well to
the idea of this list.
[1] https://github.com/physikerwelt/WikidataListGenerator/blob/master/src/main/…
[2] https://twitter.com/physikerwelt/status/683286844721741824
[3] https://phabricator.wikimedia.org/T119300
[4] https://phabricator.wikimedia.org/T126057
[5] http://jsonschema.net
[6] https://github.com/wikimedia/service-template-node/blob/master/doc/coding.m…
This is now entering its final comment period, so please weigh in at
https://phabricator.wikimedia.org/T124365.
Based on your input, the Parsing, Editing & Services teams will make a
decision on this next Wednesday, Feb 2nd.
Thanks,
Gabriel
On Thu, Jan 21, 2016 at 4:29 PM, Gabriel Wicke <gwicke(a)wikimedia.org> wrote:
> Hi,
>
> we are considering a policy for REST API end point result format
> versioning and negotiation. The background and considerations are
> spelled out in a task and mw.org page:
>
> https://phabricator.wikimedia.org/T124365
> https://www.mediawiki.org/wiki/Talk:API_versioning
>
> Based on the discussion so far, have come up with the following
> candidate solution:
>
> 1) Clearly advise clients to explicitly request the expected mime type
> with an Accept header. Support older mime types (with on-the-fly
> transformations) until usage has fallen below a very low percentage,
> with an explicit sunset announcement.
>
> 2) Always return the latest content type if no explicit Accept header
> was specified.
>
> We are interested in hearing your thoughts on this.
>
> Once we have reached rough consensus on the way forward, we intend to
> apply the newly minted policy to an evolution of the Parsoid HTML
> format, which will move the data-mw attribute to a separate metadata
> blob.
>
> Gabriel Wicke
--
Gabriel Wicke
Principal Engineer, Wikimedia Foundation
A vulnerability has been found in RESTBase v0.9.1 and earlier that
allowed attackers to read arbitrary files on the host system by
passing a specially crafted URL. This vulnerability has been fixed in
[1].
All RESTBase users are strongly encouraged to upgrade to v0.9.2
immediately. Files readable by the RESTBase service user might have
been accessed by third parties, so appropriate measures should be
taken.
mediawiki-containers [2] users with automatic updates enabled have
already been upgraded to v0.9.2.
--
Gabriel Wicke
Principal Engineer, Wikimedia Foundation
[1]: https://github.com/wikimedia/restbase/commit/1ea649306ae4e85ab2cee5a36318e9…
[2]: https://github.com/wikimedia/mediawiki-containers
Hi, i see these
"graphoid endpoints health"
/{domain}/v1/{format}/{title}/{revid}/{id} is CRITICAL: Test retrieve PNG
from mediawiki.org returned the unexpected status 400 (expecting: 200)
on sca1001 and sca1002.
Is that the same thing?
https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=sca1001…https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=sca1002…
On Tue, Dec 15, 2015 at 4:52 AM, Marko Obrovac <mobrovac(a)wikimedia.org>
wrote:
> +cc ops-l.
>
> It seems that the Graph extension and Graphoid have changed the way the
> hash is being calculated, which resulted in 404s for RESTBase's check of
> Graphoid.
>
> I have changed the hash in RESTBase's monitoring specification and
> deployed it. All checks are passing now.
>
> Yuri, given that Graph(oid) is using RESTBase for its everyday operation,
> you need to notify us (Ops/Services) of such fundamental changes so that we
> can prepare and act accordingly (and synchronise deployment with you).
>
> Cheers,
> Marko
>
> On 15 December 2015 at 06:11, Gabriel Wicke <gwicke(a)wikimedia.org> wrote:
>
>> From the graphoid log:
>>
>> {"name":"graphoid","hostname":"sca1002","pid":187,"level":30,"domain":"
>> en.wikipedia.org","format":
>>
>> "png","title":"User:Pchelolo/Graph","revid":"670213569","id":"1533aaad45c733dcc7e07614b54cbae4119a
>>
>> 6747.png","apicall":{"format":"json","formatversion":"2","action":"graph","title":"User:Pchelolo/G
>>
>> raph","hash":"1533aaad45c733dcc7e07614b54cbae4119a6747"},"apiRetError":{"code":"invalidhash","info
>> ":"No graph found.","docref":"See https://en.wikipedia.org/w/api.php
>> for API usage"},"levelPath":"
>>
>> info/mwapi-error","request_id":"2645c162-a2ea-11e5-a91c-8766eb5ddb47","msg":"[object
>> Object]","tim
>> e":"2015-12-15T05:10:25.341Z","v":0}
>>
>> On Mon, Dec 14, 2015 at 9:03 PM, Gabriel Wicke <gwicke(a)wikimedia.org>
>> wrote:
>> > This errors out:
>> >
>> >
>> http://graphoid.wikimedia.org/mediawiki.org/v1/png/Extension:Graph/0/be66c7…
>> >
>> > On Mon, Dec 14, 2015 at 8:58 PM, Gabriel Wicke <gwicke(a)wikimedia.org>
>> wrote:
>> >> Hi Yuri,
>> >>
>> >> there were just some graphoid endpoint alerts from the RESTBase health
>> >> checks. Is this known / expected?
>> >>
>> >> Gabriel
>> >
>> >
>> >
>> > --
>> > Gabriel Wicke
>> > Principal Engineer, Wikimedia Foundation
>>
>>
>>
>> --
>> Gabriel Wicke
>> Principal Engineer, Wikimedia Foundation
>>
>>
>>
> --
> Marko Obrovac, PhD
> Senior Services Engineer
> Wikimedia Foundation
>
> _______________________________________________
> Ops mailing list
> Ops(a)lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/ops
>
>
--
Daniel Zahn <dzahn(a)wikimedia.org>
Operations Engineer
Hi folks,
I'd like to get your thoughts on which topics make sense as big room
conversations at the MediaWiki Developer Summit. Here's a list of
candidates:
https://phabricator.wikimedia.org/T86028
Current version of the text copy pasted below for the lazy, but there may
be updates by the time your read this. Please make your comments in
Phabricator.
Thanks
Rob
Service-oriented architecture
Discussion or talk?
- Short intro & view back
- Layering: Stateful vs. stateless services
- special challenges at the state layer: avoid SPOFs, replication and
consistency, scaling
- High-level functional groups / service candidates
- SOA support infrastructure
- RESTBase: storage, content API and service orchestration
- Auth service
- PHP API
- public vs. private APIs
Common design goals and issues
Panel discussion
- Demand management, "back pressure"
- Authentication
- Operational / practical: monitoring, deployment
- 3rd party users / distribution / scaling down
- Caching & invalidation strategies
- Required attendees: Gabriel Wicke, Services team, Faidon Liambotis,
Sean, Chris Steipp
Looking ahead: Virtualization, CI and continuous deployment
Panel discussion
- want to share hardware between services while improving isolation for
perf and security
- moving towards virtualization both in production and CI
- what does this mean for our deployment strategy / tools?
- dark deploys
<https://www.facebook.com/note.php?note_id=96390263919> with
deployment orchestration integrating with monitoring?
- do HET deploy with different VMs?
- role of betalabs vs. dark deploys
- Participants: @akosiaris
<https://phabricator.wikimedia.org/p/akosiaris/>, @greg
<https://phabricator.wikimedia.org/p/greg/>, @hashar
<https://phabricator.wikimedia.org/p/hashar/>, @Krinkle
<https://phabricator.wikimedia.org/p/Krinkle/>, @bd808
<https://phabricator.wikimedia.org/p/bd808/>, @cscott
<https://phabricator.wikimedia.org/p/cscott/>, @GWicke
<https://phabricator.wikimedia.org/p/GWicke/>, ?
SOA proliferation through specification
Talk / tutorial by @Jdouglas <https://phabricator.wikimedia.org/p/Jdouglas/>
- specification-driven service design
- specification-driven documentation
- Swagger UI sandboxes make services easy to grok
- low client-side development/consumption impedance
- promotes the creation of numerous and innovative (client) applications
- breakout/workshop: quick client generation with swagger-codegen?
The road to multi-DC operation
Should perhaps be covered by main panel, as it affects a lot of decisions
- issues: replication consistency, caches
- possible solutions
- how this affects our software architecture
Scaling down for third-party users
Should perhaps be covered by main panel, as it affects a lot of decisions
- define minimum resources for a basic MediaWiki install (strawman ex: $2.99
/ month VM with 1G of RAM <http://www.ovh.com/us/vps/vps-classic.xml>?)
- simple & small implementations of common services for testing and
small installs
- setup automation / packaging for VMs
Security in a distributed service environment
Talk / tutorial by @csteipp <https://phabricator.wikimedia.org/p/csteipp/>.
Tracked in more detail in T86049 <https://phabricator.wikimedia.org/T86049>.
- advantages of isolation between services vs. challenges of multitude
of entry points
- what we learned from experience in our current architecture
- what others learned from experience in a SOA setting
- how to address challenges: SOA auth RFC
<https://www.mediawiki.org/wiki/Requests_for_comment/Service-oriented_archit…>
- CSP and other headers
- appropriate use of CORS
- when to use OAuth
Ideas for content representation / UI / skins
Really in the 'editing' track. TODO: Figure out if other front-end stuff /
caching fits in there.
- Move to HTML primary storage / wikitext editing?
- Micro-contributions using Parsoid HTML: API needs, how it could look
- API-powered front-ends: mobile already very far
- Fast logged-in views for mobile web and desktop: ESI vs. client-side
customizations & no-JS fall-back
- again, mobile very close already
Platform choices for new services
Not convinced that this would really be productive
- SOA gives us freedom to choose (and change) technology per service /
task
- don't want to go overboard though, as each additional technology has a
cost
- current main development platforms: PHP, client-side JS & Node.js,
some Java
- trends: industry and WMF
- new candidates: Hack, Go, Rust
- relative strengths for our use cases
- successful outcome:
- shared understanding of relative strengths and trends
- possibly guidance on what to choose in which area
- awareness of new options
Minutes and slides from the recent quarterly review meeting of the
Foundation's Parsoid, Services and OCG (Offline content generator)
teams have appeared at
https://meta.wikimedia.org/wiki/WMF_Metrics_and_activities_meetings/Quarter…
.
On Wed, Dec 19, 2012 at 6:49 PM, Erik Moeller <erik(a)wikimedia.org> wrote:
> Hi folks,
>
> to increase accountability and create more opportunities for course
> corrections and resourcing adjustments as necessary, Sue's asked me
> and Howie Fung to set up a quarterly project evaluation process,
> starting with our highest priority initiatives. These are, according
> to Sue's narrowing focus recommendations which were approved by the
> Board [1]:
>
> - Visual Editor
> - Mobile (mobile contributions + Wikipedia Zero)
> - Editor Engagement (also known as the E2 and E3 teams)
> - Funds Dissemination Committe and expanded grant-making capacity
>
> I'm proposing the following initial schedule:
>
> January:
> - Editor Engagement Experiments
>
> February:
> - Visual Editor
> - Mobile (Contribs + Zero)
>
> March:
> - Editor Engagement Features (Echo, Flow projects)
> - Funds Dissemination Committee
>
> We’ll try doing this on the same day or adjacent to the monthly
> metrics meetings [2], since the team(s) will give a presentation on
> their recent progress, which will help set some context that would
> otherwise need to be covered in the quarterly review itself. This will
> also create open opportunities for feedback and questions.
>
> My goal is to do this in a manner where even though the quarterly
> review meetings themselves are internal, the outcomes are captured as
> meeting minutes and shared publicly, which is why I'm starting this
> discussion on a public list as well. I've created a wiki page here
> which we can use to discuss the concept further:
>
> https://meta.wikimedia.org/wiki/Metrics_and_activities_meetings/Quarterly_r…
>
> The internal review will, at minimum, include:
>
> Sue Gardner
> myself
> Howie Fung
> Team members and relevant director(s)
> Designated minute-taker
>
> So for example, for Visual Editor, the review team would be the Visual
> Editor / Parsoid teams, Sue, me, Howie, Terry, and a minute-taker.
>
> I imagine the structure of the review roughly as follows, with a
> duration of about 2 1/2 hours divided into 25-30 minute blocks:
>
> - Brief team intro and recap of team's activities through the quarter,
> compared with goals
> - Drill into goals and targets: Did we achieve what we said we would?
> - Review of challenges, blockers and successes
> - Discussion of proposed changes (e.g. resourcing, targets) and other
> action items
> - Buffer time, debriefing
>
> Once again, the primary purpose of these reviews is to create improved
> structures for internal accountability, escalation points in cases
> where serious changes are necessary, and transparency to the world.
>
> In addition to these priority initiatives, my recommendation would be
> to conduct quarterly reviews for any activity that requires more than
> a set amount of resources (people/dollars). These additional reviews
> may however be conducted in a more lightweight manner and internally
> to the departments. We’re slowly getting into that habit in
> engineering.
>
> As we pilot this process, the format of the high priority reviews can
> help inform and support reviews across the organization.
>
> Feedback and questions are appreciated.
>
> All best,
> Erik
>
> [1] https://wikimediafoundation.org/wiki/Vote:Narrowing_Focus
> [2] https://meta.wikimedia.org/wiki/Metrics_and_activities_meetings
> --
> Erik Möller
> VP of Engineering and Product Development, Wikimedia Foundation
>
> Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate
>
> _______________________________________________
> Wikimedia-l mailing list
> Wikimedia-l(a)lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
--
Tilman Bayer
Senior Operations Analyst (Movement Communications)
Wikimedia Foundation
IRC (Freenode): HaeB
Hello,
Please join us on the next Wikimedia bug day:
**2014-10-08, 14:00–22:00 UTC** [1] in #wikimedia-tech on Freenode IRC.[2]
We will be triaging bug reports for the Collection extension (Book tool)
in general and PDF export in particular, which were just switched to a
new backend (OCG).[3] We have two immediate goals:
1) recover 100 % of the relevant reports from the defunct PediaPress
tracker;[4]
2) get a clean list of known PDF issues that the new backend didn't fix.
Everyone is welcome to join any time these weeks, and no technical
knowledge is needed! It's an easy way to get involved or to give
something back.
We encourage you to record your activity on the etherpad [4].
This information and more can be found here:
https://www.mediawiki.org/wiki/Bug_management/Triage/201410
For more information on triaging in general, check out
https://www.mediawiki.org/wiki/Bug_management/Triage
I look forward to seeing you there. Please distribute further by email,
talk pages etc. (Collection is used on almost 2 thousands wikis!)
Sorry for the crossposting,
Nemo
[1] Timezone converter: http://everytimezone.com/#2014-10-08,120,5x1
[2] See http://meta.wikimedia.org/wiki/IRC for more info on IRC chat
[3] http://lists.wikimedia.org/pipermail/wikitech-l/2014-July/077867.html
[4] https://etherpad.wikimedia.org/BugTriage-Collection
