Hello all!
We have been hard at work on our Graph Split experiment [1], and we now
have a working graph split that is loaded onto 3 test servers. We are
running tests on a selection of queries from our logs to help understand
the impact of the split. We need your help to validate the impact of
various use cases and workflows around Wikidata Query Service.
**What is the WDQS Graph Split experiment?**
We want to address the growing size of the Wikidata graph by splitting it
into 2 subgraphs of roughly half the size of the full graph, which should
support the growth of Wikidata for the next 5 years. This experiment is
about splitting the full Wikidata graph into a scholarly articles subgraph
and a “main” graph that contains everything else.
See our previous update for more details [2].
**Who should care?**
Anyone who uses WDQS through the UI or programmatically should check the
impact on their use cases, scripts, bots, code, etc.
**What are those test endpoints?**
We expose 3 test endpoints, for the full, main and scholarly articles
graphs. Those graphs are all created from the same dump and are not live
updated. This allows us to compare queries between the different endpoints,
with stable / non changing data (the data are from the middle of October
2023).
The endpoints are:
* https://query-full-experimental.wikidata.org/
* https://query-main-experimental.wikidata.org/
* https://query-scholarly-experimental.wikidata.org/
Each of the endpoints is backed by a single dedicated server of performance
similar to the production WDQS servers. We don’t expect performance to be
representative of production due to the different load and to the lack of
updates on the test servers.
**What kind of feedback is useful?**
We expect queries that don’t require scholarly articles to work
transparently on the “main” subgraph. We expect queries that require
scholarly articles to need to be rewritten with SPARQL federation between
the “main” and scholarly subgraphs (federation is supported for some
external SPARQL servers already [3], this just happens to be for internal
server-to-server communication). We are doing tests and analysis based on a
sample of query logs.
**We want to hear about:**
General use cases or classes of queries which break under federation
Bots or applications that need significant rewrite of queries to work with
federation
And also about use cases that work just fine!
Examples of queries and pointers to code will be helpful in your feedback.
**Where should feedback be sent?**
You can reach out to us using the project’s talk page [1], the Phabricator
ticket for community feedback [4] or by pinging directly Sannita (WMF) [5].
**Will feedback be taken into account?**
Yes! We will review feedback and it will influence our path forward. That
being said, there are limits to what is possible. The size of the Wikidata
graph is a threat to the stability of WDQS and thus a threat to the whole
Wikidata project. Scholarly articles is the only split we know of that
would reduce the graph size sufficiently. We can work together on providing
support for a migration, on reviewing the rules used for the graph split,
but we can’t just ignore the problem and continue with a WDQS that provides
transparent access to the full Wikidata graph.
Have fun!
Guillaume
[1]
https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_graph_split
[2]
https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_backend_up…
[3]
https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual#Federation
[4] https://phabricator.wikimedia.org/T356773
[5] https://www.wikidata.org/wiki/User:Sannita_(WMF)
--
Guillaume Lederrey (he/him)
Engineering Manager
Wikimedia Foundation
Hello all!
The Search Platform Team usually holds an open meeting on the first
Wednesday of each month. Come talk to us about anything related to
Wikimedia search, Wikidata Query Service (WDQS), Wikimedia Commons Query
Service (WCQS), etc.!
Feel free to add your items to the Etherpad Agenda for the next meeting.
Details for our next meeting:
Date: Wednesday, February 7, 2024
Time: 16:00-17:00 UTC / 08:00 PST / 11:00 EST / 17:00 CET
Etherpad: https://etherpad.wikimedia.org/p/Search_Platform_Office_Hours
Google Meet link: https://meet.google.com/vgj-bbeb-uyi
Join by phone: https://tel.meet/vgj-bbeb-uyi?pin=8118110806927
Have fun and see you soon!
Guillaume
--
*Guillaume Lederrey* (he/him)
Engineering Manager
Wikimedia Foundation <https://wikimediafoundation.org/>
Hello all,
As we continue with work towards the grid engine deprecation[0], we are
following through with the timeline[1] shared.
We have reached out to each maintainer via email and talk pages with
reminders.
On the 14th of this month(February), we will begin to stop tools that are
still running on the grid.
Tools that have had their maintainers reach out and are actively migrating,
can ask for extensions and will not be stopped.
Once a tool is stopped, if the maintainer has a clear plan for migrating,
they can request in the tool-specific migration task for the tool to be
re-enabled (although they will be shut down again if they miss the March
deadline).
If you have a tool that is still on the grid and you cannot meet the above
deadline, kindly reach out via the tool migration phabricator ticket or our
support channels[2]
[0]:
https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_deprecation
[1]:
https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_deprecation#…
[2]:
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/About_Toolforge#Commun…
--
Seyram Komla Sapaty
Developer Advocate
Wikimedia Cloud Services
Hi All,
Happy 2024 and welcome to the monthly MediaWiki Insights email!
We’re starting this year with celebrating contributors who got their first
patch merged in MW core, WMF deployed extensions or services in the months
of December and January:
A big thanks to Doğu Abaris, apasternak, Abador, Mehdi Zidani,
Oudedutchman, Cyn, Dominic mayers, and Houseblaster for their
contributions! Welcome :-)
Enable more people to know MediaWiki and contribute effectively
As shared in the previous MW insights email
<https://www.mediawiki.org/wiki/MediaWiki_Product_Insights/Reports/November_…>,
we have been thinking about ways to improve first time MediaWiki (core)
contributors’ experience. One key part of that is knowing who is new to be
able to say hi (and thank you). Bartosz created a script
<https://gitlab.wikimedia.org/matmarex/gerrit-new-contributors/-/blob/master…>
to find new MediaWiki contributors for a given month that got their first
patch merged (see above!). Another key part is code review for patches
submitted by (new) volunteer contributors. We’re currently thinking about
how we could organize code review for volunteer-submitted patches better
across MediaWiki Engineering and want to give a shared MW-Engineering
Gerrit dashboard a try, with primary focus on the components the group is
responsible for.
We ran a WMF-internal MediaWiki CodeJam event
<https://www.mediawiki.org/wiki/MediaWiki_Code_Jam#Results> in December,
with 54 participants (16 from the MW team) and 21 Phabricator tickets
completed. Areas of improvement included MediaWiki itself and multiple
essential extensions. See the project page for details!
Two notable “side projects” of the CodeJam: To support new contributors,
Timo has created a series of tutorials
<https://www.mediawiki.org/wiki/User:Krinkle/MediaWiki_Introduction_2023>
going over core MediaWiki concepts. The videos are also available on
youtube (part I: MediaWiki core concepts
<https://www.youtube.com/watch?v=-JnIjpRvgNY>, part II: Wikipedia’s
extensions <https://www.youtube.com/watch?v=4xYbqbabTwI>).
A "Quick" MW install
<https://www.mediawiki.org/wiki/Local_development_quickstart> was created
to make it possible to install MediaWiki in 3 steps and under 5 minutes.
See this ticket for the evaluation
<https://phabricator.wikimedia.org/T348899> on how this worked out for
testers during code jam. Thanks to Alex Paskulin, Timo Tijhof and Kosta
Harlan for their work <3
The materials as well as lessons learned from this first MW code jam will
help us run a second edition for everyone who’s interested, possibly ahead
of, and at the Wikimedia Hackathon
<https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2024>.
Project snapshot: Virtual domains, great namespaceisation, and first MW
Engineering cross-sub-team quarterly planning
We finalized the Q3 plan (= Jan-March) for MediaWiki Engineering in early
January. This is the first time where we applied a “cross-sub-team”
planning approach across MediaWiki Engineering & engaged as a group on
gathering proposals and then prioritized these. A few examples for high
priority work:
- Evolve central login to meet requirements of a changing environment
(“SUL3”): T348388 <https://phabricator.wikimedia.org/T348388>
- Support upgrade of WMF production from PHP 7.4 to 8.1: T319432
<https://phabricator.wikimedia.org/T319432>
- Continue work to move off RESTBase
<https://phabricator.wikimedia.org/project/profile/6289/>
- Continue work on exploring how essential workflows on the Wikimedia
projects are currently supported through the MediaWiki software ecosystem
(see below)
- Prepare MW-platform team owned components
<https://phabricator.wikimedia.org/T355377> for compatibility with the
upcoming Temporary Accounts for Unregistered Editors
<https://www.mediawiki.org/wiki/Help:Temporary_accounts> project.
We’ve already checked the box on preparing auth components
<https://phabricator.wikimedia.org/T326925> (OATHAuth and WebAuthn) for
compatibility - many thanks to Piotr and Gergö for their work on this!
Other highlights:
- We’ve made progress towards supporting Less 3.13
<https://phabricator.wikimedia.org/T288498> behavior: A series of bugs
were fixed in Less v2.5.3 to pave the way to enable new Less.js
capabilities to be used in the frontend.
- The Content Transform team has moved functionality out of Parsoid and
into Cite extension <https://phabricator.wikimedia.org/T354215>, which
allows both Parsoid and the default parser to use the same underlying
functionality and reduce maintenance burdens.
- To support SRE’s work, MediaWiki metrics were migrated from Graphite
to Prometheus. This is an ongoing effort with more upcoming work.
- Cleanup continued in dropping deprecated unused config variables
<https://phabricator.wikimedia.org/T166010> (also known as the great
namespaceisation effort).
- Virtual domains are now working well in MediaWiki core
<https://phabricator.wikimedia.org/T353948> and are supported in the
updater and installer.
- A minor task was to update the JavaScript syntax checker for gadgets
and user scripts for ES6 and ES7
<https://phabricator.wikimedia.org/T75714>.
- Database: Amir Sarabadani presented
<https://commons.wikimedia.org/wiki/File:Major_changes_to_MediaWiki%27s_rdbm…>
about changes done to the RDBMS library at SMW Con (video available
<https://www.youtube.com/watch?v=q0mjNEJP5Fo>). DBAccessObjectUtils is being
redesigned <https://phabricator.wikimedia.org/T354194>, including major
deprecation of indirect calls to IDBAccessObject constants.
How essential workflows are served through the MediaWiki software ecosystem
As part of developing a product strategy for MediaWiki, one research
question is how the MediaWiki software ecosystem currently supports
essential workflows on the Wikimedia projects. We also wanted to understand
commonalities and differences in how these workflows and use cases are
supported on different wikis - for example, Wikipedia A vs Wikipedia B,
Wikipedia A vs Wikidata, or Wikipedia A vs translatewiki.net - etc.
Moriel has been leading this work, in collaboration with many engineers
across different areas of expertise. A first outcome has now been
published: Unraveling Complexity: Mapping MediaWiki Software Components
into User-Driven Workflows
<https://www.mediawiki.org/wiki/MediaWiki_Product_Insights/Artifacts/Unravel…>
The next outcome is already in the making, with focus on exploring how
specific use cases are supported (differently) on 3 different Wikimedia
projects. The goal is to identify where conceptual behaviors diverge and
converge between the wikis, pinpointing base and common behaviors versus
those unique to specific cases. This brings us in the interesting fields of
software behavior on use cases such as: “An unregistered (not yet
temporary) user edits existing wikitext content (fixing typo) on desktop.”
Read the write-up above to learn more about why, and stay tuned for the
next outcome of this exploration, which will talk through some of the
findings in this multi-wiki modeling approach.
Thanks all for reading!
Birgit
--
Birgit Müller (she/her)
Director of Product, MediaWiki and Developer Experiences
Wikimedia Foundation <https://wikimediafoundation.org/>
Hello,
Community Wishathon is coming next month: March 15th-17th, 2024! So far, 78
participants have signed up to attend this online event since its first
announcement in December 2023 🎉
You can check out the project ideas and schedule on the event page at <
https://meta.wikimedia.org/wiki/Event:WishathonMarch2024>. These ideas come
from the wishes people share in Community Wishlist Survey <
https://meta.wikimedia.org/wiki/Community_Wishlist_Survey>. If you're a
user, developer, designer, or product lead, you are welcome to join! You
can sign up for the event on the wiki page and add your name, time zone,
and preferred contact method to a project by *February 15th, 2024*.
Before the event, you can learn about the projects and connect with others
who are interested. You can contact them using their preferred contact
method and consider forming project groups on Telegram.
During the event, there will be an opening session to introduce the event
and project ideas, async time for working on projects, self-organized
virtual project group meetings, and a showcase ceremony to share what each
group accomplished during the event. If you want to help with the event or
have questions, let us know on the event's talk page.
Cheers,
Srishti
On behalf of the Wishathon organizing committee
*Srishti Sethi*
Senior Developer Advocate
Wikimedia Foundation <https://wikimediafoundation.org/>
Hello everyone,
The second edition of the Language & Internationalization newsletter
(January 2024) is available at this link: <
https://www.mediawiki.org/wiki/Wikimedia_Language_engineering/Newsletter/20…
>.
This newsletter is compiled by the Wikimedia Language team. It provides
updates from October–December 2023 quarter on new feature development,
improvements in various language-related technical projects and support
efforts, details about community meetings, and contributions ideas to get
involved in projects.
To stay updated, you can subscribe to the newsletter on its wiki page. If
you have any feedback or ideas for topics to feature in the newsletter,
please share them on the discussion page, accessible here: <
https://www.mediawiki.org/w/index.php?title=Talk:Wikimedia_Language_enginee…
>.
Cheers,
Srishti
On behalf of the WMF Language team
*Srishti Sethi*
Senior Developer Advocate
Wikimedia Foundation <https://wikimediafoundation.org/>
Hi Community Metrics team,
This is your automatic monthly Phabricator statistics mail.
Accounts created in (2024-01): 321
Active Maniphest users (any activity) in (2024-01): 1102
Task authors in (2024-01): 555
Users who have closed tasks in (2024-01): 322
Projects which had at least one task moved from one column to another on
their workboard in (2024-01): 313
Tasks created in (2024-01): 2209
Tasks closed in (2024-01): 2257
Open and stalled tasks in total: 53680
* Only open tasks in total: 52675
* Only stalled tasks in total: 1005
Median age in days of open tasks by priority:
Unbreak now: 79
Needs Triage: 952
High: 1333
Normal: 2006
Low: 2571
Lowest: 2984
(How long tasks have been open, not how long they have had that priority)
To see the names of the most active task authors:
* Go to https://wikimedia.biterg.io/
* Choose "Phabricator > Overview" from the top bar
* Adjust the time frame in the upper right corner to your needs
* See the author names in the "Submitters" panel
TODO: Numbers which refer to closed tasks might not be correct, as
described in https://phabricator.wikimedia.org/T1003 .
Yours sincerely,
Fab Rick Aytor
(via community_metrics.sh on phab1004 at Thu 01 Feb 2024 12:00:25 AM UTC)