This Tuesday, Mar 14, at 21:00 UTC (14:00 PDT), we're planning a window
for updates to the production Phabricator instance. While we've
reserved 2 hours out of an abundance of caution, we intend for the
actual downtime to be much shorter.
This window is contingent on the status of the weekly deployment train.
If something important is broken in production, we'll push it to a later
date.
This update should include several minor UI improvements, but does not
otherwise change the code.
There's a Phab task for this release; as usual we'll track the work on
Etherpad and #wikimedia-operations once underway:
https://phabricator.wikimedia.org/T331915
Thanks,
--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
Hey all,
This is a quick note to highlight that in six weeks' time, the REL1_40
branch will be created for MediaWiki core and each of the extensions and
skins in Wikimedia git, with some (the 'tarball') included as sub-modules
of MediaWiki itself[0]. This is the first step in the release process for
MediaWiki 1.40, which should be out in May 2023, approximately six months
after MediaWiki 1.39.
The branches will reflect the code as of the last 'alpha' branch for the
release, 1.40.0-wmf.27, which will be deployed to Wikimedia wikis in the
week beginning 13 March 2023 for MediaWiki itself and those extensions
and skins available there.
After that point, patches that land in the main development branch of
MediaWiki and its bundled extensions and skins will be slated for the
MediaWiki 1.41 release unless specifically backported[1].
If you are working on a new feature that you wish to land for the release,
you now have a few days to finish your work and land it in the development
branch; feature changes should not be backported except in an urgent case.
If your work might not be complete in time, and yet should block release
for everyone else, please file a task against the `mw-1.40-release` project
on Phabricator.[2]
If you have tickets that are already tagged for `mw-1.40-release`, please
finish them, untag them, or reach out to get them resolved in the next few
weeks.
We hope to issue the first release candidate, 1.40.0-rc.0, two weeks after
the branch point, and if all goes well, to release MediaWiki 1.40.0 a few
weeks after that.
Tyler Cipriani (he/him)
Engineering Manager, Release Engineering
Wikimedia Foundation
[0]: <https://www.mediawiki.org/wiki/Bundled_extensions_and_skins>
[1]: <https://www.mediawiki.org/wiki/Backporting_fixes>
[2]: <https://phabricator.wikimedia.org/tag/mw-1.40-release/>
TLDR: Previously, PHPUnit enabled all caches except WANObjectCache. This is now enabled as well. If you encounter a new test failure this week on an extension repo that is not covered by the automated "wmf CI gate", then that may be the reason. This could point to a genuine bug in your code. However, you can choose to turn of all caching layers in the relevant test cases by calling $this->setMainCache( CACHE_NONE ).
Hi,
MediaWiki provides broudly four layers of object caching <https://www.mediawiki.org/wiki/Object_cache>, that are individually configurable (to optimise large wiki farms like Wikipedia). Each of these auto-configures themselves or falls back to $wgMainCacheType <https://doc.wikimedia.org/mediawiki-core/master/php/classMediaWiki_1_1MainC…>, thus in practice you don't need to be aware of their differing backends. They have a simple unified interface: BagOStuff.
In our PHPUnit context, we set each layer to 'hash' (for HashBagOStuff) which represents an in-memory cache that we automatically reset between each test and each data set. In addition to the four levels of caching, this therefore also ensures realistic caching behaviour during your tests for services such as ParserCache, MessageCache, SessionStore, MainStash, etc.
For example, your extension might have a test that asserts "no database queries" made during a cache hit, or it might assert that if an edit is made the same value is returned as before, and then you call your code's purge method, and assert the updated value.
By default and during local development, (other) CI jobs, beta cluster, and production - WANObjectCache is explicitly a wrapper for the main cache (e.g. Memcached at WMF). It is never configured separately from the main cache, nor is there any known use case (among both third parties, WMF, and local development) for doing so.
As of this week, configuring WANObjectCache separately from the "main" cache is no longer supported. This improves defaults for sysadmins and reduces the number things one has to get right for optimal performance. It also removes an apparent concept from the code going forward to ease debugging by not having a level of indirection that doesn't translate to a specific narrative or purpose.
In dong so, we found one place that does configure the WANCache backend, namely our PHPUnit config. While expected to override it, it overrode it to wrap the no-op "none" EmptyBagOStuff instead instead of the in-memory "hash" HashBagOStuff. This quirky setup was surprising a times to developers and required numerous tests to copy boilerplate around in order to opt-in to to the "normal" cache behaviour that we expect and follow in other cache levels already. The boiler plate in question was also atypical in that it wasn't covered by our standard ServiceWiring and its automatic resets (the main cache is now recognised an injectable internal service).
Our shared "wmf CI gate" job identified two extensions where this was relied upon, and I've updated those before making the change <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/889244> (example 1 <https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GlobalPreferences/+/8…>, and example 2 <https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/894228>).
If you encounter a new test failure this week on an extension repo that is not covered by the CI gate, then this may be the reason. Tests that fail with caching enabled, may expose a bug in the underlying code (e.g. after a cache it should probably not use the cache for subsequent reads of the same data, unless it is expected that it will be stale), or may reflect that the code (or the test) is missing a call to your "purge" method, or that a service reset is needed if the test is meant to reflect what a subsequent web request would see, etc. You can (temporarily or not) choose to turn of all caching layers in the relevant test cases by calling $this->setMainCache( CACHE_NONE ). Example 1-line patch <https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GlobalPreferences/+/8…>.
--
Timo Tijhof
Performance Team,
Wikimedia Foundation.
Hello everyone,
TL;DR Toolhub will have a few hours of downtime due to maintenance on
Tuesday 2023-03-07 Furthermore, if you are not deploying services to the
eqiad wikikube kubernetes
cluster, you can safely skip the rest.
Long version:
We will reinitialize the eqiad wikikube kubernetes cluster using kubernetes
version 1.23 on 2023-03-07 09:00-16:00 UTC [1] (the actual process is
expected
take a couple of hours within this window).
The date was chosen for convenience as due to the data center switchover
process, eqiad is fully depooled, receiving almost 0 traffic. This is
scheduled to change on 2023-03-08, making the process more difficult. As
all traffic
has been drained already and we expect no visible impact. However, for the
duration of the process, the kubernetes cluster will be unavailable to
deployers and thus efforts to deploy to it will fail or worse, not have the
expected outcomes.
This is normal until SRE serviceops announces that the cluster is fully
operational again.
SRE serviceops will be deploying all services before marking the cluster as
usable so there will be no need for deployers to
re-deploy their services (apart from those already informed).
Toolhub, per https://phabricator.wikimedia.org/T329319 wasn't switched over
to codfw and is still being served from wikikube eqiad. Unavoidably, it
will suffer a small downtime of a few hours. That is known and expected. To
minimize that downtime, it will be prioritized during the initialization
phase.
[1] https://phabricator.wikimedia.org/T331126
--
Alexandros Kosiaris
Principal Site Reliability Engineer
Wikimedia Foundation
This email is a summary of the Wikimedia production deployment of
1.40.0-wmf.25
- Conductor: Jaime Nuche
- Backup Conductor: Antoine Musso
- Blocker Task: T325588 <https://phabricator.wikimedia.org/T325588>
- Current Status <https://versions.toolforge.org>
📊 Numbers
Sparklines comparing with the last 5 trains.
- 298 Patches ▇█▅▁█
- 0 Rollbacks ████▁
- 0 Days of delay ▁█▁▁▁
- 1 Blockers ███▁▄
🌈 Traintastic Folks 😻
Thanks to the SRE team for their amazing work during this train, and
especially to:
- *Clément Goubert*: for handling a smooth DC switchover, no small feat
- *Giuseppe Lavagetto, Amir Sarabadani, Jaime Crespo & Matthew Vernon*:
for all their critical work to troubleshoot a puzzling issue with image
thumbnails https://phabricator.wikimedia.org/T330942
--
Jaime Nuche
Software Engineer III
Wikimedia Foundation <https://wikimediafoundation.org/>
TLDR: Invoking maintenance scripts directly will be deprecated in MW 1.40, use
maintenance/run.php instead. This affects anyone managing a MediaWiki
installation, for development, testing, or production use.
Until now, MediaWiki maintenance scripts have been handled standalonePHP scripts
- for instance, to run the script that outputs the MediaWiki version, you would use:
> php maintenance/version.php
Starting with MediaWiki 1.40, this is deprecated. The preferred way to run
maintenance scripts is now by name, using the maintenance runner:
> php maintenance/run.php version
Similarly, the preferred way to run the updater is now:
> php maintenance/run.php update
The script to run cal also be specified using the full path of the script file,
or the full PHP class name of a subclass of the Maintenance class. For more
details, run
> php maintenance/run.php --help
Rationale and History:
Treating maintenance scripts as standalone PHP scripts requires some boilerplate
code to be present at the top and at the bottom of every file. This is error
prone and makes it difficult to update the maintenance framework. But more
importantly,
for this boilerplate to work, the location of the MediaWiki installation has to
be known relative to the maintenance script, which is not reliably possible for
scripts defined in extensions.
A similar problem arises if the maintenance script needs a base class other than
the default Maintenance class: since the class is loaded before MediaWiki is
initialized, the autoloader is not yet in place, and the file containing the
base class needs to be included explicitly.
These and similar issues can be avoided by creating a wrapper script that loads
and executes the actual maintenance class. This way, the maintenance wrapper can
initialize MediaWiki before passing control to the script.
I propose creating such a wrapper as an RFC in 2018 (T99268)[^1], which was
approved in 2019. However, implementing the proposal proved challenging, and
soon stalled. I picked it up again as a side project after working on
overhauling the configuration and bootstrapping code in early 2022: With the
introduction of SettingsBuilder, it became much simpler to create a
MaintenanceRunner class, because it was no longer necessary to juggle global
variables.
Several bits and pieces got reviewed and merged over the course of 2022 (shout
out to Amir, Tim, Timo, and everyone who contributed). Now the runner is ready,
and we should stop calling maintenance scripts directly.
For now, existing maintenance scripts will function both ways[^2] : when called
using the runner, or directly. However, newly created maintenance scripts should
not be required to be callable as standalone scripts. So it's best to change all
callers to use the wrapper.
This should now work for nearly all[^2] cases, though there are still a couple
of rough edges to be smoothed out. If you are running MediaWiki 1.40, please try
the new mechanism, and report any isses on Phabricator.
Thanks,
Daniel
[^1] https://phabricator.wikimedia.org/T99268
[^2] with the exception over very old-school scripts that do not use the
Maintenance base class and rely on CommandLineInc.php instead.
--
Daniel Kinzler
Principal Software Engineer, Platform Engineering
Wikimedia Foundation
One week from today, or sometime soon after, the repos/releng/release
default branch will be renamed from “master” to “main” [0][1].
We will send notification when we complete the renaming.
If you use this repository, please make sure to change branches after the
renaming has been done [2].
Release Engineering will make any necessary changes to support this rename,
but if you have concerns about something that might be affected by this
change or overlooked, please do notify us.
Phab task for reference: https://phabricator.wikimedia.org/T329770
Thank you for your vigilance!
[0]: <https://phabricator.wikimedia.org/T281593>
[1]: <https://phabricator.wikimedia.org/T254646>
[2]: Instructions: <
https://www.mediawiki.org/wiki/GitLab/Workflows#Rename_%22master%22_to_%22m…
>
--
Jeena Huneidi
Software Engineer, Release Engineering
Wikimedia Foundation
The fresh-node16 command receives a minor Node.js update, and major upgrades to both Firefox and Chromium.
Get started by updating, installing, or learning more, at:
https://gerrit.wikimedia.org/g/fresh
Changelog:
https://gerrit.wikimedia.org/g/fresh/+/22.11.2/CHANGELOG.md
The same updates were applied to WMF CI last week. Thanks to Kosta Harlan for making this release!
Fresh is a fast way to create isolated environments from your terminal. These can be used to work more responsibly with 'npm' developer tools such as ESLint, QUnit, Grunt, Selenium, and more. Example guide: https://www.mediawiki.org/wiki/Manual:JavaScript_unit_testing. To report issues or browse past and current tasks, check Phabricator at https://phabricator.wikimedia.org/tag/fresh/.
--
Timo Tijhof,
Performance Team,
Wikimedia Foundation.
Hello,
I will upgrade our Gerrit instances from 3.5.4 to 3.5.5. I have
scheduled it for Monday March 6th at 9:00 UTC after the backport and
config window.
The service will be unavailable for a few minutes while the upgrade occurs.
Release notes: https://www.gerritcodereview.com/3.5.html#355
Our task: https://phabricator.wikimedia.org/T330663
Antoine "hashar" Musso