Hello,
today 2023-09-20 we will conduct a maintenance operation on Cloud VPS. The
operation involves moving the openstack API endpoint to a new set of hardware
servers. No action is required from your side.
The Cloud VPS control plane, including Horizon, may be intermittently
unavailable during the operations. Virtual machines should keep running
unaffected, and the same for Toolforge tools.
See also: https://phabricator.wikimedia.org/T346439
regards.
--
Arturo Borrero Gonzalez
Senior SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi there,
We recently did some DNS maintenance operations [0]. As part of the maintenance,
we moved the DNS recursor server IP address to be 172.20.255.1.
However, we have detected a number of virtual machines with broken puppet, that
did not pick up the change.
This is traditionally done via the file /etc/resolv.conf with a line like:
=== 8< ===
nameserver 172.20.255.1
=== 8< ===
Please fix the virtual machines to use the new resolver, or if the machine is
not in use, consider shutdown (or delete) it. Feel free to ask for assistance if
required [1].
After some period of time, the old resolver addresses will stop working and
unmaintained virtual machines will become even more broken.
thanks, regards.
[0]
https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.…
[1]
https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_introduction#Commun…
--
Arturo Borrero Gonzalez
Senior SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi,
today 2023-09-11 we will be conducting some internal Cloud VPS DNS service
operations:
* change the DNS recursor of every virtual machine running Cloud VPS from
208.80.154.143 and 208.80.154.24 to 172.20.255.1 (this is traditionally done via
/etc/resolv.conf)
* change the real server behind the authorizative DNS
ns1.openstack.eqiad1.wikimediacloud.org, including the IP address, from
208.80.154.11 to 185.15.56.163
This may affect briefly some virtual machines, but the new DNS servers have been
running for a while already and we are not anticipating a major impact (famous
last words?).
Please report any problems you may find.
Some phabricator tickets tracking this work are:
* https://phabricator.wikimedia.org/T345240 cloudservices1006: put into service
* https://phabricator.wikimedia.org/T346033 cloudservices1004: decomission
* https://phabricator.wikimedia.org/T342621 eqiad1: cloudlb: transition DNS
clients (VMs) to the new BGP-based recursor VIP
regards.
--
Arturo Borrero Gonzalez
Senior SRE / Wikimedia Cloud Services
Wikimedia Foundation
In order to get long running queries running in superset moving to the in
cluster database is needed. Further description in
https://phabricator.wikimedia.org/T340623
Unfortunately the external mysql db is not directly compatible with the
internal postgres db. Described in https://phabricator.wikimedia.org/T343526
As such the database move will drop the existing data. However for anything
that is of interest, the old instance will be available until 2023-10-16 at:
https://old-superset.wmcloud.org/
From there you should be able to login, go to the section with the desired
object (Dashboards, charts, datasets, saved queries), mark "Bulk Select"
select anything that is desired, then export. On the new superset
(available at superset.wmcloud.org starting on 2023-09-11), go to the same
section and push the import button (which looks suspiciously like a
download button) and import the file that you exported from the old
superset.
--
*Vivian Rook (They/Them)*
Site Reliability Engineer
Wikimedia Foundation <https://wikimediafoundation.org/>
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
After nearly a decade of mishap and delay, we have updated the WMCS
terms of use. The updated document for toolforge and cloud-vps admins
can be found here:
https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_use
and the terms of use for visitors to WMCS sites can be found here:
https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_End_User_Terms_…
There is one significant change in these terms: Cloud-vps projects which
collect personal data will need to include an explicit privacy policy
for their projects. This is section 7.3. For other WMCS users and admins
these documents do not represent any significant change in policy, but
do clarify and finalize many things that were poorly-worded in the
previous TOU, or policies that we have enforced informally without
officially stating.
Please feel free to reach out to WMCS staff if you find any part of
these documents concerning or disruptive to your work on our platforms.
-Andrew
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Just now the toolsdb database server ran out of memory and died[0] -- it
took us a few minutes to get things rolling again but all service should
be restored.
The service first went down Aug 15 23:09:12; RO only service was
restored about 45 minutes later and RW service shortly after that.
Probably no action is needed on the part of tool maintainers, but if
your tools are upset this is a good thing to suspect. Sorry for the
downtime!
-Andrew + the WMCS team
[0] https://phabricator.wikimedia.org/T344298
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
https://phabricator.wikimedia.org/T320934 was opened last year to note that
sparql was no longer building on more recent versions of notebook. A patch
submitted to the upstream to fix the issue has not been noted or merged.
Indeed the upstream has not been updated since 2020.
https://phabricator.wikimedia.org/T121478 meanwhile has been unblocked
upstream with a new version of notebook. That sparql integration with
jupyterlab appears to lack support it will be removed to unblock notebook
upgrades.
Sparql will be removed as part of https://phabricator.wikimedia.org/T338981
which has a slightly fuzzy deployment date, but not until at least
2023-08-14
--
*Vivian Rook (They/Them)*
Site Reliability Engineer
Wikimedia Foundation <https://wikimediafoundation.org/>
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
The Foundation's Machine Learning team has announced [0] that users of
ORES models should begin migrating their code to the newer Lift Wing
system. They have prepared a page on Wikitech to help bot and tool
maintainers understand what steps will be needed and where they can go
to ask for help with the process [1].
[0]: https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/…
[1]: https://wikitech.wikimedia.org/wiki/ORES
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Hey everyone,
The Wikimania Hackathon 2023
<https://wikimania.wikimedia.org/wiki/2023:Hackathon> is coming to Singapore
! This is our first in-person Wikimania Hackathon since 2019. It will take
place August 15–19 at the Suntec Singapore Convention and Exhibition Centre.
Whether you're a newcomer or an experienced technical contributor, there's
a place for you. We have a Newcomer Track with introductory sessions and
workshops, and an Unconference Track with breakout rooms for sessions
identified during the event. A main hacking space will also be available
throughout the event for collaborative work.
Check out our Diff post
<https://diff.wikimedia.org/2023/07/24/save-the-date-the-wikimania-hackathon…>
for more details about the event, including how to prepare and participate.
Important! If you're registered for Wikimania
<https://wikimania.eventyay.com/> but *didn't opt in for Hackathon updates*,
you may miss out on specific Hackathon emails. To receive these, email
sstefanova(a)wikimedia.org, and we'll add you to the Hackathon mailing list.
If you have any questions, feel free to ask on the Wikimania help desk page
<https://wikimania.wikimedia.org/wiki/2023_talk:Help_desk>.
See you in Singapore, or online!
On behalf of the Wikimania Hackathon organizing team
*Tar Lócesilion / Szymon Grabarczuk*
Wikimania 2024 Core Organizing Team
Wikimania 2023 Technology Committee