Hi,
there was a outage today 2019-05-29 in CloudVPS/Toolforge involving keystone +
NFS. All CloudVPS projects (including Toolforge) had troubles using the
NFS-based storage due to an upgrade operation we were doing in
cloudcontrol1003.wikimedia.org.
You can read more about the incident here:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20190529-NFS-key…
The incident postmorten is not completed yet, but you can already read the main
sections:
* what happened
* timeline
* things to improve in the future
regards.
--
Arturo Borrero Gonzalez
Operations Engineer / Wikimedia Cloud Services
Wikimedia Foundation