cloud-vps maintenance at 14:00 UTC - Cloud-announce

Andrew Bogott

27 Apr 27 Apr

8:03 p.m.

Reminder: this is starting shortly. On 4/27/21 12:48 AM, Andrew Bogott wrote:

We will be upgrading the Cloud-VPS OpenStack install later today beginning at 14:00 UTC (7:00 AM Pacific time). The total upgrade should take 60-90 minutes. During the upgrade period Horizon will be disabled. There may also be brief network interruptions as we restart router services. -Andrew + the WMCS Team

Reply

Brooke Storm

9:36 p.m.

New subject: [Cloud] cloud-vps maintenance at 14:00 UTC

Toolforge NFS services are not accessible to the VMs. Something went wrong during the upgrade and the team is working to restore services as quickly as possible. I have shut off the bastions to protect their local disks. Brooke Storm Staff SRE Wikimedia Cloud Services bstorm(a)wikimedia.org

...

On Apr 27, 2021, at 8:34 AM, Roy Smith <roy(a)panix.com> wrote: I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/ <https://spi-tools.toolforge.org/> and https://spi-tools-dev.toolforge.org/ <https://spi-tools-dev.toolforge.org/>. Also: ssh: connect to host dev.tools.wmflabs.org <http://dev.tools.wmflabs.org/> port 22: Network is unreachable Could the VPS work be causing this?

On Apr 27, 2021, at 10:03 AM, Andrew Bogott <abogott(a)wikimedia.org <mailto:abogott@wikimedia.org>> wrote: Reminder: this is starting shortly. On 4/27/21 12:48 AM, Andrew Bogott wrote:

We will be upgrading the Cloud-VPS OpenStack install later today beginning at 14:00 UTC (7:00 AM Pacific time). The total upgrade should take 60-90 minutes. During the upgrade period Horizon will be disabled. There may also be brief network interruptions as we restart router services. -Andrew + the WMCS Team

_______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org <mailto:Cloud-announce@lists.wikimedia.org> (formerly labs-announce(a)lists.wikimedia.org <mailto:labs-announce@lists.wikimedia.org>) https://lists.wikimedia.org/mailman/listinfo/cloud-announce <https://lists.wikimedia.org/mailman/listinfo/cloud-announce> _______________________________________________ Wikimedia Cloud Services mailing list Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud

_______________________________________________ Wikimedia Cloud Services mailing list Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud

Reply

Andrew Bogott

10:09 p.m.

New subject: [Cloud] cloud-vps maintenance at 14:00 UTC

There's still a bit of upgrading happening but the recent network interruption should now be resolved. We're about to put Horizon back online so things should be back to normal shortly. As always, some tools may have failed to recover from the network interruption so you're encouraged to check on the state of things. Sorry for the interruption! -Andrew On 4/27/21 10:37 AM, David Caro wrote:

...

Yep, it's related to the upgrade, working on it, thanks for your patience! Will update shortly. On 04/27 11:34, Roy Smith wrote:

I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/ <https://spi-tools.toolforge.org/> and https://spi-tools-dev.toolforge.org/ <https://spi-tools-dev.toolforge.org/>. Also: ssh: connect to host dev.tools.wmflabs.org port 22: Network is unreachable Could the VPS work be causing this?

On Apr 27, 2021, at 10:03 AM, Andrew Bogott <abogott(a)wikimedia.org> wrote: Reminder: this is starting shortly. On 4/27/21 12:48 AM, Andrew Bogott wrote:

We will be upgrading the Cloud-VPS OpenStack install later today beginning at 14:00 UTC (7:00 AM Pacific time). The total upgrade should take 60-90 minutes. During the upgrade period Horizon will be disabled. There may also be brief network interruptions as we restart router services. -Andrew + the WMCS Team

_______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud

_______________________________________________ Wikimedia Cloud Services mailing list Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud

Reply