[Engineering] Data center switch-over moving ahead next week: please stay available :)

Fri Apr 15 12:56:09 UTC 2016

Hi all,

As previously announced[1], our data center switch-over test is planned to
happen next week, on Tuesday April 19th, with the switch back two days
later, Thursday April 21st. It's looking good, and unless any major new
obstacles arise, we'll be moving forward with it.

A request:

The Technology team would highly appreciate it if everyone in Engineering
with knowledge of or responsible for any software/service/extension running
in production, could keep an eye on things during these 3 days, and also
stay reachable by phone just in case of need.

With such a large migration with lots of components it's always possible
that we find unanticipated issues during or after the fail-overs. Certain
features/services could fail unexpectedly, sometimes subtly so, e.g. due to
unforeseen traffic patterns from configuration mistakes, ACL/permission
mismatches, etc etc. We'll certainly try to correct these issues, but in
some cases we may need or benefit from your support/knowledge/patches, and
may want to reach you. Because we can't detect all issues immediately, it
would also be helpful if you'd keep an eye out for any site issues and
report any regressions.

If we need to reach you urgently by phone, we'll typically do so using the
phone number provided on Office Wiki's Contact List[2]. Be aware that some
phone numbers in this list have been corrupted in the past during automated
edits, or may be outdated. Therefore, please check if your phone number
listed there is still correct.

The actual switch-overs begin on Tuesday, 19 April at 14:00 UTC and
Thursday, 21 April at 14:00 UTC, respectively. Any changes to this schedule
will be noted on our Wikitech calendar[3].

To report any issues, please use one of the following channels:

1. File a Phabricator issue with project #codfw-rollout
2. Report issues on IRC: Freenode channel #wikimedia-tech (if urgent, or
during the migration)
3. Send an e-mail to the Operations list: ops at lists.wikimedia.org (any time)

Thanks!

[1] http://blog.wikimedia.org/2016/04/11/wikimedia-failover-test/
[2] https://office.wikimedia.org/wiki/Contact_list
[3]
https://wikitech.wikimedia.org/wiki/Switch_Datacenter#Schedule_for_Q3_FY2015-2016_rollout

-- 
Mark Bergsma <mark at wikimedia.org>
Lead Operations Architect
Director of Technical Operations
Wikimedia Foundation
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/engineering/attachments/20160415/f13edcff/attachment.html>