[Engineering] Gerrit downtime this Monday!

Mon Jul 25 05:45:59 UTC 2016

Hi!

So things are back up. Took a tad longer than expected.

Big caveat: something is up with the Phabricator bridge and it's not logging
what's up. I'll look at it first thing after I've gotten a few hours of
sleep :)

Other than that, everything seems to be going great. Zuul is getting along
just
fine with Gerrit so changes are getting their tests run. IRC bot is back
too.

Your SSH connections to Gerrit *might* complain about the host verification.
While the host key has remained the same, the IP address shifted. You can
either delete or update the offending line in your ~/.ssh/known_hosts if
your
client does have trouble with it.

Thanks again to Daniel, Antoine and Jaime for their help today.

-Chad

On Sun, Jul 24, 2016 at 4:02 PM, Chad Horohoe <chorohoe at wikimedia.org>
wrote:

> Friendly reminder that this is happening in about 2 hours!
>
> -Chad
>
>
> On Thu, Jul 21, 2016 at 8:08 AM, Chad Horohoe <chorohoe at wikimedia.org>
> wrote:
>
>> Hi,
>>
>> So the time is upon us to finally upgrade Gerrit. I thank everyone in
>> advance for all of your patience
>> and good work testing things out for us. The plan is outlined in detail
>> on Phabricator[0], but I'll give
>> everyone the short version here.
>>
>> The downtime will be on Monday, July 25th from 01:00 to 04:00 UTC; that's
>> Sunday night for those
>> of us who are US-based. This time was picked based because it's one of
>> our lowest traffic times
>> on Gerrit. There's never a *good* time to bring it down, it's always
>> being used, but this looks like it'll
>> impact the fewest number of users. I do not anticipate the process
>> actually taking the full 3 hours,
>> but I'm giving us a lot of extra time just in case.
>>
>> Jaime will be on hand to assist with a final DB snapshot of the old
>> version and possible rollback,
>> and Daniel Z. is going to assist me with the puppet work. Most of this is
>> already prepared so the
>> amount of "change" to do the swap has been kept to a minimum.
>>
>> Gerrit is a critical service to all developers, so the plan includes a
>> generous provision to roll back
>> if things are not operating properly--I'd rather us be on the old version
>> that works on a Monday
>> morning than be stuck broken going into the work week.
>>
>> I'll be sure to send a last minute reminder Sunday night prior to taking
>> services offline.
>>
>> Thanks again!
>>
>> -Chad
>>
>> [0] https://phabricator.wikimedia.org/T70271#2482308
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/engineering/attachments/20160724/6beaad44/attachment.html>