[Labs-admin] Labsdb migration (was: Report from ops meeting)

Tue Jan 31 20:58:14 UTC 2017

Ok, too.

On Tue, Jan 31, 2017 at 9:52 PM, Yuvi Panda <yuvipanda at gmail.com> wrote:

> Actually I've just been told that Feb 14 is valentine's day and I
> might be tasked with other duties on that day. Sorry! Feb 15?
>
> On Mon, Jan 30, 2017 at 10:57 AM, Jaime Crespo <jcrespo at wikimedia.org>
> wrote:
> > Ok to me.
> >
> > On Mon, Jan 30, 2017 at 7:54 PM, Yuvi Panda <yuvipanda at gmail.com> wrote:
> >>
> >> How about Feb 14? That gives us two weeks.
> >>
> >> On Mon, Jan 30, 2017 at 10:33 AM, Jaime Crespo <jcrespo at wikimedia.org>
> >> wrote:
> >> > As an admin, everything you should know about the upcoming labsdb1005
> >> > reimage:
> >> >
> >> > * We are ready (DBAs) to do this at any time, we just need to tell
> users
> >> > in
> >> > advance of potential outages/degradations of service
> >> > * For 99% of the users, we will just switchover them transparently to
> >> > the
> >> > slave (should not cause issues). As usual, if their application does
> not
> >> > retry to reconnect, there will be problems.
> >> > * For 3 users (databases), there will be full outage because they have
> >> > such
> >> > a heavy usage that we cannot replicate them in real time. They were
> made
> >> > aware of this limitation months ago, so it should not come as a
> >> > surprise:
> >> > https://phabricator.wikimedia.org/T127164 The users's databases are
> >> > documented at:
> >> >
> >> > https://phabricator.wikimedia.org/diffusion/OPUP/browse/
> production/templates/mariadb/tools.my.cnf.erb;
> f21ce599fe626e7c96010a5d0335370ebe510ca7$65
> >> > * Data will be copied away, server will be reimaged, then data will be
> >> > copied back That normally takes 3 hours, but things could go wrong...
> >> > * People could complain for a 10.0 upgrade (?). But some people
> actually
> >> > complained already for the lack of 5.5 -> 10 upgrade.
> >> > https://phabricator.wikimedia.org/T138517#2796682
> >> > * On switch-back, again bad-programmed application may temporarily
> fail,
> >> > but
> >> > good ones should just switch transparently; unavailable dbs should be
> >> > available again
> >> >
> >> > That should be enough background to schedule and send an email to
> users
> >> > :-)
> >> >
> >> > ---------- Forwarded message ----------
> >> > From: Yuvi Panda <yuvipanda at gmail.com>
> >> > Date: Mon, Jan 30, 2017 at 7:14 PM
> >> > Subject: [Labs-admin] Report from ops meeting
> >> > To: Labs admin list for infrastructure and discussion
> >> > <labs-admin at lists.wikimedia.org>
> >> >
> >> >
> >> > 1. Faidon talking about ip space discussions wrt asia dc discussion,
> >> > and mentioned we might / should renumber labs IP space. Not sure about
> >> > more details.
> >> > 2. Ping on labsdb migration to Jessie
> >> > 3. Mid-year review of annual goals coming up, need status about OGE
> >> > migration
> >> >
> >> > That's it.
> >> >
> >> > --
> >> > Yuvi Panda T
> >> > http://yuvi.in/blog
> >> >
> >> > _______________________________________________
> >> > Labs-admin mailing list
> >> > Labs-admin at lists.wikimedia.org
> >> > https://lists.wikimedia.org/mailman/listinfo/labs-admin
> >> >
> >> >
> >> >
> >> > --
> >> > Jaime Crespo
> >> > <http://wikimedia.org>
> >> >
> >> > _______________________________________________
> >> > Labs-admin mailing list
> >> > Labs-admin at lists.wikimedia.org
> >> > https://lists.wikimedia.org/mailman/listinfo/labs-admin
> >> >
> >>
> >>
> >>
> >> --
> >> Yuvi Panda T
> >> http://yuvi.in/blog
> >>
> >> _______________________________________________
> >> Labs-admin mailing list
> >> Labs-admin at lists.wikimedia.org
> >> https://lists.wikimedia.org/mailman/listinfo/labs-admin
> >
> >
> >
> >
> > --
> > Jaime Crespo
> > <http://wikimedia.org>
>
>
>
> --
> Yuvi Panda T
> http://yuvi.in/blog
>

-- 
Jaime Crespo
<http://wikimedia.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-admin/attachments/20170131/384dd4d5/attachment.html>