[Labs-l] labsdb1002 scheduled downtime maintenance (was: labsdb1002 possible issues)

Thu Jun 11 06:27:57 UTC 2015

Maintenance update:

We have performed the desired maintenance, including an upgrade, reboot and
filesystem check, and MariaDB is available again for connection. We believe
to have fixed the origin of the problems.

However, as the issues had caused some database integrity problems, we are
refreshing and checking the production wikis data. Until further notice,
later this week, I would recommend you to continue using labsdb1001 and
labsdb1003 or you will run the risk of working with stale data.

The user's own databases are available and are not affected by this.

Thank you for your understanding,

On Mon, Jun 8, 2015 at 4:46 PM, Yuvi Panda <yuvipanda at gmail.com> wrote:

> We will redirect *.labsdb to point to another database before the
> beginning of the window. However this might still affect you if you
> have a user database on labsdb1002.
>
> On Mon, Jun 8, 2015 at 2:38 PM, Jaime Crespo <jcrespo at wikimedia.org>
> wrote:
> > As a follow up, labsdb1002 has been scheduled for downtime maintenance,
> > starting on 2015-06-10 07:00 UTC and will be unavailable until further
> > notice.
> >
> > While we expect that the disruption will take only 1 day or less, we are
> > unsure if we will found more complicated issues during the outage that
> would
> > force us to prolong the maintenance.
> >
> > I would recommend to switch your queries before that date temporarily to
> > lasbdb1001 or labsdb1003, which will not be affected by the maintenance.
> I
> > will send here again what is the status when the works are finished.
> >
> > Regards,
> >
> > On Sun, Jun 7, 2015 at 9:49 AM, Sean Pringle <springle at wikimedia.org>
> wrote:
> >>
> >> https://phabricator.wikimedia.org/T101567
> >>
> >> labsdb1002 had a conniption, possibly related to XFS, and causing some
> >> unsettling DB replication glitches. Full investigation and (probable)
> >> resync yet to be done.
> >>
> >> So, heads up. If you spot anything odd, best to switch to one of the
> >> other replicas: lasbdb1001 or labsdb1003
> >>
> >> -s
> >>
> >> _______________________________________________
> >> Labs-l mailing list
> >> Labs-l at lists.wikimedia.org
> >> https://lists.wikimedia.org/mailman/listinfo/labs-l
> >
> >
> >
> >
> > --
> > Jaime Crespo
> > <http://wikimedia.org>
> >
> > _______________________________________________
> > Labs-l mailing list
> > Labs-l at lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/labs-l
> >
>
>
>
> --
> Yuvi Panda T
> http://yuvi.in/blog
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>

-- 
Jaime Crespo
<http://wikimedia.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20150611/0c60bd57/attachment.html>