[Labs-l] Labs datacenter migration

Wed Nov 20 16:01:41 UTC 2013

Can you please elaborate more on what we need to do in case we have
instances with selfhosted puppet master?

On Fri, Nov 15, 2013 at 8:29 PM, Andrew Bogott <abogott at wikimedia.org> wrote:
>     Almost a year ago, the Wikimedia Foundation migrated most of our
> services from our old data center in Tampa to the new one in Ashburn [1].
> In the next couple of months Labs and Tool Labs will be following suit -- we
> expect to have everything moved to Ashburn by mid-January at the latest.
>
>     This move will provide some immediate benefits (lower latency with
> production, quicker database replication) and many long-term benefits
> (better stability, happier Operations staff).  We don't yet have a specific
> timeline for stages of the migration, but there are a few things you can do
> now to help us prepare for the change and to bolster your projects against
> possible disruption.
>
> 1)  Subscribe to Labs-l, and read it. [2]  Labs-l is low-volume, and future
> migration announcements may not be sent to other lists.
>
> 2)  Tool Labs users:  As long as your tools are properly managed by the grid
> engine and can survive stops and restarts, the migration will be quite
> painless.  If your tools aren't, or can't... fix them :)
>
> 3)  Labs project admins:  Clean up old projects and instances.  If you have
> instances that are no longer of interest, delete them.  If you know of
> entire projects that are no longer in use, please contact me directly and
> I'll mop up.
>
> 4)  Labs instance owners:  Make sure that puppet is running properly on your
> instances.  If '$sudo puppetd -tv' produces any red lines, then fix them or
> contact me for help with fixing.  When instances move to the new data center
> we'll be relying on puppet to update location-specific settings, so
> instances without puppet may not survive the move.  If your instance uses
> self-hosting puppet (via puppetmaster::self or role::puppet::self) then you
> will also need to update your local puppet repo. [3]
>
> 5)  All Labs users: if you have valuable data residing on local instance
> storage, start backing it up to shared storage in /data/project.  You should
> be doing this anyway -- no instance is safe from catastrophe, ever.
>
> 6)  If your project or tool generates log files, have a look at purging old
> log data.  The last time we did a data migration there was at least one
> terabyte-sized logfile that really gummed up the works.
>
>
>     Updates about this change will be posted to this list as soon as we know
> about them.  Any potential downtime will be announced well in advance.  In
> the meantime, don't hesitate to ask questions about the above steps on IRC
> or the mailing list.
>
> -Andrew
>
>
> [1]
> https://blog.wikimedia.org/2013/01/19/wikimedia-sites-move-to-primary-data-center-in-ashburn-virginia/
>
> [2] https://lists.wikimedia.org/mailman/listinfo/labs-l
>
> [3] https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster#FAQ
>
>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l