[Labs-l] Update on DNS outage from last week

Tue Jun 19 10:37:43 UTC 2012

To ensure we don't have another outage from DNS, like last week, we've
taken the following steps:

1. Added an additional nameserver in eqiad (which also means an LDAP
replica for labs as well)
2. Changed the NS records to labs-ns0.wikimedia.org and labs-ns1.wikimedia.org
3. Made the A records for labs-ns0/1 service IPs, so that we can move
the service between hosts without interruption

Additionally, we found out that we can't simply change the NS records
with markmonitor, since they add glue records and the changing of the
glue records is a manual process on their side. We'll anticipate
problems like this had this this time in the future and plan
accordingly.

- Ryan