[Labs-l] DNS outage for labs

Fri Jun 15 19:46:36 UTC 2012

On Fri, Jun 15, 2012 at 11:12:29AM +0200, Ryan Lane wrote:
> We have an ongoing DNS outage for Labs. It's being ongoing for the
> past day. We've been troubleshooting the issue and hope to have it
> resolved soon. Sorry for the inconvenience.

We found the source of the problem, which was stale glue records on the
"org" zone for virt0.wikimedia.org & labsconsole.wikimedia.org. We are
fixing this by updating the nameservers in our domain registrar. In the
meantime also added the old IP to the machine as to mitigate the problem
until the TTLs expire (a day).

We are going to work on improving resiliency by adding a proper second
nameserver in the eqiad datacenter soon.

Apologies for the troubles!

Regards,
Faidon