[Labs-l] Outage report report - DNS (due to puppet removing our temporary fix)

Mon Jun 18 09:06:05 UTC 2012

We made a temporary fix last week to make the DNS resolver listen on
both the old and new address. Puppet manages the powerdns
configuration, so we stopped puppet and reconfigured powerdns. We
forgot to disable the cron that occasionally restarts the puppet
process, though. When puppet restarted, it reconfigured powerdns and
restarted it, removing the old address from the resolver. I've stopped
puppet and removed the cron, restoring resolver access.

We'll be adding a secondary resolver in eqiad today, and will again
attempt to address the DNS issue we're currently having.

- Ryan