[Labs-l] LDAP Search failed on new instances

Sun Jul 29 19:15:59 UTC 2012

Yep. That was an issue. I restarted the ldap server, and things came
back up. I need to investigate why the clients aren't properly failing
over to the secondary in this situation.

On Sat, Jul 28, 2012 at 10:27 AM, aude <aude.wiki at gmail.com> wrote:
> Just to follow up, there's obvious issues being reported in
> #wikimedia-operations:
>
> 17:22 < aude> 15:15 <+nagios-wm> PROBLEM - LDAP on virt0 is CRITICAL:
> Connection refused
> 17:22 < aude> from #wikimedia-operations
> 17:22 < aude> 15:59 <+nagios-wm> PROBLEM - Puppetmaster HTTPS on stafford is
> CRITICAL: CRITICAL - Socket timeout after 10 seconds
> 17:22 < aude> 16:12 <+nagios-wm> RECOVERY - Puppetmaster HTTPS on stafford
> is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.002 seconds
>
> Cheers,
> Katie
>
>
> On Sat, Jul 28, 2012 at 7:19 PM, aude <aude.wiki at gmail.com> wrote:
>>
>> I am getting failures in creating new instances on labs today.
>>
>>
>> Jul 28 16:54:24 i-00000371 puppet-agent[3950]: Starting Puppet client
>> version 2.7.11
>> Jul 28 16:54:25 i-00000371 puppet-agent[3950]: Could not retrieve catalog
>> from remote server: Error 400 on SERVER: Failed when searching for node
>> i-00000371.pmtpa.wmflabs: LDAP Search failed
>> Jul 28 16:54:25 i-00000371 puppet-agent[3950]: Using cached catalog
>> Jul 28 16:54:25 i-00000371 puppet-agent[3950]: Could not retrieve catalog;
>> skipping run
>> Jul 28 16:55:06 i-00000371 dhclient: DHCPREQUEST of 10.4.0.229 on eth0 to
>> 10.4.0.1 port 67
>> Jul 28 16:55:06 i-00000371 dhclient: DHCPACK of 10.4.0.229 from 10.4.0.1
>> Jul 28 16:55:06 i-00000371 dhclient: bound to 10.4.0.229 -- renewal in 53
>> seconds.
>>
>>
>> dhclient repeatedly retries and fails
>>
>>
>> https://labsconsole.wikimedia.org/w/index.php?title=Special:NovaInstance&action=consoleoutput&project=maps&instanceid=i-00000371
>>
>> Any ideas what the problem is? probably nothing I can do but wait for Ryan
>> or someone?
>>
>> Cheers,
>> Katie
>>
>> --
>> Board member, Wikimedia District of Columbia
>> http://wikimediadc.org
>> @wikimediadc / @wikimania2012
>>
>
>
>
> --
> Board member, Wikimedia District of Columbia
> http://wikimediadc.org
> @wikimediadc / @wikimania2012
>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>