[Labs-l] some questions on "puppet failed on ..."

Wed Jun 8 15:17:26 UTC 2016

On Wed, Jun 8, 2016 at 6:16 AM, Malte Reissig <mre at deepamehta.de> wrote:
>
> Hi there,
>
> i wanted to ask some questions to better understand how to adapt failing
> puppet configuration for my wikimedia labs instance, as i started to get
> notifications 2 days ago.

Great questions Malte.

The Puppet failures may not actually be a new thing. For you and
others, these emails may have started being delivered a couple of days
ago when some issues with outbound email from Labs hosts were
resolved. That being said it is important to keep Puppet running on
your Labs instances so that configuration changes needed for software
upgrades and infrastructure changes are applied.

> So, i found out about "Manage Puppet Groups" (in Wikitech) but i could not
> figure out what the page is exactly about (listing "all projects" ->
> "Classes"?), except that i can add so called "puppet groups" myself..

Most projects don't need to worry about changing the "Manage Puppet
Groups" page in wikitech. That page allows a project to manage the
options that are presented for configuring Puppet on each individual
instance. It's basically a form editor.

> But before trying so, i wanted to understand a bit more, specifically:
>
> - What does the "role::puppet::self" mean? when set/not-set for my instance?

This Puppet role is used both to setup a "self-hosted Puppet master"
for a project and to attach an instance to an existing self-hosted
Puppet master. This will not be enabled for a typical project. See
<https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster>
for more information.

> - What does the global group named "puppet-development" stand for? (looking
> at "configure" of my specific instance in question, this group is not
> selected)

This is the container for the "role::puppet::self" option discussed above.

> - Where can i see/find my instances puppet configuration

You can get to the Puppet configuration management screen for an
instance via "Manage Instances"
(<https://wikitech.wikimedia.org/wiki/Special:NovaInstance>). Choose
your project in the "Project filter" section at the top and click "Set
filter". You should see a table of instances below. The far right
column of this table will have a "configure" link for each instance
that leads to the Puppet configuration form for that host.

If there is no table of instances and you know that there are
instances in the project, try logging out of wikitech and logging back
in. The authentication session to the OpenStack backend sometimes gets
out of sync with the wikitech session.

> - How can i adjust my instance puppet file and (test-)run puppet until it is
> not "failing to run" but "working"
> - Where can i see/find the (error) output of puppet failing

The easiest way to debug failing Puppet runs is to ssh into the
instance and run Puppet manually:
  $ ssh my-cool-instance.eqiad.wmflabs
  $ sudo -i puppet agent --test --verbose

This should run Puppet and display detailed logs of what is going on.
For resolving particular problems you can ask questions on the
#wikimedia-labs freenode irc channel, open Phabricator tasks with
detailed log information, or ask on this mailing list.

If your instance is running Jessie, a common error that popped up a
few weeks ago is a failure to start the exim service. This is caused
by a mismatch between the configuration that Puppet provides and the
version of exim installed. Upgrading packages on the instance to the
latest versions from apt (`sudo apt-get update; sudo apt-get
dist-upgrade`) is one way to fix this. Individual packages can be
upgraded as well if you are wary of a system-wide update.

> Might it be best for me to migrate my app to a new (debian jessy based)
> instance, would that make sense (i mean, has anything changed in the
> provisioning of puppet-defaults for new instances from 02 March 2015 to 07
> June in 2016)?

You certainly can provision a new instance if that seems easiest.
Depending on how long Puppet has been broken that may in the long run
end up being the easiest option.

> OK, so much for now & Thanks for your support!

Thanks for reaching out.

Bryan
-- 
Bryan Davis              Wikimedia Foundation    <bd808 at wikimedia.org>
[[m:User:BDavis_(WMF)]]  Sr Software Engineer            Boise, ID USA
irc: bd808                                        v:415.839.6885 x6855