[Wikitech-l] Fwd: [Ops] [Gerrit] Emit alert when Ori commits on a weekend - change (operations/puppet)

Thanks for all your outstanding work, Ori! This one was so fantastic I had to forward a bit... at least you can admit you have a problem :p -Adam ---------- Forwarded message ---------- From: Ori.livneh (Code Review) <gerrit(a)wikimedia.org> Date: Thu, Dec 4, 2014 at 4:55 AM Subject: [Ops] [Gerrit] Emit alert when Ori commits on a weekend - change (operations/puppet) To: Ori.livneh has uploaded a new change for review. https://gerrit.wikimedia.org/r/177521 Change subject: Emit alert when Ori commits on a weekend ...................................................................... Emit alert when Ori commits on a weekend Provisions an Icinga check on palladium that issues an alert if the time is between Friday 21:00 and Monday 01:00 and there exists a commit from me in the last hour in operations/puppet. Change-Id: I2c4fd0d6ef907e4afe91f0248e7039f31d70696c --- A files/icinga/check-ori-commits M manifests/misc/monitoring.pp M manifests/site.pp 3 files changed, 37 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/21/177521/1 diff --git a/files/icinga/check-ori-commits b/files/icinga/check-ori-commits new file mode 100755 index 0000000..d2d9552 --- /dev/null +++ b/files/icinga/check-ori-commits @@ -0,0 +1,20 @@ +#!/bin/bash +# Icinga alert script for Ori weekend commits +# +# Alerts if the time is between 21:00 on Friday and 01:00 on Monday +# (my time zone) and there exists a commit from me in the last hour. + +TZ="America/Los_Angeles" + /usr/bin/git \ + --git-dir=/var/lib/git/operations/puppet/.git log \ + --author=ori(a)wikimedia.org \ + --since=1hour \ + --format=%cd | /bin/grep -Pq '(Fri .* 2.:|Sat|Sun)' + +if [ $? -eq 0 ]; then + echo "CRITICAL: Ori committed a change on a weekend" + exit 2 +else + echo "OK: Ori is behaving himself" + exit 0 +fi diff --git a/manifests/misc/monitoring.pp b/manifests/misc/monitoring.pp index b333799..baf49ad 100644 --- a/manifests/misc/monitoring.pp +++ b/manifests/misc/monitoring.pp @@ -613,3 +613,19 @@ ], } } + + +class misc::monitoring::ori_weekend_commits { + file { '/usr/local/lib/nagios/plugins/check-ori-weekend-commits': + source => 'puppet:///files/icinga/check-ori-weekend-commits', + owner => 'root', + group => 'root', + mode => '0555', + } + + nrpe::monitor_service { 'ori_weekend_commits': + description => 'Ori committing changes on the weekend', + nrpe_command => '/usr/local/lib/nagios/plugins/check-ori-weekend-commits', + require => File['/usr/local/lib/nagios/plugins/check-ori-weekend-commits'], + } +} diff --git a/manifests/site.pp b/manifests/site.pp index 688fa18..f3bf770 100644 --- a/manifests/site.pp +++ b/manifests/site.pp @@ -2162,6 +2162,7 @@ include role::access_new_install include role::puppetmaster::frontend include role::pybal_config + include misc::monitoring::ori_weekend_commits $domain_search = [ 'wikimedia.org', -- To view, visit https://gerrit.wikimedia.org/r/177521 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2c4fd0d6ef907e4afe91f0248e7039f31d70696c Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Ori.livneh <ori(a)wikimedia.org> _______________________________________________ Ops mailing list Ops(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops