[Labs-admin] 2017-07-11 - 2017-07-17 on-call notes

Thu Jul 20 15:06:47 UTC 2017

Tues:
* harshcro_ (GSOC) hand holding for getting started in Toolforge
** Tool is wikiemojibot
** https://phabricator.wikimedia.org/T164560
* https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#What_is_a_Tool_account.3F
is a mess
* Pinged on some project/quota request tickets for more info
* lots of phab gnome work

Wed:
* /usr/local/sbin/block_sync cron failure
** poked Madhu
* https://phabricator.wikimedia.org/T169653
* https://phabricator.wikimedia.org/T170348
* Puppet error storm hit us and prod. A patch made puppet kill itself mid run.
* Sagan/Luke081515 with project issues; called in chase, andrew
** Larger problem with instance creation
** will need an outage report when its all done
* issues with nodepool related to instance creation bug and then more
apparently from overloading rabbitmq
* Email and tech blog post on renaming efforts

Thu:
* {{done}} https://phabricator.wikimedia.org/T170178
** renaming prometheus ldap user (not a system account) broke puppet stuff
** {{done}} https://phabricator.wikimedia.org/T170174
** {{done}} https://phabricator.wikimedia.org/T158968
** {{done}} https://phabricator.wikimedia.org/T131408
* tools.wmflabs.org reported down
** bblack broke nginx https stuff for jessie boxes with old packages
** upgraded nginx-common on project-proxy and tools proxy hosts
* {{done}} https://phabricator.wikimedia.org/T164842
** fixed stashbot 'cause it broke
** fixed sal 'cause it broke
** fixed bash 'cause it broke
** wrote a short blog announcement

Fri:
* Wiki gnoming
* Pahb gnoming

Sun:
* diskspace on tools-logs-02 & tools-worker-1020
* account creation broken on wikitech because of things I messed up in LDAP
** https://phabricator.wikimedia.org/T170774

Mon:
* Fixed T170774 by putting the uids back into the correct range
* Techops meeting
* Sent techops notes to list

Bryan
-- 
Bryan Davis              Wikimedia Foundation    <bd808 at wikimedia.org>
[[m:User:BDavis_(WMF)]] Manager, Cloud Services          Boise, ID USA
irc: bd808                                        v:415.839.6885 x6855