[Labs-admin] Ops meeting and clinic duty update 7/4

Wed Jul 5 07:22:35 UTC 2017

Hey y'all,

I just realized I hadn't sent this out sooner! Sorry.

Ops meeting notes

* Meeting notes at -
https://office.wikimedia.org/wiki/Operations/Operations_Meeting_Notes/TechOps-2017-07-03
* The ganeti incident has documentation now if you were interested -
https://wikitech.wikimedia.org/wiki/Incident_documentation/20170629-ganeti
* They gave updates on all the final goals for the upcoming quarter - cumin
and salt deprecation on labs was mentioned.
* DB team brought up https://phabricator.wikimedia.org/T169488 for our
attention, Bryan already responded
* Ops offsite - I brought this up and we talked about it for a few minutes.
Summary is - nothing is decided yet, Mark has just started to worry about
it a bit, and with no Rachel to help, and no decisions on when the
managers' meeting is, everything is up in the air at this point. They are
looking for volunteers to form an organizing team, ping Mark if you're
interested.

Clinic duty updates
* Both quarry and PAWS had issues on Friday - both were 'fixed' by me/Bryan
by restarting things. We should learn more about these services and how
they are set up!
* https://phabricator.wikimedia.org/T169118 was reported, I don't have a
lot of context here, and I haven't responded to it yet because I haven't
looked enough
* Folks still don't really ping me or use !help so I'm not responding if
I'm not constantly watching the channel.
* There have been couple pings about mysterious data disappearance - Stas
reported that his data on /data/scratch/wdqs was missing and had to rebuild
it. SigmaWP reported files missing on his tool sigma
at ~/cherry/cherryhtml/, things seem normal to me - it doesn't seem like
the two are related either, just really weird. He reported it late evening
today, so I haven't looked at the backup in codfw yet and compared, but
will check tomorrow.

Best,
-- 
Madhu :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-admin/attachments/20170705/388c4225/attachment.html>