Hello,
Pretty much everyone who's dealt with creating views for new wikis on the
labs hosts have experienced issues with "Access denied" sometimes.
This was usually due to the MariaDB grant role being missed. We tried to
workaround this by including the grant addition on the maintain-views
script.
Unfortunately, we ran into very weird problems when doing so and this is an
example: https://phabricator.wikimedia.org/T193187#4273281
After lots of back and forth we decided to create a bug to MariaDB (
https://jira.mariadb.org/browse/MDEV-16466) which was confirmed by MariaDB
yesterday and pointed to a similar issue (
https://jira.mariadb.org/browse/MDEV-14732).
The expected fix will come in 10.4 (we are in 10.1), so quite long ahead of
us.
So, for now, the workaround before adding new views is to manually add the
GRANT on the DB and then run the script:
GRANT SELECT, SHOW VIEW ON `newiki\_p`.* to labsdbuser';
Hopefully with this email everyone is on the same page now.
Thanks everyone (specially Brooke for helping me out with the
troubleshooting!)
Manuel.
2018-08-15 20:00:02,328 INFO force is enabled
2018-08-15 20:00:02,394 INFO removing misc-project-backup
2018-08-15 20:00:02,511 INFO removing misc-project-backup
2018-08-15 20:00:02,991 INFO creating misc-project-backup at 2T
2018-08-15 20:00:04,013 INFO force is enabled
2018-08-15 20:00:04,047 INFO removing misc-snap
2018-08-15 20:00:04,090 INFO removing misc-snap
2018-08-15 20:00:04,483 INFO creating misc-snap at 1T
2018-08-14 20:00:02,818 INFO force is enabled
2018-08-14 20:00:02,868 INFO removing tools-project-backup
2018-08-14 20:00:02,951 INFO removing tools-project-backup
2018-08-14 20:00:03,589 INFO creating tools-project-backup at 2T
2018-08-14 20:00:04,565 INFO force is enabled
2018-08-14 20:00:04,623 INFO removing tools-snap
2018-08-14 20:00:04,662 INFO removing tools-snap
2018-08-14 20:00:06,198 INFO creating tools-snap at 1T
2018-08-08 20:00:03,064 INFO force is enabled
2018-08-08 20:00:03,111 INFO removing misc-project-backup
2018-08-08 20:00:03,194 INFO removing misc-project-backup
2018-08-08 20:00:03,636 INFO creating misc-project-backup at 2T
2018-08-08 20:00:04,620 INFO force is enabled
2018-08-08 20:00:04,687 INFO removing misc-snap
2018-08-08 20:00:04,744 INFO removing misc-snap
2018-08-08 20:00:05,166 INFO creating misc-snap at 1T
2018-08-07 20:00:03,018 INFO force is enabled
2018-08-07 20:00:03,036 INFO removing tools-project-backup
2018-08-07 20:00:03,093 INFO removing tools-project-backup
2018-08-07 20:00:03,662 INFO creating tools-project-backup at 2T
2018-08-07 20:00:04,592 INFO force is enabled
2018-08-07 20:00:04,640 INFO removing tools-snap
2018-08-07 20:00:04,678 INFO removing tools-snap
2018-08-07 20:00:07,181 INFO creating tools-snap at 1T
For the preceding week, things were relatively quiet.
* Two new wikis were created and needed setup on the replicas (satwiki and zhwikiversity).
* Efforts to build dumps torrents caused very high load on labstore1007
* Some pages came in from a puppet issue, which appears to be on the production puppet server.
From the SRE meeting:
* Jaime proposed standardization around monitoring naming. https://gerrit.wikimedia.org/r/c/operations/puppet/+/448503 <https://gerrit.wikimedia.org/r/c/operations/puppet/+/448503>
* prometheus-logstash-exporter is packaged and rolling out
* Most other things don’t directly affect cloud, yet. (progress on datacenter switch over, php7 work moving forward, etc)
Brooke Storm
Operations Engineer
Wikimedia Cloud Services
bstorm(a)wikimedia.org
IRC: bstorm_
Hey there,
we've just pushed the DNS changes required to serve grafana,
grafana-labs, and grafana-labs-admin through cache_text instead of
cache_misc (see T164609 for context).
Please do let Traffic know if you notice anything strange while using
grafana.
Thanks,
ema
In anticipation of the upcoming eqiad1 migration, I propose we have
another opt-in VPS project purge. I've made a dump of all projects and
instances with a proposed purge timeline here:
https://wikitech.wikimedia.org/wiki/Cloud_VPS_2018_Purge
I'm running this by all of you before I send anything to cloud-announce
in case you have suggestions for making this a smoother process and/or
think my proposed timeline is too draconian.
Thanks!
-A
2018-08-01 20:00:02,755 INFO force is enabled
2018-08-01 20:00:02,819 INFO removing misc-project-backup
2018-08-01 20:00:02,926 INFO removing misc-project-backup
2018-08-01 20:00:03,342 INFO creating misc-project-backup at 2T
2018-08-01 20:00:04,407 INFO force is enabled
2018-08-01 20:00:04,507 INFO removing misc-snap
2018-08-01 20:00:04,656 INFO removing misc-snap
2018-08-01 20:00:05,114 INFO creating misc-snap at 1T
2018-07-31 20:00:02,798 INFO force is enabled
2018-07-31 20:00:02,832 INFO removing tools-project-backup
2018-07-31 20:00:02,877 INFO removing tools-project-backup
2018-07-31 20:00:03,514 INFO creating tools-project-backup at 2T
2018-07-31 20:00:04,534 INFO force is enabled
2018-07-31 20:00:04,642 INFO removing tools-snap
2018-07-31 20:00:04,693 INFO removing tools-snap
2018-07-31 20:00:06,475 INFO creating tools-snap at 1T