Hi all, There are a couple of major changes to our Cloud VPS o11y stack that I'm planning to make in the near term. Most of this should be visible on Phabricator as well, but I wanted to make everyone aware here regardless since following activity on Phabricator is hard and I don't want to cause any major surprises here. I'm sure some of this will have an effect on our users and I/we need to communicate it beforehand, but I'm not at that stage quite yet. == 1. New Grafana instance: grafana.wmcloud.org == The first and hopefully least impactful change is replacing the current grafana-cloud.wikimedia.org (aka grafana-labs.wikimedia.org) Grafana instance with a new one. The reason is that the current one runs directly on hardware (cloudmetrics*.eqiad.wmnet), and due to upstream Grafana changes it soon won't be able to reach out to Prometheus instances living on cloud-vps VMs. This work is tracked as T307465, and has patches up for review starting from https://gerrit.wikimedia.org/r/c/operations/puppet/+/869210/. == 2. Diamond removal == The Prometheus instance in metricsinfra now scrapes all Cloud VPS VMs. This was the primary blocker for getting rid of Diamond (a Python 2 program that collected node metrics and pushed them to Graphite). I hope that this transition will be mostly invisible to users if we migrate the most used Grafana dashboard (cloud-vps-project-board) to pull the metrics from Prometheus instead. This is tracked as T317032. == 3. Statsd/Graphite removal (once Diamond is gone) == My understanding is that the statsd/Graphite service was originally not intended as a generic service for cloud-vps users (although it certainly is used like one today). Either way we don't really have a good replacement for it except some limited cases that could use node-exporter text files instead. I'm not sure how big of a deal that is if we never claimed to support it anyway? This is tracked as T326266. Any questions or comments on the above? Taavi