Notification Type: RECOVERY
Service: Galera haproxy failover
Host: cloudcontrol1003
Address: 208.80.154.23
State: OK
Date/Time: Wed Jan 26 21:11:33 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy
Acknowledged by :
Additional Info:
OK check_failover servers up 35 down 0:
Notification Type: RECOVERY
Service: Check unit status of labs-ip-alias-dump
Host: cloudservices1004
Address: 208.80.154.11
State: OK
Date/Time: Wed Jan 26 20:40:25 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_time…
Acknowledged by :
Additional Info:
OK: Status of the systemd unit labs-ip-alias-dump
Notification Type: RECOVERY
Service: Check systemd state
Host: cloudservices1004
Address: 208.80.154.11
State: OK
Date/Time: Wed Jan 26 20:31:27 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
Acknowledged by :
Additional Info:
OK - running: The system is fully operational
Notification Type: RECOVERY
Service: Galera haproxy failover
Host: cloudcontrol1004
Address: 208.80.154.132
State: OK
Date/Time: Wed Jan 26 19:45:58 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy
Acknowledged by :
Additional Info:
OK check_failover servers up 0 down 0:
Notification Type: PROBLEM
Service: Check systemd state
Host: cloudservices1004
Address: 208.80.154.11
State: CRITICAL
Date/Time: Wed Jan 26 19:41:02 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
Acknowledged by :
Additional Info:
CRITICAL - degraded: The following units failed: labs-ip-alias-dump.service
Notification Type: PROBLEM
Service: Check unit status of labs-ip-alias-dump
Host: cloudservices1004
Address: 208.80.154.11
State: CRITICAL
Date/Time: Wed Jan 26 19:35:14 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_time…
Acknowledged by :
Additional Info:
CRITICAL: Status of the systemd unit labs-ip-alias-dump
Notification Type: ACKNOWLEDGEMENT
Service: WMCS Galera Database
Host: cloudcontrol1004
Address: 208.80.154.132
State: CRITICAL
Date/Time: Wed Jan 26 19:00:37 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting
Acknowledged by andrew bogott: Andrew investigating
Additional Info:
Error during connection: Cant connect to MySQL server on 208.80.154.132 (115)
Notification Type: ACKNOWLEDGEMENT
Service: WMCS Galera Cluster
Host: cloudcontrol1004
Address: 208.80.154.132
State: CRITICAL
Date/Time: Wed Jan 26 19:00:37 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting
Acknowledged by andrew bogott: Andrew investigating
Additional Info:
Error during connection: Cant connect to MySQL server on 208.80.154.132 (115)