April 2024 - Cloud-admin-feed - lists.wikimedia.org

** PROBLEM alert - cloudbackup1002-dev/Disk space is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: Disk space Host: cloudbackup1002-dev Address: 10.64.16.59 State: CRITICAL Date/Time: Tue Apr 30 19:24:38 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudbac… Acknowledged by : Additional Info: DISK CRITICAL - free space: /srv/cinder-backups 333MiB (1% inode=98%):

4 days, 6 hours

1
0
0 0

[RESOLVED] PawsJupyterHubDown paws (hub-paws.wmcloud.org:443 jupyterhub critical wmcs)

by root＠wmflabs.org

5 days, 9 hours

1
0
0 0

[FIRING:1] PawsJupyterHubDown paws (hub-paws.wmcloud.org:443 jupyterhub critical wmcs)

by root＠wmflabs.org

5 days, 10 hours

1
0
0 0

[FIRING:1] ToolforgeKubernetesWorkerTooManyDProcesses tools (tools-k8s-worker-nfs-42 node-exporter warning prometheus D wmcs)

by root＠wmflabs.org

5 days, 15 hours

1
0
0 0

RECOVERY Host cloudbackup1003 - PING OK - Packet loss = 0%, RTA = 0.25 ms

by nagios＠alert1001.wikimedia.org

Notification Type: RECOVERY Host: cloudbackup1003 State: UP Address: 10.64.20.22 Info: PING OK - Packet loss = 0%, RTA = 0.25 ms Date/Time: Mon Apr 29 03:18:23 UTC 2024 Acknowledged by :

5 days, 22 hours

1
0
0 0

ACKNOWLEDGEMENT Host cloudbackup1003 - PING CRITICAL - Packet loss = 100%

by nagios＠alert1001.wikimedia.org

Notification Type: ACKNOWLEDGEMENT Host: cloudbackup1003 State: DOWN Address: 10.64.20.22 Info: PING CRITICAL - Packet loss = 100% Date/Time: Mon Apr 29 03:16:44 UTC 2024 Acknowledged by Andrew Bogott: rebooting/reimaging

5 days, 22 hours

1
0
0 0

** ACKNOWLEDGEMENT alert - cloudbackup1003/SSH is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: ACKNOWLEDGEMENT Service: SSH Host: cloudbackup1003 Address: 10.64.20.22 State: CRITICAL Date/Time: Mon Apr 29 03:16:43 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/SSH/monitoring Acknowledged by Andrew Bogott: rebooting/reimaging Additional Info: CRITICAL - Socket timeout after 10 seconds

5 days, 22 hours

1
0
0 0

PROBLEM Host cloudbackup1003 - PING CRITICAL - Packet loss = 100%

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Host: cloudbackup1003 State: DOWN Address: 10.64.20.22 Info: PING CRITICAL - Packet loss = 100% Date/Time: Mon Apr 29 03:15:55 UTC 2024 Acknowledged by :

5 days, 22 hours

1
0
0 0

[FIRING:1] TfInfraTestApplyFailed tf-infra-test (tf-bastion node critical wmcs)

by root＠wmflabs.org

6 days, 1 hour

1
0
0 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: Redis set/get is OK **

by nagios＠alert1001.wikimedia.org

Notification Type: RECOVERY Service: toolschecker: Redis set/get Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Sun Apr 28 14:35:50 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.012 second response time

6 days, 11 hours

1
0
0 0

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed April 2024