May 2024 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - alert1001/MediaWiki CirrusSearch update rate - codfw is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: MediaWiki CirrusSearch update rate - codfw Host: alert1001 Address: 208.80.154.88 State: CRITICAL Date/Time: Wed May 15 17:38:11 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23No_updates https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44… Acknowledged by : Additional Info: CRITICAL: 20.00% of data under the critical threshold [50.0]

50 minutes

1
0
0 0

** PROBLEM alert - alert1001/MediaWiki CirrusSearch update rate - codfw is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: MediaWiki CirrusSearch update rate - codfw Host: alert1001 Address: 208.80.154.88 State: CRITICAL Date/Time: Wed May 15 15:25:03 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23No_updates https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44… Acknowledged by : Additional Info: CRITICAL: 20.00% of data under the critical threshold [50.0]

3 hours, 3 minutes

1
0
0 0

** PROBLEM alert - alert1001/MediaWiki CirrusSearch update rate - codfw is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: MediaWiki CirrusSearch update rate - codfw Host: alert1001 Address: 208.80.154.88 State: CRITICAL Date/Time: Wed May 15 13:49:01 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23No_updates https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44… Acknowledged by : Additional Info: CRITICAL: 100.00% of data under the critical threshold [50.0]

4 hours, 39 minutes

1
0
0 0

** PROBLEM alert - alert1001/MediaWiki CirrusSearch update rate - codfw is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: MediaWiki CirrusSearch update rate - codfw Host: alert1001 Address: 208.80.154.88 State: CRITICAL Date/Time: Wed May 15 11:24:50 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23No_updates https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44… Acknowledged by : Additional Info: CRITICAL: 20.00% of data under the critical threshold [50.0]

7 hours, 3 minutes

1
0
0 0

** PROBLEM alert - alert1001/MediaWiki CirrusSearch update rate - codfw is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: MediaWiki CirrusSearch update rate - codfw Host: alert1001 Address: 208.80.154.88 State: CRITICAL Date/Time: Wed May 15 09:25:42 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23No_updates https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44… Acknowledged by : Additional Info: CRITICAL: 20.00% of data under the critical threshold [50.0]

9 hours, 2 minutes

1
0
0 0

Airflow alert: <TaskInstance: image_suggestions_weekly.convert_to_esbulk scheduled__2024-04-29T00:00:00+00:00 [failed]>

by search

16 hours, 18 minutes

1
0
0 0

** RECOVERY alert - alert1001/MediaWiki CirrusSearch update rate - codfw is OK **

by nagios＠alert1001.wikimedia.org

Notification Type: RECOVERY Service: MediaWiki CirrusSearch update rate - codfw Host: alert1001 Address: 208.80.154.88 State: OK Date/Time: Tue May 14 22:34:12 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23No_updates https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44… Acknowledged by : Additional Info: OK: Less than 1.00% under the threshold [80.0]

19 hours, 54 minutes

1
0
0 0

Airflow alert: <TaskInstance: transfer_to_es_hourly.convert_to_esbulk scheduled__2024-05-14T17:00:00+00:00 [failed]>

by search

21 hours, 12 minutes

1
0
0 0

[FIRING:1] CirrusStreamingUpdaterFlinkJobUnstable search-platform (cirrus_streaming_updater_consumer_search_eqiad cirrus-streaming-updater k8s consumer-search critical eqiad prometheus)

by sre-observability＠wikimedia.org

21 hours, 56 minutes

1
0
0 0

[FIRING:1] SystemdUnitFailed search-platform (elastic1083:9100 push_cross_cluster_settings_9600.service ops critical eqiad prometheus)

by sre-observability＠wikimedia.org

1 day, 19 hours

1
0
0 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts May 2024