May 2024 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - graphite1005/CirrusSearch full_text eqiad 95th percentile latency is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: CirrusSearch full_text eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: CRITICAL Date/Time: Mon May 27 22:07:50 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 30.00% of data above the critical threshold [1000.0]

1 day, 10 hours

1
0
0 0

[FIRING:1] RdfStreamingUpdaterFlinkJobUnstable search-platform (WDQS_Streaming_Updater rdf-streaming-updater k8s wikidata critical codfw prometheus)

by sre-observability＠wikimedia.org

1 day, 20 hours

1
0
0 0

[FIRING:2] ProbeDown search-platform (wdqs1018:443 probes/custom ops info eqiad prometheus)

by sre-observability＠wikimedia.org

1 day, 21 hours

1
0
0 0

** RECOVERY alert - graphite1005/CirrusSearch more_like eqiad 95th percentile latency is OK **

by nagios＠alert1001.wikimedia.org

Notification Type: RECOVERY Service: CirrusSearch more_like eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: OK Date/Time: Mon May 27 04:04:03 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [1000.0]

2 days, 4 hours

1
0
0 0

** RECOVERY alert - graphite1005/CirrusSearch full_text eqiad 95th percentile latency is OK **

by nagios＠alert1001.wikimedia.org

Notification Type: RECOVERY Service: CirrusSearch full_text eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: OK Date/Time: Mon May 27 04:03:09 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [500.0]

2 days, 4 hours

1
0
0 0

** PROBLEM alert - graphite1005/CirrusSearch more_like eqiad 95th percentile latency is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: CirrusSearch more_like eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: CRITICAL Date/Time: Mon May 27 03:53:59 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 20.00% of data above the critical threshold [1500.0]

2 days, 4 hours

1
0
0 0

** PROBLEM alert - graphite1005/CirrusSearch full_text eqiad 95th percentile latency is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: CirrusSearch full_text eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: CRITICAL Date/Time: Mon May 27 03:53:09 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 40.00% of data above the critical threshold [1000.0]

2 days, 4 hours

1
0
0 0

Airflow alert: <TaskInstance: drop_old_data_daily.drop_dated_directories scheduled__2024-05-26T00:00:00+00:00 [failed]>

by search

2 days, 8 hours

1
0
0 0

Airflow alert: <TaskInstance: image_suggestions_weekly.convert_to_esbulk scheduled__2024-05-13T00:00:00+00:00 [failed]>

by search

3 days, 8 hours

1
0
0 0

Airflow alert: <TaskInstance: drop_old_data_daily.drop_dated_directories scheduled__2024-05-25T00:00:00+00:00 [failed]>

by search

3 days, 8 hours

1
0
0 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts May 2024