March 2023 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - relforge1004/ElasticSearch health check for shards on 9400 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9400 Host: relforge1004 Address: 10.64.21.126 State: CRITICAL Date/Time: Tue Mar 7 14:29:04 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 year, 3 months

1
0
0 0

** PROBLEM alert - relforge1004/ElasticSearch health check for shards on 9200 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: relforge1004 Address: 10.64.21.126 State: CRITICAL Date/Time: Tue Mar 7 14:28:48 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 year, 3 months

1
0
0 0

[FIRING:1] WdqsStreamingUpdaterFlinkJobNotRunning (WDQS_Streaming_Updater rdf-streaming-updater k8s critical eqiad prometheus search-platform)

by sre-observability＠wikimedia.org

1 year, 3 months

1
0
0 0

[FIRING:1] WcqsStreamingUpdaterFlinkJobNotRunning (WCQS_Streaming_Updater rdf-streaming-updater k8s critical eqiad prometheus search-platform)

by sre-observability＠wikimedia.org

1 year, 3 months

1
0
0 0

Airflow alert: <TaskInstance: transfer_to_es_hourly.upload_to_swift 2023-03-07T06:00:00+00:00 [failed]>

by noreply＠wikimedia.org

1 year, 3 months

1
0
0 0

** PROBLEM alert - cloudelastic1002/ElasticSearch health check for shards on 9400 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9400 Host: cloudelastic1002 Address: 208.80.154.133 State: CRITICAL Date/Time: Tue Mar 7 09:37:54 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 389 threshold =0.2 breach: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, active_primary_shards: 779, active_shards: 1170, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 389, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 75.04810776138551

1 year, 3 months

1
0
0 0

** PROBLEM alert - cloudelastic1006/ElasticSearch health check for shards on 9400 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9400 Host: cloudelastic1006 Address: 208.80.154.138 State: CRITICAL Date/Time: Tue Mar 7 09:36:56 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 389 threshold =0.2 breach: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, active_primary_shards: 779, active_shards: 1170, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 389, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 75.04810776138551

1 year, 3 months

1
0
0 0

** PROBLEM alert - cloudelastic1003/ElasticSearch health check for shards on 9400 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9400 Host: cloudelastic1003 Address: 208.80.154.83 State: CRITICAL Date/Time: Tue Mar 7 09:36:50 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 389 threshold =0.2 breach: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, active_primary_shards: 779, active_shards: 1170, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 389, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 75.04810776138551

1 year, 3 months

1
0
0 0

** PROBLEM alert - cloudelastic1004/ElasticSearch health check for shards on 9400 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9400 Host: cloudelastic1004 Address: 208.80.155.107 State: CRITICAL Date/Time: Tue Mar 7 09:36:36 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 389 threshold =0.2 breach: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, active_primary_shards: 779, active_shards: 1170, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 389, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 75.04810776138551

1 year, 3 months

1
0
0 0

[FIRING:1] CirrusSearchJobQueueLagTooHigh (k8s cirrusSearchElasticaWrite-cpjobqueue-partitioned-mediawiki-job-cirrusSearchElasticaWrite warning eqiad prometheus search-platform)

by sre-observability＠wikimedia.org

1 year, 3 months

1
0
0 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts March 2023