April 2019 - Discovery-alerts - lists.wikimedia.org

** RECOVERY alert - logstash1012/ElasticSearch health check for shards on 9200 is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: logstash1012 Address: 10.64.48.177 State: OK Date/Time: Tue Apr 9 12:18:29 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: OK - elasticsearch status production-logstash-eqiad: unassigned_shards: 36, delayed_unassigned_shards: 0, cluster_name: production-logstash-eqiad, status: yellow, task_max_waiting_in_queue_millis: 53726, number_of_data_nodes: 3, number_of_nodes: 6, number_of_pending_tasks: 4, timed_out: False, number_of_in_flight_fetch: 0, initializing_shards: 4, active_shards: 162, relocating_shards: 0, active_shards_percent_as_number: 80.19801980198021, active_primary_shards: 86

5 years, 1 month

1
0
0 0

** PROBLEM alert - logstash1012/ElasticSearch health check for shards on 9200 is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: logstash1012 Address: 10.64.48.177 State: CRITICAL Date/Time: Tue Apr 9 12:17:29 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Max retries exceeded with url: /_cluster/health (Caused by NewConnectionError(requests.packages.urllib3.connection.HTTPConnection object at 0x7fc83aba8320: Failed to establish a new connection: [Errno 111] Connection refused,))

5 years, 1 month

1
0
0 0

** ACKNOWLEDGEMENT alert - icinga1001/Mediawiki Cirrussearch update lag - eqiad is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: ACKNOWLEDGEMENT Service: Mediawiki Cirrussearch update lag - eqiad Host: icinga1001 Address: 208.80.154.84 State: CRITICAL Date/Time: Fri Apr 5 10:14:51 UTC 2019 Notes URLs: https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44… Additional Info: CRITICAL: 100.00% of data above the critical threshold [50.0]

5 years, 1 month

1
0
0 0

** ACKNOWLEDGEMENT alert - icinga1001/Mediawiki Cirrussearch update lag - codfw is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: ACKNOWLEDGEMENT Service: Mediawiki Cirrussearch update lag - codfw Host: icinga1001 Address: 208.80.154.84 State: CRITICAL Date/Time: Fri Apr 5 10:14:51 UTC 2019 Notes URLs: https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44… Additional Info: CRITICAL: 100.00% of data above the critical threshold [50.0]

5 years, 1 month

1
0
0 0

** RECOVERY alert - search.svc.eqiad.wmnet/ElasticSearch health check for shards on 9443 is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9443 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: OK Date/Time: Fri Apr 5 08:34:21 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: OK - elasticsearch status production-search-omega-eqiad: number_of_data_nodes: 18, status: yellow, number_of_pending_tasks: 11, active_shards: 2917, task_max_waiting_in_queue_millis: 4653, cluster_name: production-search-omega-eqiad, active_primary_shards: 1107, timed_out: False, initializing_shards: 6, unassigned_shards: 397, number_of_nodes: 18, number_of_in_flight_fetch: 0, relocating_shards: 0, active_shards_percent_as_number: 87.86144578313252, delayed_unassigned_shards: 0

5 years, 1 month

1
0
0 0

** PROBLEM alert - search.svc.eqiad.wmnet/ElasticSearch health check for shards on 9443 is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9443 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: CRITICAL Date/Time: Fri Apr 5 08:31:47 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: CRITICAL - elasticsearch inactive shards 634 threshold =0.15 breach: cluster_name: production-search-omega-eqiad, active_shards_percent_as_number: 80.90361445783132, initializing_shards: 0, number_of_data_nodes: 15, delayed_unassigned_shards: 0, active_shards: 2686, task_max_waiting_in_queue_millis: 0, relocating_shards: 0, timed_out: False, active_primary_shards: 1107, number_of_nodes: 15, status: yellow, number_of_pending_tasks: 0, unassigned_shards: 634, number_of_in_flight_fetch: 0

5 years, 1 month

1
0
0 0

** RECOVERY alert - search.svc.eqiad.wmnet/ElasticSearch health check for shards on 9643 is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9643 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: OK Date/Time: Fri Apr 5 08:24:43 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, initializing_shards: 4, status: yellow, number_of_in_flight_fetch: 0, number_of_nodes: 17, relocating_shards: 0, task_max_waiting_in_queue_millis: 40696, unassigned_shards: 19, number_of_pending_tasks: 5, active_primary_shards: 1087, number_of_data_nodes: 17, timed_out: False, delayed_unassigned_shards: 0, active_shards: 3238, active_shards_percent_as_number: 99.29469487887151

5 years, 1 month

1
0
0 0

** PROBLEM alert - search.svc.eqiad.wmnet/ElasticSearch health check for shards on 9643 is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9643 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: CRITICAL Date/Time: Fri Apr 5 08:22:07 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: CRITICAL - elasticsearch inactive shards 592 threshold =0.15 breach: number_of_nodes: 15, number_of_in_flight_fetch: 0, status: yellow, timed_out: False, number_of_pending_tasks: 0, active_shards: 2669, active_shards_percent_as_number: 81.84605949095369, unassigned_shards: 592, relocating_shards: 0, initializing_shards: 0, task_max_waiting_in_queue_millis: 0, active_primary_shards: 1087, delayed_unassigned_shards: 0, cluster_name: production-search-psi-eqiad, number_of_data_nodes: 15

5 years, 1 month

1
0
0 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9643 is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Thu Apr 4 14:00:13 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: OK - elasticsearch status production-search-psi-codfw: number_of_in_flight_fetch: 0, number_of_data_nodes: 14, cluster_name: production-search-psi-codfw, active_shards_percent_as_number: 85.70990493713585, active_shards: 2795, unassigned_shards: 466, relocating_shards: 0, task_max_waiting_in_queue_millis: 0, status: yellow, number_of_nodes: 14, timed_out: False, initializing_shards: 0, delayed_unassigned_shards: 466, active_primary_shards: 1087, number_of_pending_tasks: 0

5 years, 1 month

1
0
0 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9643 is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Thu Apr 4 13:56:29 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: CRITICAL - elasticsearch https://search.svc.codfw.wmnet:9643/_cluster/health error while fetching: HTTPSConnectionPool(host=search.svc.codfw.wmnet, port=9643): Read timed out. (read timeout=4)

5 years, 1 month

1
0
0 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts April 2019