September 2020 - Discovery-alerts

Update on build jenkins-wikidata-query-rdf-maven-release-docker-wdqs-25

by jenkins-bot

The build 25 for wikidata/query/rdf has finished. See https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-release-d… for details.

3 years, 7 months

1
0
0 0

** RECOVERY alert - elastic2029/Rate of JVM GC Old generation-s runs - elastic2029-production-search-psi-codfw is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: Rate of JVM GC Old generation-s runs - elastic2029-production-search-psi-codfw Host: elastic2029 Address: 10.192.16.192 State: OK Date/Time: Tue Sept 15 09:19:28 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_… https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-… Acknowledged by : Additional Info: (C)100 gt (W)80 gt 55.93

3 years, 8 months

1
0
0 0

** PROBLEM alert - elastic2029/Rate of JVM GC Old generation-s runs - elastic2029-production-search-psi-codfw is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: Rate of JVM GC Old generation-s runs - elastic2029-production-search-psi-codfw Host: elastic2029 Address: 10.192.16.192 State: CRITICAL Date/Time: Tue Sept 15 08:28:54 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_… https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-… Acknowledged by : Additional Info: 107.8 gt 100

3 years, 8 months

1
0
0 0

** RECOVERY alert - graphite1004/CirrusSearch codfw 95th percentile latency - more_like is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: CirrusSearch codfw 95th percentile latency - more_like Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Mon Sept 14 21:09:08 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?panelI… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [1200.0]

3 years, 8 months

1
0
0 0

** ACKNOWLEDGEMENT alert - graphite1004/CirrusSearch codfw 95th percentile latency - more_like is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: ACKNOWLEDGEMENT Service: CirrusSearch codfw 95th percentile latency - more_like Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Mon Sept 14 20:16:39 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?panelI… Acknowledged by Ryan Kemper: https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Appears to just be related to the dc switchover - eqiad has been offline (intentionally) Additional Info: CRITICAL: 80.00% of data above the critical threshold [2000.0]

3 years, 8 months

1
0
0 0

** PROBLEM alert - graphite1004/CirrusSearch codfw 95th percentile latency - more_like is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: CirrusSearch codfw 95th percentile latency - more_like Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Mon Sept 14 19:41:17 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?panelI… Acknowledged by : Additional Info: CRITICAL: 70.00% of data above the critical threshold [2000.0]

3 years, 8 months

1
0
0 0

** RECOVERY alert - elastic2029/Rate of JVM GC Old generation-s runs - elastic2029-production-search-psi-codfw is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: Rate of JVM GC Old generation-s runs - elastic2029-production-search-psi-codfw Host: elastic2029 Address: 10.192.16.192 State: OK Date/Time: Mon Sept 14 17:42:46 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_… https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-… Acknowledged by : Additional Info: (C)100 gt (W)80 gt 78.31

3 years, 8 months

1
0
0 0

Update on build jenkins-wikidata-query-rdf-maven-release-docker-wdqs-24

by jenkins-bot

The build 24 for wikidata/query/rdf has finished. See https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-release-d… for details.

3 years, 8 months

1
0
0 0

** PROBLEM alert - elastic2029/Rate of JVM GC Old generation-s runs - elastic2029-production-search-psi-codfw is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: Rate of JVM GC Old generation-s runs - elastic2029-production-search-psi-codfw Host: elastic2029 Address: 10.192.16.192 State: CRITICAL Date/Time: Mon Sept 14 16:23:40 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_… https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-… Acknowledged by : Additional Info: 113.9 gt 100

3 years, 8 months

1
0
0 0

** RECOVERY alert - logstash1007/ElasticSearch health check for shards on 9200 is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: logstash1007 Address: 10.64.0.37 State: OK Date/Time: Mon Sept 14 15:02:53 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-logstash-eqiad: task_max_waiting_in_queue_millis: 0, cluster_name: production-logstash-eqiad, timed_out: False, number_of_pending_tasks: 0, initializing_shards: 0, active_shards: 976, active_shards_percent_as_number: 100.0, number_of_nodes: 6, relocating_shards: 0, unassigned_shards: 0, status: green, delayed_unassigned_shards: 0, number_of_in_flight_fetch: 0, active_primary_shards: 543, number_of_data_nodes: 3

3 years, 8 months

1
0
0 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts September 2020