Hello!
Related to my previous incident report [1], we also had an issue with
logstash [2].
Logstash stops collecting logs while elasticsearch / cirrus is down.
This is most probably related to API Feature logging, which are sent
by logstash to the cirrus cluster. Sadly, there are no obvious fix at
this point. It might be possible to tune the elasticsearch output
plugin to fail fast, but that is not obvious from the documentation.
[1]
https://wikitech.wikimedia.org/wiki/Incident_documentation/20170920-Elastic…
[2]
https://wikitech.wikimedia.org/wiki/Incident_documentation/20170920-Logstash
--
Guillaume Lederrey
Operations Engineer, Discovery
Wikimedia Foundation
UTC+2 / CEST