[discovery] Logstash not collecting logs while the elasticsearch cirrus cluster is down

Hello! Related to my previous incident report [1], we also had an issue with logstash [2]. Logstash stops collecting logs while elasticsearch / cirrus is down. This is most probably related to API Feature logging, which are sent by logstash to the cirrus cluster. Sadly, there are no obvious fix at this point. It might be possible to tune the elasticsearch output plugin to fail fast, but that is not obvious from the documentation. [1] https://wikitech.wikimedia.org/wiki/Incident_documentation/20170920-Elastic… [2] https://wikitech.wikimedia.org/wiki/Incident_documentation/20170920-Logstash -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation UTC+2 / CEST