[Engineering] Logstash goofyness from 2016-01-01 through 2016-01-20 (seeking input)

• Previous message: [Engineering] [Analytics] Eventlogging Mysql consumers downtime
• Next message: [Engineering] Logstash goofyness from 2016-01-01 through 2016-01-20 (seeking input)
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Something in the logstash process on logstash1001 went bananas when
midnight 2015-12-31 rolled over to 2016-01-01. Specifically it decided
that 2015 was so much fun that it would back up and do it again. This
wasn't noticed until late in the UTC day on 2016-01-20 when comparing
fatalmonitor output on fluorine to the kibana dashboard of the same
name. I restarted the logstash process and it started using the
correct year.

The open question I have now is should try and salvage the 80 million
logs events that are in the wrong Elasticsearch indices or just delete
them? Fixing them will require dumping the log record data out as json
blobs, changing the date and loading them back into the correct
Elasticsearch index. I'm leaning towards just deleting them unless
someone has a very strong objection.

Bryan
-- 
Bryan Davis              Wikimedia Foundation    <bd808 at wikimedia.org>
[[m:User:BDavis_(WMF)]]  Sr Software Engineer            Boise, ID USA
irc: bd808                                        v:415.839.6885 x6855

• Previous message: [Engineering] [Analytics] Eventlogging Mysql consumers downtime
• Next message: [Engineering] Logstash goofyness from 2016-01-01 through 2016-01-20 (seeking input)
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]