Update on this. Stefan and I got it running again. Everything seems to be
in order but please let us know right away if something's wrong.
As far as the lost data - "Get Data" won't work for past reports. They are
in the log files and if you really need the old reports and can't re-run
them, I can try to rescue them out of the service log files. I apologize
for this nasty inconvenience. We will stress test redis and make sure it
behaves well in the future.
Dan
On Wed, Oct 23, 2013 at 6:47 PM, Dan Andreescu <dandreescu(a)wikimedia.org>wrote;wrote:
Hi,
I had to reboot the redis server that stores all of your report results
today. This uncovered the very unpleasant truth that redis was never
backing up properly. Unfortunately, any report results that you had saved
are now lost. They can be rerun of course, but that gets me to the second
piece of bad news.
Even more frustratingly, I can't seem to get redis to come back up again.
We are working on this problem and will try to fix it ASAP. I will post
here once it's fixed.
Dan