Hi all,
Etherpad [0], our real-time colaborative editing tool suffered an
outage due to what we only know for now was database corruption. This
was detected shortly after it happened 14:27 UTC and we (ops in charge
of the service and the database) worked to reestablish the service.
As the service continued crashing despite our efforts, we decided to
recover a database backup from 2016-06-22 01:00:01 UTC. The service is
now back up and working since 16:11 UTC, but that means that you may
have lost a day and a half of edits in the current available etherpad
[0].
I understand that that may cause a lot of inconveniences, specially
for the people at Wikimania. *We are now trying to recover more than
that*, but as the corruption could come back, or not all could be
recovered, and people need the service the plan is the following:
- Keep the current pads as is, not delete or add anything from now.
You can continue using etherpad now as usual.
- If possible, recover the last days of edits on a separate location.
See [1] for progress if you are affected.
Sorry for the inconveniences. Please, more than ever, follow the
recommendation we added at the beginning of every empty pad:
"Keep in mind as well that there is no guarantee
that a pad's contents will always be available. A pad may be corrupted, deleted or
similar. Please keep a copy of important data somewhere else as well"
The
reason for this is that wiki content has proper HA and redundancy,
etherpad does not.
Again, my most sincere apologies,
[0] <https://etherpad.wikimedia.org/>
[1] <https://phabricator.wikimedia.org/T138516>
--
Jaime Crespo
<http://wikimedia.org>