Hi all,
The maintenance took around two hours of downtime. GitLab was switched over
to the other datacenter.
Unfortunately due to a misconfiguration *we lost some data*. All actions
(like pushes, merges, builds and comments) *between Feb 28th 0:30 UTC and
Feb 28th 2:00 UTC* (so one and a half hours) are lost. If you used GitLab
during that time make sure to re-apply your changes.
Apologies for the trouble. We took immediate action to prevent this from
happening again.
Some more context: The gitlab replicas (passive instances) run a restore
from the most recent backup every day at 2:00 UTC to test the restore and
keep them in sync with the production host. This was configured manually in
Puppet/hiera and was not updated during the switchover in
https://phabricator.wikimedia.org/T329931. So the restore run on the
production host from the most recent backup (finished at around 0:30 UTC).
In the future we want to automatically enable and disable the restore,
depending on the status of the machine (replica or production host). This
should prevent any unwanted restores on the production host in the future.
Again, sorry for the inconveniences. If you need help feel free to reach
out in T329931 <https://phabricator.wikimedia.org/T329931>[1] or on IRC in
the libera.chat #wikimedia-gitlab channel[2].
Greetings
[1]: <https://phabricator.wikimedia.org/T329931>
[2]: <ircs://irc.libera.chat:6697/wikimedia-gitlab>
On Sun, Feb 26, 2023 at 2:30 AM Tyler Cipriani <tcipriani(a)wikimedia.org>
wrote:
Hello
*GitLab will be down this coming Monday, 27 February, from 10:00 AM–noon
UTC* for the datacenter switchover[0][1].
Much of this switchover window will be downtime, during which you will be
unable to access GitLab through ssh or the web interface.
Apologies for any inconvenience :(
The process will involve a full backup of the GitLab server in the eqiad
datacenter, which serviceops will then restore on the GitLab server in the
codfw datacenter.
You can follow along on IRC in the libera.chat #wikimedia-gitlab
channel[2].
Tyler Cipriani (he/him)
Engineering Manager, Release Engineering
Wikimedia Foundation
[0]: <https://phabricator.wikimedia.org/T327920>
[1]: <https://phabricator.wikimedia.org/T329931>
[2]: <ircs://irc.libera.chat:6697/wikimedia-gitlab>
_______________________________________________
Wikitech-l mailing list -- wikitech-l(a)lists.wikimedia.org
To unsubscribe send an email to wikitech-l-leave(a)lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/