Hi all,
The maintenance took around two hours of downtime. GitLab was switched over to the other datacenter.
Unfortunately due to a misconfiguration we lost some data. All actions (like pushes, merges, builds and comments) between Feb 28th 0:30 UTC and Feb 28th 2:00 UTC (so one and a half hours) are lost. If you used GitLab during that time make sure to re-apply your changes.
Apologies for the trouble. We took immediate action to prevent this from happening again.
Some more context: The gitlab replicas (passive instances) run a restore from the most recent backup every day at 2:00 UTC to test the restore and keep them in sync with the production host. This was configured manually in Puppet/hiera and was not updated during the switchover in https://phabricator.wikimedia.org/T329931. So the restore run on the production host from the most recent backup (finished at around 0:30 UTC). In the future we want to automatically enable and disable the restore, depending on the status of the machine (replica or production host). This should prevent any unwanted restores on the production host in the future.
Again, sorry for the inconveniences. If you need help feel free to reach out in T329931[1] or on IRC in the libera.chat #wikimedia-gitlab channel[2].
Greetings