[Labs-l] Recovering from role::labs::vagrant data loss caused by puppet today

Tue Jul 15 21:43:15 UTC 2014

Thanks, those directions fixed my instance!

-- brion

On Tue, Jul 15, 2014 at 2:21 PM, Bryan Davis <bd808 at wikimedia.org> wrote:

> I made a patch to migrate the MediaWiki-Vagrant clone used by
> role::labs::vagrant from /mnt/vagrant to /srv/vagrant [0] that was
> merged into operations/puppet.git today. Unfortunately, this patch has
> a bug that leads to non-deterministic behavior. The outcome in the
> crap-bryan-why-did-you-break-that case is that the contents of the old
> /mnt/vagrant directory were copied to /srv/vagrant *before* the new
> disk partition was mounted. This happened to at least 2 labs-vagrant
> hosts that I know of. When it does happen, all is not lost!
>
> == Recovering old /mnt/vagrant contents ==
> # Becoming root just makes typing the rest of this easier
> $ su -i
>
> # Make a temporary mount point
> $ mkdir /mnt/bd808-is-a-dork
>
> # Re-mount the root disk at the temporary mount point
> $ mount --bind / /mnt/bd808-is-a-dork
>
> # Chdir to the shadowed /srv directory
> $ cd /mnt/bd808-is-a-dork/srv
>
> # Move the "new" copy of MediaWiki-Vagrant out of the way
> $ mv /srv/vagrant /srv/vagrant-that-bd808-messed-up
>
> # Copy the prior vagrant directory to /srv/vagrant
> $ tar cf - vagrant | (cd /srv; tar xvf -)
>
> # Re-provision via labs-vagrant
> $ cd /vagrant
> $ labs-vagrant provision
>
> # Check to make sure things are ok again.
> # Your instance may have had other content in /srv that you wish to
> restore.
>
> # Clean up the mess.
> $ rm -rf /mnt/bd808-is-a-dork/srv/vagrant
> $ umount /mnt/bd808-is-a-dork
> $ rm -rf /srv/vagrant-that-bd808-messed-up
>
>
> [0]: https://gerrit.wikimedia.org/r/#/c/145974
>
> Bryan
> --
> Bryan Davis              Wikimedia Foundation    <bd808 at wikimedia.org>
> [[m:User:BDavis_(WMF)]]  Sr Software Engineer            Boise, ID USA
> irc: bd808                                        v:415.839.6885 x6855
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/labs-l/attachments/20140715/aa2fd592/attachment.html>