[Labs-l] Processing dumps with Wikimedia Utilities

Sun May 18 16:55:40 UTC 2014

Is this the currentevents tool? It looks like a huge export is currently 
running on tools-login:

cp/public/dumps/public/eswiki/20140509/eswiki-20140509-pages-meta-history1.xml.bz2/public/dumps/public/eswiki/20140509/eswiki-20140509-pages-meta-history2.xml.bz2/public/dumps/public/eswiki/20140509/eswiki-20140509-pages-meta-history3.xml.bz2/public/dumps/public/eswiki/20140509/eswiki-20140509-pages-meta-history4.xml.bz2

Thanks to the cpu this is chewing on the box is rather laggy with a load 
of ~5. Please don't run this on tools-login, use the dev box or an 
execution node.

- Damian

On 18/05/2014 17:14, Emilio J. Rodríguez-Posada wrote:
> Well, the dump is reachable from the machine. But my script do this:
>
> fp = subprocess.Popen('7za e -bd -so %s 2>/dev/null' % dumpfilename, 
> shell=True, stdout=subprocess.PIPE, bufsize=65535)
>
> Is this correct in a grid?
>
>
> 2014-05-18 17:57 GMT+02:00 Emilio J. Rodríguez-Posada 
> <emijrp at gmail.com <mailto:emijrp at gmail.com>>:
>
>     Yeah... I have done that, but now a new problem. I can't read the
>     dump in the destination machine.
>
>     Where do I have to copy the dump?
>
>
>     2014-05-18 17:21 GMT+02:00 Jeremy Baron <jeremy at tuxmachine.com
>     <mailto:jeremy at tuxmachine.com>>:
>
>         On May 18, 2014 11:16 AM, "Emilio J. Rodríguez-Posada"
>         <emijrp at gmail.com <mailto:emijrp at gmail.com>> wrote:
>         > I have created the virtualenv for Python3, installed
>         mediawiki-utilities, etc. I can launch my script in that
>         virtualenv and works fine, but when I do 'jsub', the
>         destination machine obviously doesn't have that module:
>
>         Sounds like you need to use the activate script for your
>         virtualenv. You could do that in a wrapper shell script.
>
>         http://virtualenv.readthedocs.org/en/latest/virtualenv.html#activate-script
>
>         -Jeremy
>
>
>         _______________________________________________
>         Labs-l mailing list
>         Labs-l at lists.wikimedia.org <mailto:Labs-l at lists.wikimedia.org>
>         https://lists.wikimedia.org/mailman/listinfo/labs-l
>
>
>
>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/labs-l/attachments/20140518/1ae24383/attachment.html>