On Mon, Mar 25, 2013 at 1:34 PM, dan entous <d_entous(a)yahoo.com> wrote:
context
-------
i’m working on a mediawiki extension,
http://www.mediawiki.org/wiki/Extension:GWToolset,
which has as one of its goals, the ability to upload media files to a wiki. the extension,
among other tasks, will process an xml file that has a list of urls to media files and
upload those media files to the wiki. our ideal goal is to have this extension run on
http://commons.wikimedia.org/.
job queue goals
---------------
1. setup the processing of the xml file as a job queue job
2. each media file upload to be setup as a job queue job
current implementation
----------------------
i have been able to achieve goal 2 and will sort out goal 1 shortly.
issues/questions
----------------
1. each of these jobs can take several seconds to complete. i have noticed in my local
wiki that each of these jobs is picked up with each wiki visit and slows down the response
of the wiki by however many seconds the job takes to run, a sleep in the job shows that if
the job takes 15 seconds to run the wiki will be slowed down by that amount of time; i
don't want this to happen on my local wiki or on commons.
a. are jobs on commons run as part of each wiki visit?
b. is there a cron job that takes care of the job queue on commons instead of using
each wiki visit?
c. if not, is there a way to indicate that the job should only be run as a cron job
and not with a wiki visit?
2. if there's no solution to running the job with each wiki visit and slowing down
the site, what other suggestions are there on processing the xml file and individual media
file uploads?
thanks in advance!
dan
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Given your use-case, it seems like making a maintenance script that
reads the xml file and imports images might make more sense. Mass
uploads are rare enough events that I imagine going through the
process of somebody running the maintinance script would not be
prohibitive (But that's just a guess).
a. are jobs on commons run as part of each wiki
visit?
b. is there a cron job that takes care of the job queue on commons instead of using
each wiki visit?
No, commons uses runJobs.php (I believe. At the very least jobs are
not run from webrequests). I doubt cron is used, I imagine there are
enough jobs that there is a web server (probably more than one)
dedicated solely to dealing with the job queue at all times.
--bawolff
p.s. As an aside, looking for this source code I noticed its developed
on github "until it is ready for full review". You do realize if you
wanted to you could put the code in Wikimedia's gerrit and ignore the
review features until you're ready for the full review.