GWToolset already has several throttles in place,
http://www.mediawiki.org/wiki/Extension:GWToolset/Technical_Design#Throttle…,
that limit how many background uploads are picked up with each background job run, and how
many total GWToolset background jobs can exist in the entire job queue. on the beta
cluster the background job seemed to vary in regards to how often it ran for GWToolset
varying between 7-30. that seems like enough time for additional images to get processed
in-between GWToolset images.
wouldn’t it be better to throttle the application/tool that generates thumbnails so that
it doesn’t try to produce too many thumbnails at once?
with kind regards,
dan
On Apr 25, 2014, at 20:41 , Gergo Tisza <gtisza(a)wikimedia.org> wrote:
On Fri, Apr 25, 2014 at 11:13 AM, Fæ
<faewik(a)gmail.com> wrote:
With no obvious immediate fix/work-around on the table from WMF ops, I
have proposed to re-start my uploads for this project with an
effective throttle by using 2 threads (this is a setting on the first
screen of the GWToolset. In practice, having tried a run of a couple
of hundred, this means that the tool is uploading 100MB sized images
at a rate of 2 every 5 minutes. This seems to not be causing any
issues.
The issue was not directly with the uploads; there is no thumbnail rendering happening on
upload, so GWToolset adding lots of large TIFFs quickly would not cause problems in
itself. The upload speed was problematic because that meant GWToolset saturated pages like
Special:NewFiles, and when somebody looked at such pages, *that* triggered lots of
thumbnail renderings of huge TIFF files at the same time. If GWToolset is slowed down and
lots of miscellaneous files are uploaded between the TIFFs, those special pages won't
be problematic, but something like a gallery or category of huge TIFF files could still
be.
_______________________________________________
Glamtools mailing list
Glamtools(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/glamtools