On Thu, Apr 21, 2016 at 1:45 AM, Brion Vibber <bvibber(a)wikimedia.org> wrote:
Over in TimedMediaHandler extension, we've had a
number of cases where old
code did things that were convenient in terms of squishing read-write
operations into data getters, that got removed due to problems with long
running transactions or needing to refactor things to support
future-facing multi-DC work where we want requests to be able to more
reliably distinguish between read-only and read-write. And we sometimes
want to put some of those clever hacks back and add more. ;)
For instance in
https://gerrit.wikimedia.org/r/284368 we'd like to remove
transcode derivative files of types/resolutions that have been disabled
automatically when we come across them. But I'm a bit unsure it's safe to
do so.
Note that we could fire off a job queue background task to do the actual
removal... But is it also safe to do that on a read-only request?
https://www.mediawiki.org/wiki/Requests_for_comment/Master_%26_slave_dataceā¦
seems to indicate job queueing will be safe, but would like to confirm
that. :)
Similarly in
https://gerrit.wikimedia.org/r/#/c/284269/ we may wish to
trigger missing transcodes to run on demand, similarly. The actual re
encoding happens in a background job, but we have to fire it off, and we
have to record that we fired it off so we don't duplicate it...
(This would require a second queue to do the high-priority state table
update and queue the actual transcoding job; we can't put them in one queue
because a backup of transcode jobs would prevent the high priority job from
running in a timely fashion.)
A best practices document on future-proofing for multi DC would be pretty
awesome! Maybe factor out some of the stuff from the RfC into a nice dev
doc page...
-- brion
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
When doing something like that from a read request, there's also the
problem for a popular page that there might be lots of views (maybe
thousands if the queue is a little backed up) before the job is
processed. So if the view triggers the job, and it will only stop
triggering inserting the job after the job has been executed, this
might cause a large number of useless jobs to be en-queued until one
of them is finally executed.
--
-bawolff