On Sat, Oct 11, 2008 at 12:32 PM, Thomas Dalton
<thomas.dalton(a)gmail.com>wrote;wrote:
Exactly. The only way you can speed up the
smaller dumps is the slow
down the bigger ones (or throw more money at the problem), and no-one
has given any reason why we should prioritise smaller dumps.
What about the original idea, to do the "current" dumps in a separate thread
from the "history" dumps. With only one thread doing the "history"
dumps
they'd take a really long time, but at least the "current" dumps would
finish in a reasonable amount of time.
There's something in that. Speed up the quick parts of all the dumps
at the expense of the slow parts of all the dumps. It's fair to each
project and probably correlates well with how people actually use the
dumps (do we have download logs to see how often people actually
download the whole thing?).