[Foundation-l] Wikimedia Projects Growth Animated

Wed Mar 26 20:46:29 UTC 2008

Hello,

> I do lots of cluster computing, and offered my cluster to Erik, but  
> his job
> can't be run that way. I also have jobs that are very difficult to  
> run on a
> cluster, such as computing en.WP's pagerank (this can be done with the
> Parallel Boost Graph Library, but are you guys really going to  
> install MPI
> and PBS/Maui?). I image there are lots more.

Well, I expect that not every paralleled computing needs real-time  
clustering libs, merely properly queueing jobs and handling them  
could be enough.
Pity that Erik's task is that resource hungry.

> how long does it take?). You'll have to have a Large Instance ($.40  
> per
> instance hour) to run the stats job which is four times more  
> expensive than
> the small instance.

Well, one can use node-local storage, and handle just uncompressed  
stream, instead of placing complete uncompressed dump on file system  
or even S3.
Anyway, I do not know exact costs. Last time I was looking at S3-for- 
wikipedia, the costs were far too high.

Was just simply thinking, that we could go into grid direction, if  
tasks were actually grid-able :)
We just really don't seem to have spare cycles for dedicating high- 
capacity nodes.

Now, should we buy one? Foundation hasn't been into scientific  
computing & number crunching so far, so that would mean entering  
slightly other domain.

-- 
Domas Mituzas -- http://dammit.lt/ -- [[user:midom]]