[Foundation-l] Donation of DVDs to Wikimedia Foundation Projects

Tue Jan 2 17:50:06 UTC 2007

Anthony wrote:

>On 1/2/07, Robert Scott Horning <robert_horning at netzero.net> wrote:
>  
>
>>I'm sure you
>>could do some specific accounting to determine what it would cost for
>>the WMF to host just as storage on a hard drive about 1GB of data,
>>together with serving 1TB of data as bandwidth (assuming that this 1GB
>>file is downloaded at least 1000 times.... a rather conservative notion
>>given crawling engines alone not to mention interested Wikimedia users).
>>    
>>
>
>FWIW, none of the major crawlers download entire gigabyte sized files.
>
>For something that huge though, bittorrent seems like a much smarter
>choice, especially if others could be convinced to act as seeds.
>
>Anthony
>
Now that might be a very reasonable suggestion, and certainly an 
alternative to be discussed.  There are some problems with bittorrents, 
viewing from an archival point of view, but it is certainly an option if 
files this size become very popular.  Of course there have been several 
attempts to try and do a P2P version of Wikipedia as well, which has 
vetted many of the issues that would relate to bittorrents as well for 
things like this.  Some infrastructure issues still would have to be 
addressed if this is the route we would like to go for files this large.

And as far as crawlers are concerned, who said anything about "major 
crawlers" here?  Problems from idiots who write their own crawlers 
without thinking, and crazies who try to mirror Wikimedia projects the 
hard way can cause enough problems.  Besides, are you sure the Internet 
Archive doesn't download gigabyte sized files right now?  They 
constantly surprise me on what they do, and I've grabbed archived 
downloads from them in the past that were rather large.

And the curious alone would be a source of some bandwidth.

-- 
Robert Scott Horning