Neil Harris wrote Basing yourself on ... ?

Ashar Voultoiz wrote: Hence my choice of London and NL. There's plenty of capacity between the two, as well. -- Neil

Anthere wrote: Then Paris is fine by me! Please don't take my comments as denigrating Paris; it's on the same pan-European fibre rings as most of the other big western EU cities. -- Neil

Anthere wrote: Clearly one of the most important points :o) -- Ashar Voultoiz

On Wed, 21 Jul 2004 20:01:49 +0200, Anthere <anthere9(a)yahoo.com> wrote: The English Wikis use a greater portion of the available resources. That means the new servers would see a similar proportion of their resources used for serving up English pages. Or, put another way, an across-the-board 10% increase in the number of pages served translates into a far larger number for the English Wikis than anyone else. I don't really agree that it's a very good argument (after all, everybody would still see the same percentage performance increase) but it does mean that the English Wikipedia will see more absolute benefit from the new servers than anyone else. -Bill Clark

That sounds excellent, but isn't that technically difficult? Maybe I've missed some discussion of how to do this. It's easy to fix it so that fr, es, de, are all routed through the European squid proxies: we just point the dns entries at the new ip numbers for the French squid cluster, no problem. It's much harder (right?) to identify where a user is coming from and route them through the proxy that's best for them. This is what Akamai and people like this charge big bucks for doing. Or, is there a free (or cheap) way to approximate that? --Jimbo

A few years ago, I took the databases of RIPE and the APNIC (ARIN's were unavailable) and, using a little bit of C and Perl, converted them into multiple-terminal binary decision diagrams mapping each IP address to a country. The resulting database fit on a floppy disk. Checking one IP address needed at most 32 pointer dereferences in a table that easily fits into a small part of the RAM of modern computers. I haven't updated the database since then but this may be done by downloading tables from RIPE and APNIC and processing them. Regards

Jimmy (Jimbo) Wales wrote: == IP --> location mapping == The key thing here is that we don't need to be very accurate, providing we are reasonably consistent. The simplest way to handle things is to assume that all RIPE-registered IP addresses are somewhere in Europe, and all other addresses (ARIN, APNIC,. LACNIC...) are somewhere else "nearer" the US. Then we can use Wikimedia's own DNS servers to serve different lists of server IP addresses depending on the source network of the request. IANA allocate /8 blocks of IP addresses to the regional registries, and maintain a helpful up-to-date machine-readable list at http://www.iana.org/assignments/ipv4-address-space That will probably get you 80%-90% of the goodness of a commercial solution. For another alternative, you might want to look at http://countries.nerd.dk/ which seems to be taken from more fine-grained inspecting of registry data. == Serving the DNS responses == I'm still not sure about what the best way is to do this. Running a couple of instances of BIND on different ports on the authoritative servers for wikipedia.org and doing reverse-NAT based on source addresses for packets that arrive on port 53 is a crude way of doing things, but it's the first that comes to mind. Another is to hack an existing small DNS server implementation: unless someone has done it already? -- Neil

On Thu, 22 Jul 2004 14:01:21 -0400 (EDT), Ricky Beam <jfbeam(a)bluetronic.net> wrote: user_Jamesday pointed out another solution using the Supersparrow libraries, which I initially poo-pood as potential vaporware but upon further investigation find to be rather interesting. They have a patch for BIND9 that allows some BGP-type magic to happen as a replacement for views, which should improve the capabilities of such a system tremendously. I got it mostly installed (their patch for BIND9 was broken, but mostly easily fixed, although I'm still having a few problems I probably won't have the time to sort out until after this weekend... going to a wedding). They also have an implementation that uses the Dents nameserver, but I'm not familiar with that so I don't know if that's useful at all. -Bill Clark

Luc Van Oostenryck wrote: I agree with Luc on this, but I freely admit that to me this is the only option realistically possible anyway, unless we have some way that I have not learned about to figure out where people are just by their ip number. --Jimbo

On Wed, 21 Jul 2004 18:07:05 -0400, Bill Clark <wclarkxoom(a)gmail.com> wrote: Here's the full breakdown: http://spamid.servebeer.com:8081/utils/include/spamid/registry_mappings.jsp -Bill Clark

On Wed, 21 Jul 2004 18:45:18 -0400, Bill Clark <wclarkxoom(a)gmail.com> wrote: (Oh, and for the record, djbdns CAN do this as well, I'd just never used that feature before.) -Bill Clark

In a discussion with French developers, it was apparent that the proposed colocation choice implied that, so as to minimize bandwidth to the US, the logged-in users should *not* be routed through the European squid proxies, but should be redirected to the US proxies. The simplest solution for this would be to have two sets of hostnames: language.wikipedia.org for non logged-in users (where language = en, fr, de, jp, whatever...) language.logged.wikipedia.org for logged-in users The former would be redirected to the latter when they log-in, and the latter to the former at log-out. Only the former would be subject to DNS answers depending on the geographical region. This seems a very easy solution to implement.

Am Wed, 21 Jul 2004 14:49:53 -0700 hat Jimmy (Jimbo) Wales <jwales(a)wikia.com> geschrieben: I see it regurlarly that german people are carrying IP's from other countries depending on their service provider in my shop. AOL for instance assigns US registered IP's to whoever they feel fit. --Manfred

On Wed, 21 Jul 2004 11:34:41 -0700, Jimmy (Jimbo) Wales <jwales(a)wikia.com> wrote: I'd just assumed that mindset was already there, and thought that it might impact the willingness of some to donate time and/or resources. However, I see your point about wanting to discourage that attitude, and so I guess that should trump other concerns. Sure it's true: Those new servers (except the Euro squid boxes) are going to spend more cycles serving up English pages than any other kind, just like all the other servers. Granted, if they weren't serving up English pages then many of those cycles would be wasted anyway -- but some people would probably rather see some cycles wasted if it meant slightly better performance for the pages in their own language. That's the same attitude you've already said you want to strongly discourage, though. I guess I was arguing more for using different clusters for different Wikis, and you're right that this doesn't really have anything directly to do with geography. Truth be told, I've always been an opponent of centralization in any form. I just don't trust the idea of having all of the machines in the same location, watched over by the same engineers, configured the same way, etc. Although it might be easier to reconfigure clusters if they're in the same location, it means that the same people with the same biases would be doing so. So I guess my point was that geographic diversity would also imply procedural and cultural diversity in the operation and configuration of the equipment, and THAT could provide for more flexibility in testing different ways of doing things. Maybe the French systems administrators and network engineers would have a different way of approaching a problem that wouldn't ever be tried if all of the servers remained in Florida. I just worry about the eggs-in-one-basket scenarios: Airplanes crashes, hurricanes and earthquakes (yes I know it's Florida :) and things of that nature. Like I said, I'm fundamentally opposed to centralization (and probably more than a bit paranoid). If nobody else really sees much benefit to having an independent datacenter setup, then I guess I'll stop trying to think up reasons to justify it. :) -Bill Clark

On Wed, 21 Jul 2004 13:28:11 -0700, Jimmy (Jimbo) Wales <jwales(a)wikia.com> wrote: Different local policies. I've had servers in many different datacenters at different times, and no two places seem to do things the same way. The same goes for engineering teams. This is part of the reason a lot of large companies set up R&D shops that are geographically and culturally distinct from their main engineering teams. You don't want everyone thinking the same, because it invariably leads to stagnation. If everyone responsible for physical management of the clusters is in the same place, they're likely to end up thinking the same way (which is a good thing too, at least in terms of everyone getting along). I'm probably wrong in this instance, though. There are enough people who can simply download the software and build their own custom clusters if they wanted, so this situation isn't really comparable to an in-house R&D team. In a sense, we ALREADY have many, many independent R&D shops, each with their own hardware. (Sorry, I correct and/or contradict myself all the time, because I like brainstorming ideas more than I do editing them... which ties in with my whole anti-centralization philosophy -- I don't want to unilaterally censor any ideas that pop into my head, but would rather voice them even if I have my doubts about their viability.) I'll cheerfully withdraw my suggestion that there's any significant benefit to keeping the servers separate, in this case. :) -Bill Clark