----- Original Message ----- Oh, of course. It might; how would a 6to4NAT affect blocking? I mused on NANOG yesterday as to what was going to happen when network techs started realizing they couldn't carry around a bunch of IPs in their heads anymore... His phrasing seemed a bit.. insufficiently clear, to me. That was me, attempting to clarify. Sure. The S/N is still lower than the Hats would prefer, but that's the nature of an expanding universe. Cheers, - jra

In article <30181972.4621.1296767510190.JavaMail.root(a)benjamin.baylink.com>om>, Jay Ashworth <jra(a)baylink.com> wrote: No, it won't. The internal network IPs (which are used for communication between the proxy and the back-end Apache) are not publicly visible and are completely inconsequential to users. ISP NATs are a separate issue, and might be interesting; if nothing else, as one reason (however small) for ISPs to provide IPv6 to end users. ("Help! I can't edit Wikipedia because my ISP's CGNAT pool was blocked!".) The general situation with existing ISPs that use transparent proxies is that sometimes users just can't edit. Admins try to document such addresses and avoid blocking them for too long. Okay. I feel your clarification was not very clear ;-) ARIN didn't issue any /8s today, IANA did. ARIN was one of the *recipients* of those /8s. - river.

On Thu, Feb 3, 2011 at 1:41 PM, Jay Ashworth <jra(a)baylink.com> wrote: There's no reason to NAT between the squid proxies and apaches -- they share a private network, with a private IPv4 address space which is nowhere near being exhausted. Front-end proxies need to speak IPv6 to the outside world so they can accept connections from IPv6 clients, add the clients' IPv6 addresses to the HTTP X-Forwarded-For header which gets passed to the Apaches, and then return the response body back to the client. The actual backend Apache servers can happily hum along on IPv4 internally, with no impact on IPv6 accessibility of the site. -- brion

On Thu, Feb 3, 2011 at 1:53 PM, River Tarnell <r.tarnell(a)ieee.org> wrote: (The obvious fix there is to use IPv6 for that...) IIRC the Amsterdam proxies connect to the Tampa proxies on the internet, not directly to the back-end Tampa Apaches on the internal network. Something NAT-ish actually shouldn't hurt here, since the Amsterdam proxy addresses are whitelisted and thus skipped over for IP tracking -- we'd still have the original requestor's native IPv4 or IPv6 address in the X-Forwarded-For, and it won't matter if we see a particular proxy's IP or the proxy cluster's NAT IP on the other end. I'm not sure offhand how the cache-clearing signals are working these days, so not sure if that'd be affected (notifications from MediaWiki that particular pages have been updated and their URLs must be purged from cache need to be delivered to all our front-end proxies; this at least used to be done with local-network multicast and a proxy that rebroadcasted the multicast over in Amsterdam; if it still works like that then I don't think it'll be too affected). -- brion

On Thu, Feb 3, 2011 at 4:45 PM, Brion Vibber <brion(a)pobox.com> wrote: Interesting. Is there a standard for using IPv6 inside X-Forwarded-For headers? I would think you'd need a new header altogether. (Yes, this is just used internally so it doesn't matter, but I'm still curious.

On Thu, Feb 3, 2011 at 5:10 PM, River Tarnell <r.tarnell(a)ieee.org> wrote: Not even a de-facto one?

In article <9259756.4629.1296769269783.JavaMail.root(a)benjamin.baylink.com>om>, Jay Ashworth <jra(a)baylink.com> wrote: No, since the client IP is passed via the XFF header. (In any case, putting NAT there doesn't seem very likely to me.) - river.

On Thu, Feb 3, 2011 at 1:21 PM, George Herbert <george.herbert(a)gmail.com> wrote: Q: Are we doing tproxy between the squids and apache servers? That's the obvious not-supported situation with Squid and IPv6 with IPv4 backends. (That would be solved by adding IPv6 addresses to the Apaches, however). -- -george william herbert george.herbert(a)gmail.com

On Thu, Feb 3, 2011 at 1:35 PM, River Tarnell <r.tarnell(a)ieee.org> wrote: As Jay pointed out - handling of blocks (and logins) is an issue (at least, strongly potentially). But without knowing which shaped bricks are in use... -- -george william herbert george.herbert(a)gmail.com

----- Original Message ----- I rarely have answer, but I do try to ask good questions. And yes, NAT was a poor choice of terms. Cheers, -- jra

In article &lt;AANLkTikBWLOYHZy4jLN6jwkpHFjoTGO-PpqxfwuPf1EZ(a)mail.gmail.com&gt;om>, George Herbert &lt;george.herbert(a)gmail.com&gt; wrote: As far as I know, yes. I don't know if the plan is to update to a newer Squid, or to switch to Varnish entirely. <http://wikitech.wikimedia.org/view/IPv6_deployment> mentions either using another front-end proxy, or upgrading. - river.

On 04/02/11 08:13, George Herbert wrote: It's not necessary for the main Squid cluster to support IPv6 in order to serve the main website via IPv6. The amount of IPv6 traffic will presumably be very small in the short term. We can just set up a single proxy server in each location (Tampa and Amsterdam), and point all of the relevant AAAA records to it. All the proxy has to do is add an X-Forwarded-For header, and then forward the request on to the relevant IPv4 virtual IP. The request will then be routed by LVS to a frontend squid. MediaWiki already supports IPv6, so that's it, that's all you have to do. It would be trivial, except for the need to handle complaints from users and ISPs with broken IPv6 routing. What will be more difficult is setting up IPv6 support for all our miscellaneous services: Bugzilla, OTRS, Subversion, mail, etc. Many of those will be harder to set up than the main website. -- Tim Starling

In article &lt;AANLkTikS7Kcenbz94UjhfOYi6usRGSSf5VBrQCpK=V+J(a)mail.gmail.com&gt;om>, George Herbert &lt;george.herbert(a)gmail.com&gt; wrote: This is actually more of an issue than you might think... many users *already* have broken IPv6 connectivity[0], and it's only going to get worse with early adopters, since most (IPv4) users won't notice the problem. That might not be too bad, except most users tend not to report problems, and just assume the site is broken. Of course, in a couple of years when more sites support IPv6, broken connectivity will be much more obvious, and users will just complain to their ISP. - river. [0] Several years back I gave en.wikipedia.org an AAAA record for testing. (In hindsight, that was probably a bad idea, but anyway.) One of the users who was unable to access the site, and couldn't work out why, was another Wikimedia sysadmin.

On Thu, Feb 3, 2011 at 6:29 PM, Tim Starling &lt;tstarling(a)wikimedia.org&gt; wrote: I would recommend upgrading the Squid cluster because it's run on a very significantly old version of the software, lacks several years worth of general patches and maintenance, and because it's not THAT big a deal. As I mentioned earlier in thread, I spent several years running Squid (at the time, 3.0-stablevarious and 3.1 beta tests) at a large site, and it didn't take that much time and effort despite working actively with Amos and others on what turned out to be an uninitialized buffer problem for over a year and having to compile, tune, and seriously test all the versions from 3.0-STABLE3 through ... 19, it looks like. It was perhaps 20% of my total work for about 3 years, and would have been far less had it not been for the one persistent bug (going from the prior 2.6 squids to 3.0 took about 3 months of me 1/4 time-ish). Performance was noticeably better with 3.0 vs 2.6 and 2.7. Avoidance of obsolete version software rot is a key operations technique. My current main commercial consulting customer has 5 years-past-end-of-support key enterprise infrastructure software that they don't even quite know how to upgrade, it's so old now. Don't let your versions get that old... Yes, 2.7 is still getting necessary Squid project patches, latest to STABLE9 in March 2010, but still. It's old 8-) -- -george william herbert george.herbert(a)gmail.com

----- Original Message ----- That's so obvious I'm embarassed I didn't think of it. Given how big we are, though "very" small may be most websites "medium traffic day". :-) Cheers, -- jra

I'm glad this thread soon got to the point where we realise the problem is on the application layer level. So what are exactly the implications for blocking and related issues when we will start to see ISP level NATing? Am I right to assume that we will start seeing requests from say a global ISP NAT which may cover many clients, XFF 10.x.x.x? If so, do we need to be able to send both the ISP NAT IP, and the XFF IP to the servers, and amend the software so that we are able to block on the combination (so we can block, for example IP 9.10.11.12 XFF 10.45.68.15?) Will we be needing anon user- and user talk pages for a combination of ISP NAT IP and XFF IP? when ISP level NAT's show up? kind regards, Martijn. On Thu, Feb 3, 2011 at 11:01 PM, George Herbert &lt;george.herbert(a)gmail.com&gt; wrote:

On Thu, Feb 3, 2011 at 5:20 PM, River Tarnell &lt;r.tarnell(a)ieee.org&gt; wrote: But, "supports IPv6" could be as simple as having an http proxy server which sends (fake) IPv6 XFF headers. By fake, I mean that there's not even a need for the client to actually use that IPv6 address, so long as each user/session gets a different IP within a block controlled by that ISP.

* Martijn Hoekstra &lt;martijnhoekstra(a)gmail.com&gt; [Thu, 3 Feb 2011 23:12:27 +0100]: I already do something like that with IPv4 in my poll extension. I've noticed how many people are posting from private addresses behind the proxies with internet address, so I record both IP/XFF as anonymous user name (however private IP XFF by default is not recorded, you have to enable it with $wgUsePrivateIPs = true; XFF value becomes a subpage in NS_USER user page. Dmitriy

On Thu, Feb 3, 2011 at 3:21 PM, Aryeh Gregor &lt;Simetrical+wikilist(a)gmail.com&gt; wrote: There won't be much choice when the ISPs run out of IPv4 space to allocate new users. As I said - we'll see it in Asia soon enough, and then the US down the road a bit longer. -- -george william herbert george.herbert(a)gmail.com

On Thu, Feb 3, 2011 at 6:29 PM, George Herbert &lt;george.herbert(a)gmail.com&gt; wrote: You mean, when they have so little IPv4 space that they can't even fit all of their customers behind NAT? If they have enough IPv4 addresses at present to give out dedicated addresses to all users, they'll run out of addresses using NAT when they have maybe 10,000 to 100,000 times as many users as now, which seems unlikely to be anytime in the foreseeable future -- especially if traffic starts shifting to IPv6. NAT isn't a cure-all, but it works fine for browsing websites, which is all that directly concerns Wikipedia. The point remains, websites that are only accessible via IPv4 are not in any danger of becoming unreachable anytime soon by a large number of people. On Thu, Feb 3, 2011 at 7:02 PM, River Tarnell &lt;r.tarnell(a)ieee.org&gt; wrote: Right.

Wouldn't it be nice to just set up ipv6 a test? Something similar to https://secure.wikimedia.org/ . That way I can just open https://ipv6.wikimedia.org/wikipedia/en/wiki/Main_Page if I want to browse Wikipedia using ipv6. Maarten Ps. Of course https://secure6.wikimedia.org/ is even better ;-)

In article &lt;AANLkTikgM845ZOvSGqpdvq81juHN8Wm3RwZcxvBqN3oX(a)mail.gmail.com&gt;om>, Aryeh Gregor &lt;Simetrical+wikilist(a)gmail.com&gt; wrote: ISPs will probably do this, but I don't think it's right to say they'll *just* do this. In the US, for example, Comcast has been running IPv6 trials for a while, and expects to start giving end-user IPv6 addresses this year. So IPv6 for end users is coming, it's just taking longer than it should have. - river.