So how does WMF set up its Apaches? - Wikitech-l - lists.wikimedia.org

List overview All Threads
Download

So how does WMF set up its Apaches?

Message tags

Please review: MediaWiki...

David Gerard

27 Oct 2011 27 Oct '11

12:30 p.m.

As I understand it, WMF has (a) hundreds of Apaches (b) running Ubuntu. How do you set up the Apaches to a substantially-identical configuration? I looked on the wikitech wiki and couldn't see it. (I ask only for my own interest - I currently want to take the Debian/Ubuntu apache2 out and have it shot.) - d.

Reply

Show replies by date

Chad

27 Oct 27 Oct

12:34 p.m.

On Thu, Oct 27, 2011 at 8:30 AM, David Gerard <dgerard(a)gmail.com> wrote:

As I understand it, WMF has (a) hundreds of Apaches (b) running Ubuntu. How do you set up the Apaches to a substantially-identical configuration? I looked on the wikitech wiki and couldn't see it. (I ask only for my own interest - I currently want to take the Debian/Ubuntu apache2 out and have it shot.)

Puppet :D -Chad

Reply

David Gerard

12:40 p.m.

On 27 October 2011 13:34, Chad <innocentkiller(a)gmail.com> wrote:

On Thu, Oct 27, 2011 at 8:30 AM, David Gerard <dgerard(a)gmail.com> wrote:

> As I understand it, WMF has (a) hundreds of Apaches (b) running > Ubuntu. How do you set up the Apaches to a substantially-identical > configuration? I looked on the wikitech wiki and couldn't see it.

Puppet :D

I was afraid of that. (Someone else I asked says the same.) /me queues another yak up for the shaving - d.

Reply

Roan Kattouw

12:42 p.m.

On Thu, Oct 27, 2011 at 2:40 PM, David Gerard <dgerard(a)gmail.com> wrote:

I was afraid of that. (Someone else I asked says the same.) /me queues another yak up for the shaving

Most of our puppet config was recently made public in a git repository, see https://blog.wikimedia.org/2011/09/19/ever-wondered-how-the-wikimedia-serve… . Roan

Reply

David Gerard

1:07 p.m.

On 27 October 2011 13:42, Roan Kattouw <roan.kattouw(a)gmail.com> wrote:

On Thu, Oct 27, 2011 at 2:40 PM, David Gerard <dgerard(a)gmail.com> wrote:

> I was afraid of that. (Someone else I asked says the same.) > /me queues another yak up for the shaving

Most of our puppet config was recently made public in a git repository, see https://blog.wikimedia.org/2011/09/19/ever-wondered-how-the-wikimedia-serve…

*splutter* our previous setup, on Solaris, shoved all the config in a standardised httpd.conf with a very small localisation file. I was wondering how to do it in Debian apache2, with its bureaucratic arrangement of ordained subfolders. So I see you do it by ignoring the Debian way and shoving it all in a standardised httpd.conf - I'll do it that way and say I'm working to industry standards then! - d.

Reply

Jay Ashworth

2:41 p.m.

----- Original Message -----

From: "David Gerard" <dgerard(a)gmail.com>

As I understand it, WMF has (a) hundreds of Apaches (b) running Ubuntu. How do you set up the Apaches to a substantially-identical configuration? I looked on the wikitech wiki and couldn't see it. (I ask only for my own interest - I currently want to take the Debian/Ubuntu apache2 out and have it shot.)

You may be asking only for your own interest. I, on the other hand... well, I'd like to know for my own interest as well. On the other hand.... Boss-people: at what point is it cost effective for Wikimedia to detail someone at least part time to cleaning up and updating all of the various bits of outdated technical documentation on wm.org and meta and other such places to current and accurate? This seems like it would benefit the WMF technical staff just as much as we the public... Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274

Reply

Jon Davis

3:54 p.m.

"wm.org" referring to which site? Wikimedia.org? Mediawiki.org? Granted not everything is up to date, but there is a fair amount on Wikitech [1] that is being updated. A lot of the live configs are also on NOC [2]. Between sites like NOC and access to Puppet (via Git), you've got a majority of the data AS it is actually used (rather than as it was, when written, on wiki). -Jon [1] http://wikitech.wikimedia.org/view/Special:RecentChanges [2] http://noc.wikimedia.org/ On Thu, Oct 27, 2011 at 07:41, Jay Ashworth <jra(a)baylink.com> wrote:

----- Original Message -----

From: "David Gerard" <dgerard(a)gmail.com>

As I understand it, WMF has (a) hundreds of Apaches (b) running Ubuntu. How do you set up the Apaches to a substantially-identical configuration? I looked on the wikitech wiki and couldn't see it. (I ask only for my own interest - I currently want to take the Debian/Ubuntu apache2 out and have it shot.)

You may be asking only for your own interest. I, on the other hand... well, I'd like to know for my own interest as well. On the other hand.... Boss-people: at what point is it cost effective for Wikimedia to detail someone at least part time to cleaning up and updating all of the various bits of outdated technical documentation on wm.org and meta and other such places to current and accurate? This seems like it would benefit the WMF technical staff just as much as we the public... Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274 _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

-- Jon [[User:ShakataGaNai]] / KJ6FNQ http://snowulf.com/ http://ipv6wiki.net/

Reply

Jay Ashworth

4:52 p.m.

----- Original Message -----

From: "Jon Davis" <wiki(a)konsoletek.com>

I keep getting a sort of twisted Garfield reference out of your name, Jon. :-)

"wm.org" referring to which site? Wikimedia.org? Mediawiki.org?

On wikimedia/wikitech, about mediawiki.

Granted not everything is up to date, but there is a fair amount on Wikitech

I've gone through some of that lately, and it seemed to me like *a lot* of it was either actually out of date, or merely marked that way.

[1] that is being updated. A lot of the live configs are also on NOC [2]. Between sites like NOC and access to Puppet (via Git), you've got a majority of the data AS it is actually used (rather than as it was, when written, on wiki).

Well, sure, but that's nap-of-earth; without a solid 5000ft view grounding you, I suspect it's hard to make use of. I haven't yet read the book-prep page that was posted the other day; on reflection, I suspect that (and the book chapter that comes from it) will tell the things I'm looking for. Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274

Reply

Ryan Lane

5:32 p.m.

[1] that is being updated. A lot of the live configs are also on NOC [2]. Between sites like NOC and access to Puppet (via Git), you've got a majority of the data AS it is actually used (rather than as it was, when written, on wiki).

Well, sure, but that's nap-of-earth; without a solid 5000ft view grounding you, I suspect it's hard to make use of. I haven't yet read the book-prep page that was posted the other day; on reflection, I suspect that (and the book chapter that comes from it) will tell the things I'm looking for.

Basically everything is on noc and in puppet. Those will always be more up-to-date than our documentation. That said, most of our newer services are very well documented, and many of our older services have at least adequate documentation. AFAIK, the only services with less than adequate documentation are the PDF servers and search. Mobile is slightly out of date since we just changed that entire architecture, but it'll be up to date soon. - Ryan

Reply

Happy Melon

5:39 p.m.

On 27 October 2011 18:32, Ryan Lane <rlane32(a)gmail.com> wrote:

[1] that is being updated. A lot of the live configs are also on NOC [2]. Between sites like NOC and access to Puppet (via Git), you've got a majority of the data AS it is actually used (rather than as it was, when written, on wiki).

Well, sure, but that's nap-of-earth; without a solid 5000ft view

grounding

you, I suspect it's hard to make use of. I haven't yet read the book-prep page that was posted the other day; on reflection, I suspect that (and the book chapter that comes from it) will tell the things I'm looking for.

Basically everything is on noc and in puppet. Those will always be more up-to-date than our documentation. That said, most of our newer services are very well documented, and many of our older services have at least adequate documentation. AFAIK, the only services with less than adequate documentation are the PDF servers and search. Mobile is slightly out of date since we just changed that entire architecture, but it'll be up to date soon. - Ryan

That's not to say that the configuration files *themselves* couldn't do with a bit of a spring-clean, though. There are endless lines of commented-out hacks and twisted override-chains that make it very difficult to see where some things are configured from and what they are configured too. Of course, changing them runs the risk of inadvertantly removing some edgecase that was providing "expected behaviour" for some site; but that risk is run every time the config is updated *anyway*, it might as well be done in a controlled fashion. --HM

Reply

Ryan Kaldari

8:12 p.m.

We did actually hire a technical documentation person at one point, but it didn't work out. I hope we find another one some day. Ryan Kaldari On 10/27/11 7:41 AM, Jay Ashworth wrote:

----- Original Message -----

From: "David Gerard"<dgerard(a)gmail.com> As I understand it, WMF has (a) hundreds of Apaches (b) running Ubuntu. How do you set up the Apaches to a substantially-identical configuration? I looked on the wikitech wiki and couldn't see it. (I ask only for my own interest - I currently want to take the Debian/Ubuntu apache2 out and have it shot.)

You may be asking only for your own interest. I, on the other hand... well, I'd like to know for my own interest as well. On the other hand.... Boss-people: at what point is it cost effective for Wikimedia to detail someone at least part time to cleaning up and updating all of the various bits of outdated technical documentation on wm.org and meta and other such places to current and accurate? This seems like it would benefit the WMF technical staff just as much as we the public... Cheers, -- jra

Reply

Jay Ashworth

28 Oct 28 Oct

4:18 a.m.

----- Original Message -----

From: "Ryan Kaldari" <rkaldari(a)wikimedia.org>

We did actually hire a technical documentation person at one point, but it didn't work out. I hope we find another one some day.

I wish I didn't have a fulltime retainer right now. :-) If my site (the one for which I want to steal ideas from yours :-) doesn't pan out, perhaps I'll ping you on that. Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274

Reply

4588

days inactive

4589

days old

wikitech-l@lists.wikimedia.org

Manage subscription

11 comments

8 participants

tags (0)

participants (8)

Chad
David Gerard
Happy Melon
Jay Ashworth
Jon Davis
Roan Kattouw
Ryan Kaldari
Ryan Lane