Troubleshooting stuck Apache requests

List overview All Threads
Download

newer

older

Re: [MediaWiki-l] problem...

Apache vs. Nginx

Justin Lloyd

12 Nov 2015 12 Nov '15

4:07 a.m.

Hi all, *TL;DR - How I can determine what Apache processes are occasionally stuck waiting on (often leading to 502s due to hitting MaxClients) when trying to service MediaWiki requests?* I'm dealing with a problem where occasionally one or more of my wiki servers will hit its Apache limit of 100 connections (calculated based on total server memory and per-Apache process memory usage). Sometimes it will clear up on its own, often not. This is on Ubuntu 12.04 and often I'll see Apache processes stuck in the "sending" state via the server-status page, but I cannot figure out what it's waiting on aside from the Request column on the /server-status page, whenever I can actually request that page since Apache is usually unresponsive due to being at MaxClients. Other times, though, I'll see a stuck process despite being below MaxClients, but again I cannot figure out what the process is stuck waiting on. I have a hunch that it's image-heavy pages due to tons of thumbnails, but my wiki community controls that as I'm not a MediaWiki editor and that doesn't help me in figuring out what processes get stuck on. I've tried strace, lsof, pstack, viewing /proc/$pid/stack directly, Apache logs, etc. but none of that has helped me figure out why some processes hang, thus crowding out new ones and often leading to 502s. I have four load-balanced web servers and this hitting MaxClients sometimes happens on all four, leading to steady 502s, while other times it can be fewer than four servers, leading to broken-looking pages and/or intermittent 502s. Architectural considerations: * Each web server runs Varnish on port 80 with Apache (using APC) 2.2 hosting several MediaWiki 1.24.2 wikis as named-based vhosts on 127.0.0.01:8080 as the Varnish backend (and all four Varnishes in $wgMemCachedServers). Varnish connections tend to remain steady in their usual patterns while the Apaches spike to MaxClients, so it's not an unusual spike in Internet traffic to the wikis. * Six wikis are configured as Vhosts in Apache, load balanced by a separate set of front-end servers, where two of the wikis are for private internal use and the other four are public, though the traffic to one of the public wikis dwarfs the rest and it's the wiki giving me problems. * The upload directory is a symlink into an NFS-mounted filesystem with a subdirectory per wiki, e.g. $IP/images is a symlink to /var/www/images/$wikiname, where /var/www/images is the NFS mount. I never see NFS issues and the NFS server's Graphite dashboard shows the server to be *very* lightly loaded. * Apache talks to a separate beefy MySQL server and two dedicated Memcached servers for session and query caching. There are four MySQL instances on the server, via mysqld_multi, but the problem wiki's database has its own dedicated instance, so I am able to separate out database traffic from the rest of the wikis (and another web application that uses its own dedicated instance) and manage and tune the wiki's database instance independently. I'm mainly looking right now for how to troubleshoot the stuck processes, but any advice regarding this architecture is also welcome, as I feel it could use some improvement but I'm not sure how just yet. Justin

Show replies by date

Marcin Cieslak

12 Nov 12 Nov

9:42 a.m.

On 2015-11-12, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote:

...

* Six wikis are configured as Vhosts in Apache, load balanced by a separate set of front-end servers, where two of the wikis are for private internal use and the other four are public, though the traffic to one of the public wikis dwarfs the rest and it's the wiki giving me problems.

(...)

...

I'm mainly looking right now for how to troubleshoot the stuck processes, but any advice regarding this architecture is also welcome, as I feel it could use some improvement but I'm not sure how just yet.

Justin Lloyd

4:02 p.m.

Marcin, It's the biggest and most heavily trafficked of our wikis because its the English-language version of the wiki. We also have German, French, and Spanish, but the English-speaking community is by far the largest and most active. There are some tiny configuration differences between the wikis (e.g. the value of $wgJobRunRate, the specific extensions loaded) but nothing very significant I don't believe. I should also add that all four of these wikis (we have a 5th, for 7 total, not 6 as I'd originally said) also use Semantic MediaWiki extensively. I believe the other three wikis would run into the same problem if they had same amount of traffic as the English one. However, since they all are vhosts within the same Apache instances, the English one's problems affect all of them. Justin On Thu, Nov 12, 2015 at 1:42 AM, Marcin Cieslak <saper(a)saper.info> wrote:

...

On 2015-11-12, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote:

* Six wikis are configured as Vhosts in Apache, load balanced by a

separate

set of front-end servers, where two of the wikis are for private internal use and the other four are public, though the traffic to one of the

public

wikis dwarfs the rest and it's the wiki giving me problems.

(...)

The question that immediately comes to my mind before I start digging any further - how is the wiki making problems special? Is it just getting most of the traffic (it is the "most interesting" one) or is its configuration slightly different? Marcin Cieślak https://www.mediawiki.org/wiki/User:Saper _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

Jurij Byrda

4:35 p.m.

Tamstos preles geros bet man ne situ prekiu reikia paprasciausio klausai sustiprinti neprigirdinciam 2015 lap. 12 18:03 "Justin Lloyd" <jlloyd.wiki(a)gmail.com> rašė:

...

On 2015-11-12, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote:

* Six wikis are configured as Vhosts in Apache, load balanced by a

separate > set of front-end servers, where two of the wikis are for private

internal

use and the other four are public, though the traffic to one of the

public

wikis dwarfs the rest and it's the wiki giving me problems.

(...) > I'm mainly looking right now for how to troubleshoot the stuck

processes,

> but any advice regarding this architecture is also welcome, as I feel

could use some improvement but I'm not sure how just yet.

_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

Justin Lloyd

18 Nov 18 Nov

7:59 p.m.

Hey everyone, Yesterday I posted this to /r/mediawiki (https://redd.it/3t2apu) and cross-posted to /r/apache as well, but unfortunately I've still not received any feedback other than the one request here for clarification and a couple of suggestions on reddit that I'd already covered in the post. It's possible no one has any suggestions for me regarding this issue (it is a somewhat complex application stack that could be requiring configuration and/or tuning in multiple places, for example), but given how severe of a problem this is for my production sites, I wanted to bump it once in hopes of possibly getting at least some pointers of things to consider that I may not have already, especially with respect to diagnostics I could perform on the live web servers beyond just server-status and the collectd apache plugin (which is basically the same thing), for example. On Thu, Nov 12, 2015 at 8:02 AM, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote:

...

On 2015-11-12, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote:

* Six wikis are configured as Vhosts in Apache, load balanced by a

separate

set of front-end servers, where two of the wikis are for private

internal

use and the other four are public, though the traffic to one of the

public

wikis dwarfs the rest and it's the wiki giving me problems.

(...)

I'm mainly looking right now for how to troubleshoot the stuck

processes,

but any advice regarding this architecture is also welcome, as I feel it could use some improvement but I'm not sure how just yet.

Dave Humphrey

8:25 p.m.

My usual strategy is to check server-status and if I need more detail go with debugging tools (gdp etc..., see http://serverfault.com/questions/487530/find-out-what-high-cpu-usage-apache…). It seems you have done this, however, and I'm wondering why you haven't at least been able to narrow down the issue? You should at least be able to know which PHP file is locking up/crashing or the rough area/cause? Once you know roughly where it is you can add temporary PHP logging commands in the code to help narrow down the issue further. If you also know roughly where/how the lockups are you can try testing/replicating the behavior to get a bit more control on it. On 18 November 2015 at 14:59, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote:

...

most

active. There are some tiny configuration differences between the wikis (e.g. the value of $wgJobRunRate, the specific extensions loaded) but nothing very significant I don't believe. I should also add that all four of these wikis (we have a 5th, for 7 total, not 6 as I'd originally said) also use Semantic MediaWiki extensively. I believe the other three wikis would run into the same problem if they had same amount of traffic as the English one. However, since they all are vhosts within the same Apache instances, the English one's problems affect all of them. Justin On Thu, Nov 12, 2015 at 1:42 AM, Marcin Cieslak <saper(a)saper.info>

wrote:

> On 2015-11-12, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote: > > * Six wikis are configured as Vhosts in Apache, load balanced by a > separate > > set of front-end servers, where two of the wikis are for private > internal > > use and the other four are public, though the traffic to one of the > public > > wikis dwarfs the rest and it's the wiki giving me problems. > > (...) > > > I'm mainly looking right now for how to troubleshoot the stuck > processes, > > but any advice regarding this architecture is also welcome, as I feel

> > could use some improvement but I'm not sure how just yet. > > The question that immediately comes to my mind before I start digging > any further - how is the wiki making problems special? Is it just

getting

most of the traffic (it is the "most interesting" one) or is its configuration slightly different? Marcin Cieślak https://www.mediawiki.org/wiki/User:Saper _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

-- Dave Humphrey -- dave(a)uesp.net Founder/Server Admin of the Unofficial Elder Scrolls Pages -- www.uesp.net www.viud.net - Building the world's toughest USB drive

Justin Lloyd

8:42 p.m.

My speculation is that it's image heavy pages, not one specific php page. This is for the Guild Wars 2 wikis, specifically the English wiki at wiki.guildwars2.com. The Game Updates page used to be problematic, causing a massive backlog because a game update or hotfix was released and people hammered that page to see the list of changes. Our main editors changed how the page works, primarily breaking it up into subpages that DPL integrates the most recent of which into the main page, but also changing the templates that were used for displaying trait and skill icons. Further analysis of the Apache logs, after adding the %D field to the log format, showed a lot of pages taking sometimes minutes to complete, which ultimately result in 502s. The ones that appear to take the longest are those with a lot of these thumbnail images, which is why I think it's still a template issue, but it would be really nice to be able to back up that hypothesis with actual data from process diagnostics, stack traces, etc. (I really miss DTrace on Solaris. I know it exists for Linux but I'm wary of trying it, especially on production systems. Anyone here have experience with it?) On Wed, Nov 18, 2015 at 12:25 PM, Dave Humphrey <dave(a)uesp.net> wrote:

...

My usual strategy is to check server-status and if I need more detail go with debugging tools (gdp etc..., see http://serverfault.com/questions/487530/find-out-what-high-cpu-usage-apache… ). It seems you have done this, however, and I'm wondering why you haven't at least been able to narrow down the issue? You should at least be able to know which PHP file is locking up/crashing or the rough area/cause? Once you know roughly where it is you can add temporary PHP logging commands in the code to help narrow down the issue further. If you also know roughly where/how the lockups are you can try testing/replicating the behavior to get a bit more control on it. On 18 November 2015 at 14:59, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote:

and

a couple of suggestions on reddit that I'd already covered in the post. It's possible no one has any suggestions for me regarding this issue (it

a somewhat complex application stack that could be requiring

configuration

and/or tuning in multiple places, for example), but given how severe of a problem this is for my production sites, I wanted to bump it once in

hopes

of possibly getting at least some pointers of things to consider that I

may

not have already, especially with respect to diagnostics I could perform

the live web servers beyond just server-status and the collectd apache plugin (which is basically the same thing), for example. On Thu, Nov 12, 2015 at 8:02 AM, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote: > Marcin, > > It's the biggest and most heavily trafficked of our wikis because its

the

English-language version of the wiki. We also have German, French, and Spanish, but the English-speaking community is by far the largest and

most

wrote: > >> On 2015-11-12, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote: >> > * Six wikis are configured as Vhosts in Apache, load balanced by a >> separate >> > set of front-end servers, where two of the wikis are for private >> internal >> > use and the other four are public, though the traffic to one of the >> public >> > wikis dwarfs the rest and it's the wiki giving me problems. >> >> (...) >> >> > I'm mainly looking right now for how to troubleshoot the stuck >> processes, >> > but any advice regarding this architecture is also welcome, as I

feel

getting

_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

-- Dave Humphrey -- dave(a)uesp.net Founder/Server Admin of the Unofficial Elder Scrolls Pages -- www.uesp.net www.viud.net - Building the world's toughest USB drive _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

Dave Humphrey

8:55 p.m.

That actually sounds very close to an issue we had after upgrading to 1.22 earlier this year. Pages with a lot of images/thumbnails took a long time to render (100s of images took over a minute). We eventually tracked it down to having the default $wgTmpDirectory pointing to the upload/images directory which was on a NFS share. Each file creation (or access?) on a NFS share takes a fixed 50ms so you multiply that by multiple accesses and you get the delay. We fixed it by simply changing $wgTmpDirectory to point to a path on the local fixed drive. Since your setup sounds similar to ours it may be worth trying it out. If this is indeed your issue you can force a "slow" page load by purging a page with a lot of images on it. Test it before and after the change. On 18 November 2015 at 15:42, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote:

...

My usual strategy is to check server-status and if I need more detail go with debugging tools (gdp etc..., see

http://serverfault.com/questions/487530/find-out-what-high-cpu-usage-apache…

). It seems you have done this, however, and I'm wondering why you haven't

least been able to narrow down the issue? You should at least be able to know which PHP file is locking up/crashing or the rough area/cause? Once you know roughly where it is you can add temporary PHP logging commands in the code to help narrow down the issue further. If you also know roughly where/how the lockups are you can try testing/replicating

the

behavior to get a bit more control on it. On 18 November 2015 at 14:59, Justin Lloyd <jlloyd.wiki(a)gmail.com>

wrote:

and > a couple of suggestions on reddit that I'd already covered in the post. > > It's possible no one has any suggestions for me regarding this issue

(it

a somewhat complex application stack that could be requiring

configuration > and/or tuning in multiple places, for example), but given how severe

of a

problem this is for my production sites, I wanted to bump it once in

hopes

of possibly getting at least some pointers of things to consider that I

may > not have already, especially with respect to diagnostics I could

perform

the > > English-language version of the wiki. We also have German, French,

and

> > Spanish, but the English-speaking community is by far the largest and > most > > active. There are some tiny configuration differences between the

wikis

> > (e.g. the value of $wgJobRunRate, the specific extensions loaded) but > > nothing very significant I don't believe. > > > > I should also add that all four of these wikis (we have a 5th, for 7 > > total, not 6 as I'd originally said) also use Semantic MediaWiki > > extensively. I believe the other three wikis would run into the same > > problem if they had same amount of traffic as the English one.

However,

> > since they all are vhosts within the same Apache instances, the

English

> > one's problems affect all of them. > > > > Justin > > > > > > On Thu, Nov 12, 2015 at 1:42 AM, Marcin Cieslak <saper(a)saper.info> > wrote: > > > >> On 2015-11-12, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote: > >> > * Six wikis are configured as Vhosts in Apache, load balanced by a > >> separate > >> > set of front-end servers, where two of the wikis are for private > >> internal > >> > use and the other four are public, though the traffic to one of

the

>> public >> > wikis dwarfs the rest and it's the wiki giving me problems. >> >> (...) >> >> > I'm mainly looking right now for how to troubleshoot the stuck >> processes, >> > but any advice regarding this architecture is also welcome, as I

feel > it > >> > could use some improvement but I'm not sure how just yet. > >> > >> The question that immediately comes to my mind before I start

digging

> any further - how is the wiki making problems special? Is it just

getting

> most of the traffic (it is the "most interesting" one) or is its > configuration slightly different? > > Marcin Cieślak > https://www.mediawiki.org/wiki/User:Saper > > > _______________________________________________ > MediaWiki-l mailing list > To unsubscribe, go to: > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l >

_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

-- Dave Humphrey -- dave(a)uesp.net Founder/Server Admin of the Unofficial Elder Scrolls Pages --

www.uesp.net

www.viud.net - Building the world's toughest USB drive _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

-- Dave Humphrey -- dave(a)uesp.net Founder/Server Admin of the Unofficial Elder Scrolls Pages -- www.uesp.net www.viud.net - Building the world's toughest USB drive

Justin Lloyd

8:57 p.m.

Intriguing! I'll definitely investigate this and report back. Thanks! :) Justin On Wed, Nov 18, 2015 at 12:55 PM, Dave Humphrey <dave(a)uesp.net> wrote:

...

causing

a massive backlog because a game update or hotfix was released and people hammered that page to see the list of changes. Our main editors changed

how

the page works, primarily breaking it up into subpages that DPL

integrates

the most recent of which into the main page, but also changing the templates that were used for displaying trait and skill icons. Further analysis of the Apache logs, after adding the %D field to the log format, showed a lot of pages taking sometimes minutes to complete, which ultimately result in 502s. The ones that appear to take the longest are those with a lot of these thumbnail images, which is why I think it's

still

a template issue, but it would be really nice to be able to back up that hypothesis with actual data from process diagnostics, stack traces, etc. (I really miss DTrace on Solaris. I know it exists for Linux but I'm wary of trying it, especially on production systems. Anyone here have

experience

with it?) On Wed, Nov 18, 2015 at 12:25 PM, Dave Humphrey <dave(a)uesp.net> wrote: > My usual strategy is to check server-status and if I need more detail

with debugging tools (gdp etc..., see

http://serverfault.com/questions/487530/find-out-what-high-cpu-usage-apache…

). It seems you have done this, however, and I'm wondering why you haven't

at > least been able to narrow down the issue? You should at least be able

know which PHP file is locking up/crashing or the rough area/cause? Once you know roughly where it is you can add temporary PHP logging commands in the code to help narrow down the issue further. If you also know roughly where/how the lockups are you can try testing/replicating

the

behavior to get a bit more control on it. On 18 November 2015 at 14:59, Justin Lloyd <jlloyd.wiki(a)gmail.com>

wrote: > > > Hey everyone, > > > > Yesterday I posted this to /r/mediawiki (https://redd.it/3t2apu) and > > cross-posted to /r/apache as well, but unfortunately I've still not > > received any feedback other than the one request here for

clarification

> and > > a couple of suggestions on reddit that I'd already covered in the

post.

> > It's possible no one has any suggestions for me regarding this issue

(it

a somewhat complex application stack that could be requiring

configuration > and/or tuning in multiple places, for example), but given how severe

of a > > problem this is for my production sites, I wanted to bump it once in > hopes > > of possibly getting at least some pointers of things to consider

that I

may > not have already, especially with respect to diagnostics I could

perform > on > > the live web servers beyond just server-status and the collectd

apache

> plugin (which is basically the same thing), for example. > > > > On Thu, Nov 12, 2015 at 8:02 AM, Justin Lloyd <jlloyd.wiki(a)gmail.com

> > wrote: > > > > > Marcin, > > > > > > It's the biggest and most heavily trafficked of our wikis because

its

the > > English-language version of the wiki. We also have German, French,

and > > > Spanish, but the English-speaking community is by far the largest

and

> most > > active. There are some tiny configuration differences between the

wikis > > > (e.g. the value of $wgJobRunRate, the specific extensions loaded)

but

> > > nothing very significant I don't believe. > > > > > > I should also add that all four of these wikis (we have a 5th, for

> > > total, not 6 as I'd originally said) also use Semantic MediaWiki > > > extensively. I believe the other three wikis would run into the

same

> > problem if they had same amount of traffic as the English one.

However,

> > since they all are vhosts within the same Apache instances, the

English > > > one's problems affect all of them. > > > > > > Justin > > > > > > > > > On Thu, Nov 12, 2015 at 1:42 AM, Marcin Cieslak <saper(a)saper.info> > > wrote: > > > > > >> On 2015-11-12, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote: > > >> > * Six wikis are configured as Vhosts in Apache, load balanced

by a

> >> separate > >> > set of front-end servers, where two of the wikis are for private > >> internal > >> > use and the other four are public, though the traffic to one of

the

feel > it > >> > could use some improvement but I'm not sure how just yet. > >> > >> The question that immediately comes to my mind before I start

digging

>> any further - how is the wiki making problems special? Is it just getting >> most of the traffic (it is the "most interesting" one) or is its >> configuration slightly different? >> >> Marcin Cieślak >> https://www.mediawiki.org/wiki/User:Saper >> >> >> _______________________________________________ >> MediaWiki-l mailing list >> To unsubscribe, go to: >> https://lists.wikimedia.org/mailman/listinfo/mediawiki-l >> > > _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

-- Dave Humphrey -- dave(a)uesp.net Founder/Server Admin of the Unofficial Elder Scrolls Pages --

www.uesp.net

_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

Dave Humphrey

9 p.m.

Another note if that doesn't happen to work for you: We discovered the source of our issue by enabling profiling/debugging on the wiki (on a non-public server/install so you can control the page loads and profiling outputs). You can see pretty quickly what areas/code are taking the longest and begin to dig down. Eventually I added custom profile sections to further narrow down the issue to a single "open" call. On 18 November 2015 at 15:57, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote:

...

Intriguing! I'll definitely investigate this and report back. Thanks! :) Justin On Wed, Nov 18, 2015 at 12:55 PM, Dave Humphrey <dave(a)uesp.net> wrote:

That actually sounds very close to an issue we had after upgrading to

1.22

earlier this year. Pages with a lot of images/thumbnails took a long time to render (100s of images took over a minute). We eventually tracked it down to having the default $wgTmpDirectory pointing to the upload/images directory which was on a NFS share. Each file creation (or access?) on a NFS share takes a fixed 50ms so you multiply that by multiple accesses

and

you get the delay. We fixed it by simply changing $wgTmpDirectory to point to a path on the local fixed drive. Since your setup sounds similar to ours it may be

worth

trying it out. If this is indeed your issue you can force a "slow" page load by purging a page with a lot of images on it. Test it before and

after

the change. On 18 November 2015 at 15:42, Justin Lloyd <jlloyd.wiki(a)gmail.com>

wrote:

> My speculation is that it's image heavy pages, not one specific php

page.

This is for the Guild Wars 2 wikis, specifically the English wiki at wiki.guildwars2.com. The Game Updates page used to be problematic,

causing > a massive backlog because a game update or hotfix was released and

people

hammered that page to see the list of changes. Our main editors changed

how

the page works, primarily breaking it up into subpages that DPL

integrates > the most recent of which into the main page, but also changing the > templates that were used for displaying trait and skill icons. > > Further analysis of the Apache logs, after adding the %D field to the

log

> format, showed a lot of pages taking sometimes minutes to complete,

which

ultimately result in 502s. The ones that appear to take the longest are those with a lot of these thumbnail images, which is why I think it's

still > a template issue, but it would be really nice to be able to back up

that

> hypothesis with actual data from process diagnostics, stack traces,

etc.

> > (I really miss DTrace on Solaris. I know it exists for Linux but I'm

wary

of trying it, especially on production systems. Anyone here have

experience

with it?) On Wed, Nov 18, 2015 at 12:25 PM, Dave Humphrey <dave(a)uesp.net> wrote: > My usual strategy is to check server-status and if I need more detail

with debugging tools (gdp etc..., see

http://serverfault.com/questions/487530/find-out-what-high-cpu-usage-apache…

> > ). > > It seems you have done this, however, and I'm wondering why you

haven't

at > least been able to narrow down the issue? You should at least be able

to > > know which PHP file is locking up/crashing or the rough area/cause? > > > > Once you know roughly where it is you can add temporary PHP logging > > commands in the code to help narrow down the issue further. If you

also

> > know roughly where/how the lockups are you can try

testing/replicating

> the > > behavior to get a bit more control on it. > > > > On 18 November 2015 at 14:59, Justin Lloyd <jlloyd.wiki(a)gmail.com> > wrote: > > > > > Hey everyone, > > > > > > Yesterday I posted this to /r/mediawiki (https://redd.it/3t2apu)

and

> > cross-posted to /r/apache as well, but unfortunately I've still not > > received any feedback other than the one request here for

clarification

> and > > a couple of suggestions on reddit that I'd already covered in the

post. > > > > > > It's possible no one has any suggestions for me regarding this

issue

> (it > > is > > > a somewhat complex application stack that could be requiring > > configuration > > > and/or tuning in multiple places, for example), but given how

severe

> of a > > > problem this is for my production sites, I wanted to bump it once

> hopes > > of possibly getting at least some pointers of things to consider

that I

may > not have already, especially with respect to diagnostics I could

perform > on > > the live web servers beyond just server-status and the collectd

apache > > > plugin (which is basically the same thing), for example. > > > > > > > > > > > > On Thu, Nov 12, 2015 at 8:02 AM, Justin Lloyd <

jlloyd.wiki(a)gmail.com

> > wrote: > > > > > Marcin, > > > > > > It's the biggest and most heavily trafficked of our wikis because

its > > the > > > > English-language version of the wiki. We also have German,

French,

and > > > Spanish, but the English-speaking community is by far the largest

and

> most > > active. There are some tiny configuration differences between the

wikis > > > (e.g. the value of $wgJobRunRate, the specific extensions loaded)

but > > > > nothing very significant I don't believe. > > > > > > > > I should also add that all four of these wikis (we have a 5th,

for

> > > total, not 6 as I'd originally said) also use Semantic MediaWiki > > > extensively. I believe the other three wikis would run into the

same > > > > problem if they had same amount of traffic as the English one. > However, > > > > since they all are vhosts within the same Apache instances, the > English > > > > one's problems affect all of them. > > > > > > > > Justin > > > > > > > > > > > > On Thu, Nov 12, 2015 at 1:42 AM, Marcin Cieslak <

saper(a)saper.info>

> > wrote: > > > > > >> On 2015-11-12, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote: > > >> > * Six wikis are configured as Vhosts in Apache, load balanced

by a > > > >> separate > > > >> > set of front-end servers, where two of the wikis are for

private

> > > >> internal > > > >> > use and the other four are public, though the traffic to one

> the > > > >> public > > > >> > wikis dwarfs the rest and it's the wiki giving me problems. > > > >> > > > >> (...) > > > >> > > > >> > I'm mainly looking right now for how to troubleshoot the stuck > > > >> processes, > > > >> > but any advice regarding this architecture is also welcome,

as I

> > feel > > > it > > > >> > could use some improvement but I'm not sure how just yet. > > > >> > > > >> The question that immediately comes to my mind before I start > digging > > > >> any further - how is the wiki making problems special? Is it

just

> getting > >> most of the traffic (it is the "most interesting" one) or is its > >> configuration slightly different? > >> > >> Marcin Cieślak > >> https://www.mediawiki.org/wiki/User:Saper > >> > >> > >> _______________________________________________ > >> MediaWiki-l mailing list > >> To unsubscribe, go to: > >> https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > >> > > > > > _______________________________________________ > MediaWiki-l mailing list > To unsubscribe, go to: > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > -- Dave Humphrey -- dave(a)uesp.net Founder/Server Admin of the Unofficial Elder Scrolls Pages --

www.uesp.net

_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

-- Dave Humphrey -- dave(a)uesp.net Founder/Server Admin of the Unofficial Elder Scrolls Pages --

www.uesp.net

_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

-- Dave Humphrey -- dave(a)uesp.net Founder/Server Admin of the Unofficial Elder Scrolls Pages -- www.uesp.net www.viud.net - Building the world's toughest USB drive

Justin Lloyd

19 Nov 19 Nov

5:52 p.m.

So I confirmed that $wgTmpDirectory is defaulting to /tmp on my systems, so that's not the problem. As for profiling, I enabled it on one of the four web servers and that appears to actually trigger the problem, or a very similar one. The Apache processes quickly climbed to their MaxClients limit of 100 and just stayed there, forcing me to restart Apache after first commenting out the profiling settings in LocalSettings.php, where $wgProfileLimit was set to 2. On Wed, Nov 18, 2015 at 1:00 PM, Dave Humphrey <dave(a)uesp.net> wrote:

...

Intriguing! I'll definitely investigate this and report back. Thanks! :) Justin On Wed, Nov 18, 2015 at 12:55 PM, Dave Humphrey <dave(a)uesp.net> wrote:

That actually sounds very close to an issue we had after upgrading to

1.22 > earlier this year. Pages with a lot of images/thumbnails took a long

time

> to render (100s of images took over a minute). We eventually tracked it > down to having the default $wgTmpDirectory pointing to the

upload/images

> directory which was on a NFS share. Each file creation (or access?) on

NFS share takes a fixed 50ms so you multiply that by multiple accesses

and > you get the delay. > > We fixed it by simply changing $wgTmpDirectory to point to a path on

the

local fixed drive. Since your setup sounds similar to ours it may be

worth

trying it out. If this is indeed your issue you can force a "slow" page load by purging a page with a lot of images on it. Test it before and

after

the change. On 18 November 2015 at 15:42, Justin Lloyd <jlloyd.wiki(a)gmail.com>

wrote:

> My speculation is that it's image heavy pages, not one specific php

page.

This is for the Guild Wars 2 wikis, specifically the English wiki at wiki.guildwars2.com. The Game Updates page used to be problematic,

causing > a massive backlog because a game update or hotfix was released and

people > > hammered that page to see the list of changes. Our main editors

changed

how

the page works, primarily breaking it up into subpages that DPL

log

> format, showed a lot of pages taking sometimes minutes to complete,

which > > ultimately result in 502s. The ones that appear to take the longest

are

those with a lot of these thumbnail images, which is why I think it's

still > a template issue, but it would be really nice to be able to back up

that

> hypothesis with actual data from process diagnostics, stack traces,

etc.

> > (I really miss DTrace on Solaris. I know it exists for Linux but I'm

wary > > of trying it, especially on production systems. Anyone here have > experience > > with it?) > > > > > > On Wed, Nov 18, 2015 at 12:25 PM, Dave Humphrey <dave(a)uesp.net>

wrote:

> > > > > My usual strategy is to check server-status and if I need more

detail

> with debugging tools (gdp etc..., see > >

http://serverfault.com/questions/487530/find-out-what-high-cpu-usage-apache…

> > ). > > It seems you have done this, however, and I'm wondering why you

haven't > > at > > > least been able to narrow down the issue? You should at least be

able

also

> > know roughly where/how the lockups are you can try

testing/replicating

and > > > > cross-posted to /r/apache as well, but unfortunately I've still

not

> > received any feedback other than the one request here for

clarification

> and > > a couple of suggestions on reddit that I'd already covered in the

post. > > > > > > It's possible no one has any suggestions for me regarding this

issue

> (it > > is > > > a somewhat complex application stack that could be requiring > > configuration > > > and/or tuning in multiple places, for example), but given how

severe

> of a > > > problem this is for my production sites, I wanted to bump it once

> hopes > > of possibly getting at least some pointers of things to consider

that I

> may > > not have already, especially with respect to diagnostics I could perform > on > > the live web servers beyond just server-status and the collectd

apache > > > plugin (which is basically the same thing), for example. > > > > > > > > > > > > On Thu, Nov 12, 2015 at 8:02 AM, Justin Lloyd <

jlloyd.wiki(a)gmail.com > > > > > > wrote: > > > > > > > > > Marcin, > > > > > > > > > > It's the biggest and most heavily trafficked of our wikis

because

its > > the > > > > English-language version of the wiki. We also have German,

French, > > and > > > > > Spanish, but the English-speaking community is by far the

largest

> and > > > > most > > > > > active. There are some tiny configuration differences between

the

> > wikis > > > > > (e.g. the value of $wgJobRunRate, the specific extensions

loaded)

but > > > > nothing very significant I don't believe. > > > > > > > > I should also add that all four of these wikis (we have a 5th,

for > 7 > > > > > total, not 6 as I'd originally said) also use Semantic

MediaWiki

> > > extensively. I believe the other three wikis would run into the

saper(a)saper.info> > > > > wrote: > > > > > > > > > >> On 2015-11-12, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote: > > > > >> > * Six wikis are configured as Vhosts in Apache, load

balanced

by a > > > >> separate > > > >> > set of front-end servers, where two of the wikis are for

private

> > > >> internal > > > >> > use and the other four are public, though the traffic to one

of > > the > > > > >> public > > > > >> > wikis dwarfs the rest and it's the wiki giving me problems. > > > > >> > > > > >> (...) > > > > >> > > > > >> > I'm mainly looking right now for how to troubleshoot the

stuck

> > > >> processes, > > > >> > but any advice regarding this architecture is also welcome,

as I

just > > > > getting > > > > >> most of the traffic (it is the "most interesting" one) or is

its

> > >> configuration slightly different? > > >> > > >> Marcin Cieślak > > >> https://www.mediawiki.org/wiki/User:Saper > > >> > > >> > > >> _______________________________________________ > > >> MediaWiki-l mailing list > > >> To unsubscribe, go to: > > >> https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > >> > > > > > > > > _______________________________________________ > > MediaWiki-l mailing list > > To unsubscribe, go to: > > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > > > > > -- > Dave Humphrey -- dave(a)uesp.net > Founder/Server Admin of the Unofficial Elder Scrolls Pages -- www.uesp.net

_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

-- Dave Humphrey -- dave(a)uesp.net Founder/Server Admin of the Unofficial Elder Scrolls Pages --

www.uesp.net

_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

Justin Lloyd

6:03 p.m.

I'm wondering if the RunJobs requests might be related to the problem. The settings for this particular wiki have been the same for some time ($wgJobRunRate = 0.1), but the problem didn't start until the day we launched our expansion, which led to a significant increase in the amount of editing of the wiki, which makes extensive use of templates, DPL, and SMW. Looking at just yesterday's access logs, we'll see 10 to 60 RunJobs messages per minute, or around 1400 per hour. On Thu, Nov 19, 2015 at 9:52 AM, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote:

...

Intriguing! I'll definitely investigate this and report back. Thanks! :) Justin On Wed, Nov 18, 2015 at 12:55 PM, Dave Humphrey <dave(a)uesp.net> wrote:

That actually sounds very close to an issue we had after upgrading to

1.22 > earlier this year. Pages with a lot of images/thumbnails took a long

time

> to render (100s of images took over a minute). We eventually tracked

> down to having the default $wgTmpDirectory pointing to the

upload/images

> directory which was on a NFS share. Each file creation (or access?)

on a

NFS share takes a fixed 50ms so you multiply that by multiple accesses

and > you get the delay. > > We fixed it by simply changing $wgTmpDirectory to point to a path on

the

local fixed drive. Since your setup sounds similar to ours it may be

worth > trying it out. If this is indeed your issue you can force a "slow"

page

load by purging a page with a lot of images on it. Test it before and

after

the change. On 18 November 2015 at 15:42, Justin Lloyd <jlloyd.wiki(a)gmail.com>

wrote:

> My speculation is that it's image heavy pages, not one specific php

page.

> This is for the Guild Wars 2 wikis, specifically the English wiki at > wiki.guildwars2.com. The Game Updates page used to be problematic, causing > a massive backlog because a game update or hotfix was released and

people > > hammered that page to see the list of changes. Our main editors

changed

> how > > the page works, primarily breaking it up into subpages that DPL > integrates > > the most recent of which into the main page, but also changing the > > templates that were used for displaying trait and skill icons. > > > > Further analysis of the Apache logs, after adding the %D field to

the

log

> format, showed a lot of pages taking sometimes minutes to complete,

which > > ultimately result in 502s. The ones that appear to take the longest

are

> > those with a lot of these thumbnail images, which is why I think

it's

still > a template issue, but it would be really nice to be able to back up

that

> hypothesis with actual data from process diagnostics, stack traces,

etc.

> > (I really miss DTrace on Solaris. I know it exists for Linux but I'm

wary > > of trying it, especially on production systems. Anyone here have > experience > > with it?) > > > > > > On Wed, Nov 18, 2015 at 12:25 PM, Dave Humphrey <dave(a)uesp.net>

wrote:

> > > > > My usual strategy is to check server-status and if I need more

detail

go > > with debugging tools (gdp etc..., see > > > > >

http://serverfault.com/questions/487530/find-out-what-high-cpu-usage-apache…

> > ). > > It seems you have done this, however, and I'm wondering why you

haven't > > at > > > least been able to narrow down the issue? You should at least be

able

> to > > > know which PHP file is locking up/crashing or the rough

area/cause?

> > > > > > Once you know roughly where it is you can add temporary PHP

logging

> > commands in the code to help narrow down the issue further. If you

also

> > know roughly where/how the lockups are you can try

testing/replicating

> the > > behavior to get a bit more control on it. > > > > On 18 November 2015 at 14:59, Justin Lloyd <jlloyd.wiki(a)gmail.com

> > wrote: > > > > > > > Hey everyone, > > > > > > > > Yesterday I posted this to /r/mediawiki (https://redd.it/3t2apu

)

and > > > > cross-posted to /r/apache as well, but unfortunately I've still

not

> > > > received any feedback other than the one request here for > clarification > > > and > > > > a couple of suggestions on reddit that I'd already covered in

the

post. > > > > > > It's possible no one has any suggestions for me regarding this

issue

> (it > > is > > > a somewhat complex application stack that could be requiring > > configuration > > > and/or tuning in multiple places, for example), but given how

severe > > of a > > > > problem this is for my production sites, I wanted to bump it

once

> > hopes > > > of possibly getting at least some pointers of things to consider that I > > may > > > not have already, especially with respect to diagnostics I could > perform > > on > > > the live web servers beyond just server-status and the collectd apache > > > plugin (which is basically the same thing), for example. > > > > > > > > > > > > On Thu, Nov 12, 2015 at 8:02 AM, Justin Lloyd <

jlloyd.wiki(a)gmail.com > > > > > > wrote: > > > > > > > > > Marcin, > > > > > > > > > > It's the biggest and most heavily trafficked of our wikis

because

its > > the > > > > English-language version of the wiki. We also have German,

French, > > and > > > > > Spanish, but the English-speaking community is by far the

largest

> and > > > > most > > > > > active. There are some tiny configuration differences between

the

> > wikis > > > > > (e.g. the value of $wgJobRunRate, the specific extensions

loaded)

but > > > > nothing very significant I don't believe. > > > > > > > > I should also add that all four of these wikis (we have a 5th,

for > 7 > > > > > total, not 6 as I'd originally said) also use Semantic

MediaWiki

> > > > > extensively. I believe the other three wikis would run into

the

> same > > > > > problem if they had same amount of traffic as the English one. > > However, > > > > > since they all are vhosts within the same Apache instances,

the

> English > > > > one's problems affect all of them. > > > > > > > > Justin > > > > > > > > > > > > On Thu, Nov 12, 2015 at 1:42 AM, Marcin Cieslak <

saper(a)saper.info> > > > > wrote: > > > > > > > > > >> On 2015-11-12, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote: > > > > >> > * Six wikis are configured as Vhosts in Apache, load

balanced

by a > > > >> separate > > > >> > set of front-end servers, where two of the wikis are for

private > > > > >> internal > > > > >> > use and the other four are public, though the traffic to

one

stuck

> > > >> processes, > > > >> > but any advice regarding this architecture is also welcome,

as I

just > > > > getting > > > > >> most of the traffic (it is the "most interesting" one) or is

its

> > > >> configuration slightly different? > > > >> > > > >> Marcin Cieślak > > > >> https://www.mediawiki.org/wiki/User:Saper > > > >> > > > >> > > > >> _______________________________________________ > > > >> MediaWiki-l mailing list > > > >> To unsubscribe, go to: > > > >> https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > > >> > > > > > > > > > > > _______________________________________________ > > > MediaWiki-l mailing list > > > To unsubscribe, go to: > > > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > > > > > > > > > > -- > > Dave Humphrey -- dave(a)uesp.net > > Founder/Server Admin of the Unofficial Elder Scrolls Pages -- > www.uesp.net > > www.viud.net - Building the world's toughest USB drive > > _______________________________________________ > > MediaWiki-l mailing list > > To unsubscribe, go to: > > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > > _______________________________________________ > MediaWiki-l mailing list > To unsubscribe, go to: > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > -- Dave Humphrey -- dave(a)uesp.net Founder/Server Admin of the Unofficial Elder Scrolls Pages --

www.uesp.net

_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

Dave Humphrey

6:11 p.m.

Don't enable debug/profiling on a server that gets public hits, at least in general. Profiling slows down the page load and if you're already borderline it will make a bad situation worse. That and you'll generate so much profiling output it will be hard to go through and make anything meaningful from it. Put it on a unused server or copy the wiki installation to a new folder and do it from there. Then you can control the page loads being profiled. On 19 November 2015 at 12:52, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote:

...

longest

and begin to dig down. Eventually I added custom profile sections to further narrow down the issue to a single "open" call. On 18 November 2015 at 15:57, Justin Lloyd <jlloyd.wiki(a)gmail.com>

wrote:

> Intriguing! I'll definitely investigate this and report back. Thanks!

Justin On Wed, Nov 18, 2015 at 12:55 PM, Dave Humphrey <dave(a)uesp.net> wrote:

That actually sounds very close to an issue we had after upgrading to

1.22 > earlier this year. Pages with a lot of images/thumbnails took a long

time > > to render (100s of images took over a minute). We eventually tracked

> down to having the default $wgTmpDirectory pointing to the

upload/images > > directory which was on a NFS share. Each file creation (or access?)

a > > NFS share takes a fixed 50ms so you multiply that by multiple

accesses

and > you get the delay. > > We fixed it by simply changing $wgTmpDirectory to point to a path on

the > > local fixed drive. Since your setup sounds similar to ours it may be > worth > > trying it out. If this is indeed your issue you can force a "slow"

page

> > load by purging a page with a lot of images on it. Test it before and > after > > the change. > > > > On 18 November 2015 at 15:42, Justin Lloyd <jlloyd.wiki(a)gmail.com> > wrote: > > > > > My speculation is that it's image heavy pages, not one specific php > page. > > > This is for the Guild Wars 2 wikis, specifically the English wiki

> wiki.guildwars2.com. The Game Updates page used to be problematic, causing > a massive backlog because a game update or hotfix was released and

people > > hammered that page to see the list of changes. Our main editors

changed > > how > > > the page works, primarily breaking it up into subpages that DPL > > integrates > > > the most recent of which into the main page, but also changing the > > > templates that were used for displaying trait and skill icons. > > > > > > Further analysis of the Apache logs, after adding the %D field to

the

log

> format, showed a lot of pages taking sometimes minutes to complete,

which > > ultimately result in 502s. The ones that appear to take the longest

are > > > those with a lot of these thumbnail images, which is why I think

it's

> > still > > > a template issue, but it would be really nice to be able to back up > that > > > hypothesis with actual data from process diagnostics, stack traces, > etc. > > > > > > (I really miss DTrace on Solaris. I know it exists for Linux but

I'm

wary > > of trying it, especially on production systems. Anyone here have > experience > > with it?) > > > > > > On Wed, Nov 18, 2015 at 12:25 PM, Dave Humphrey <dave(a)uesp.net>

wrote:

> > > > > My usual strategy is to check server-status and if I need more

detail

go > > with debugging tools (gdp etc..., see > > > > >

http://serverfault.com/questions/487530/find-out-what-high-cpu-usage-apache…

> > ). > > It seems you have done this, however, and I'm wondering why you

haven't > > at > > > least been able to narrow down the issue? You should at least be

able > > to > > > > know which PHP file is locking up/crashing or the rough

area/cause?

> > > > > > > > Once you know roughly where it is you can add temporary PHP

logging

> > > > commands in the code to help narrow down the issue further. If

you

> also > > > > know roughly where/how the lockups are you can try > testing/replicating > > > the > > > > behavior to get a bit more control on it. > > > > > > > > On 18 November 2015 at 14:59, Justin Lloyd <

jlloyd.wiki(a)gmail.com>

> > > wrote: > > > > > > > > > Hey everyone, > > > > > > > > > > Yesterday I posted this to /r/mediawiki (

https://redd.it/3t2apu)

and > > > > cross-posted to /r/apache as well, but unfortunately I've still

not > > > > > received any feedback other than the one request here for > > clarification > > > > and > > > > > a couple of suggestions on reddit that I'd already covered in

the

> > post. > > > > > > > > > > It's possible no one has any suggestions for me regarding this > issue > > > (it > > > > is > > > > > a somewhat complex application stack that could be requiring > > > > configuration > > > > > and/or tuning in multiple places, for example), but given how > severe > > > of a > > > > > problem this is for my production sites, I wanted to bump it

once

> in > > > > hopes > > > > > of possibly getting at least some pointers of things to

consider

> > that I > > > > may > > > > > not have already, especially with respect to diagnostics I

could

> perform > > on > > > the live web servers beyond just server-status and the collectd apache > > > plugin (which is basically the same thing), for example. > > > > > > > > > > > > On Thu, Nov 12, 2015 at 8:02 AM, Justin Lloyd <

jlloyd.wiki(a)gmail.com > > > > > > wrote: > > > > > > > > > Marcin, > > > > > > > > > > It's the biggest and most heavily trafficked of our wikis

because

its > > the > > > > English-language version of the wiki. We also have German,

French, > > and > > > > > Spanish, but the English-speaking community is by far the

largest

> and > > > > most > > > > > active. There are some tiny configuration differences between

the

> > wikis > > > > > (e.g. the value of $wgJobRunRate, the specific extensions

loaded) > > but > > > > > > nothing very significant I don't believe. > > > > > > > > > > > > I should also add that all four of these wikis (we have a

5th,

for > 7 > > > > > total, not 6 as I'd originally said) also use Semantic

MediaWiki > > > > > > extensively. I believe the other three wikis would run into

the

> > same > > > > > > problem if they had same amount of traffic as the English

one.

> > > However, > > > > > > since they all are vhosts within the same Apache instances,

the

> English > > > > one's problems affect all of them. > > > > > > > > Justin > > > > > > > > > > > > On Thu, Nov 12, 2015 at 1:42 AM, Marcin Cieslak <

saper(a)saper.info> > > > > wrote: > > > > > > > > > >> On 2015-11-12, Justin Lloyd <jlloyd.wiki(a)gmail.com> wrote: > > > > >> > * Six wikis are configured as Vhosts in Apache, load

balanced > > by a > > > > > >> separate > > > > > >> > set of front-end servers, where two of the wikis are for > private > > > > > >> internal > > > > > >> > use and the other four are public, though the traffic to

one

> of > > > the > > > > > >> public > > > > > >> > wikis dwarfs the rest and it's the wiki giving me

problems.

> > > > >> > > > > >> (...) > > > > >> > > > > >> > I'm mainly looking right now for how to troubleshoot the

stuck > > > > > >> processes, > > > > > >> > but any advice regarding this architecture is also

welcome,

> as I > > > > feel > > > > > it > > > > > >> > could use some improvement but I'm not sure how just yet. > > > > > >> > > > > > >> The question that immediately comes to my mind before I

start

> digging > > > >> any further - how is the wiki making problems special? Is it

just > > > > getting > > > > >> most of the traffic (it is the "most interesting" one) or is

its

www.uesp.net

_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

-- Dave Humphrey -- dave(a)uesp.net Founder/Server Admin of the Unofficial Elder Scrolls Pages --

www.uesp.net

_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

-- Dave Humphrey -- dave(a)uesp.net Founder/Server Admin of the Unofficial Elder Scrolls Pages -- www.uesp.net www.viud.net - Building the world's toughest USB drive

Justin Lloyd

6:33 p.m.

The problem only occurs on the production systems, I'm not able to replicate it on my test environment, and beyond that, my test environment has already been upgraded to MW 1.25 while production is still 1.24. That said, even under normal conditions I get pages that take so long to load that they lead to timeouts. I'm trying to analyze which pages or types of pages take so long; they tend to be ones that are very image heavy, typically pages for crafting materials and recipes, e.g. http://wiki.guildwars2.com/wiki/Bolt_of_Cotton. That page will usually load but a similar one that won't is http://wiki.guildwars2.com/wiki/Bottle_of_elonian_wine. On Thu, Nov 19, 2015 at 10:11 AM, Dave Humphrey <dave(a)uesp.net> wrote:

...

So I confirmed that $wgTmpDirectory is defaulting to /tmp on my systems,

that's not the problem. As for profiling, I enabled it on one of the four web servers and that appears to actually trigger the problem, or a very similar one. The Apache processes quickly climbed to their MaxClients

limit

of 100 and just stayed there, forcing me to restart Apache after first commenting out the profiling settings in LocalSettings.php, where $wgProfileLimit was set to 2. On Wed, Nov 18, 2015 at 1:00 PM, Dave Humphrey <dave(a)uesp.net> wrote: > Another note if that doesn't happen to work for you: We discovered the > source of our issue by enabling profiling/debugging on the wiki (on a > non-public server/install so you can control the page loads and

profiling

outputs). You can see pretty quickly what areas/code are taking the

longest

and begin to dig down. Eventually I added custom profile sections to further narrow down the issue to a single "open" call. On 18 November 2015 at 15:57, Justin Lloyd <jlloyd.wiki(a)gmail.com>

wrote:

> Intriguing! I'll definitely investigate this and report back. Thanks!

:) > > > > Justin > > > > > > On Wed, Nov 18, 2015 at 12:55 PM, Dave Humphrey <dave(a)uesp.net>

wrote:

> > > > > That actually sounds very close to an issue we had after upgrading

> > 1.22 > > > earlier this year. Pages with a lot of images/thumbnails took a

long

> time > > > to render (100s of images took over a minute). We eventually

tracked

> down to having the default $wgTmpDirectory pointing to the

upload/images > > directory which was on a NFS share. Each file creation (or access?)

a > > NFS share takes a fixed 50ms so you multiply that by multiple

accesses > > and > > > you get the delay. > > > > > > We fixed it by simply changing $wgTmpDirectory to point to a path

> the > > > local fixed drive. Since your setup sounds similar to ours it may

> worth > > trying it out. If this is indeed your issue you can force a "slow"

page > > > load by purging a page with a lot of images on it. Test it before

and

> > after > > > the change. > > > > > > On 18 November 2015 at 15:42, Justin Lloyd <jlloyd.wiki(a)gmail.com> > > wrote: > > > > > > > My speculation is that it's image heavy pages, not one specific

php

> page. > > > This is for the Guild Wars 2 wikis, specifically the English wiki

at > > > > wiki.guildwars2.com. The Game Updates page used to be

problematic,

> > > causing > > > > a massive backlog because a game update or hotfix was released

and

> > people > > > > hammered that page to see the list of changes. Our main editors > changed > > > how > > > > the page works, primarily breaking it up into subpages that DPL > > > integrates > > > > the most recent of which into the main page, but also changing

the

> > > templates that were used for displaying trait and skill icons. > > > > > > Further analysis of the Apache logs, after adding the %D field to

the > > log > > > > format, showed a lot of pages taking sometimes minutes to

complete,

> > which > > > > ultimately result in 502s. The ones that appear to take the

longest

are > > > those with a lot of these thumbnail images, which is why I think

it's > > > still > > > > a template issue, but it would be really nice to be able to back

> > that > > > > hypothesis with actual data from process diagnostics, stack

traces,

> etc. > > > > > > (I really miss DTrace on Solaris. I know it exists for Linux but

I'm

wary > > of trying it, especially on production systems. Anyone here have > experience > > with it?) > > > > > > On Wed, Nov 18, 2015 at 12:25 PM, Dave Humphrey <dave(a)uesp.net>

wrote:

> > > > > My usual strategy is to check server-status and if I need more

detail

> go > > > with debugging tools (gdp etc..., see > > > > > > > > >

http://serverfault.com/questions/487530/find-out-what-high-cpu-usage-apache…

> > > > > ). > > > > > It seems you have done this, however, and I'm wondering why you > > haven't > > > > at > > > > > least been able to narrow down the issue? You should at least

able > > to > > > > know which PHP file is locking up/crashing or the rough

area/cause?

> > > > > > > > Once you know roughly where it is you can add temporary PHP

logging

> > > > commands in the code to help narrow down the issue further. If

you

jlloyd.wiki(a)gmail.com>

> > > wrote: > > > > > > > > > Hey everyone, > > > > > > > > > > Yesterday I posted this to /r/mediawiki (

https://redd.it/3t2apu) > > and > > > > > > cross-posted to /r/apache as well, but unfortunately I've

still

not > > > > > received any feedback other than the one request here for > > clarification > > > > and > > > > > a couple of suggestions on reddit that I'd already covered in

the > > > post. > > > > > > > > > > > > It's possible no one has any suggestions for me regarding

this

> issue > > > (it > > > > is > > > > > a somewhat complex application stack that could be requiring > > > > configuration > > > > > and/or tuning in multiple places, for example), but given how > severe > > > of a > > > > > problem this is for my production sites, I wanted to bump it

once

> in > > > > hopes > > > > > of possibly getting at least some pointers of things to

consider

> > that I > > > > may > > > > > not have already, especially with respect to diagnostics I

could > > > > perform > > > > > on > > > > > > the live web servers beyond just server-status and the

collectd

> > > apache > > > > > > plugin (which is basically the same thing), for example. > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Nov 12, 2015 at 8:02 AM, Justin Lloyd < > > jlloyd.wiki(a)gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > Marcin, > > > > > > > > > > > > > > It's the biggest and most heavily trafficked of our wikis > because > > > its > > > > > the > > > > > > > English-language version of the wiki. We also have German, > > French, > > > > and > > > > > > > Spanish, but the English-speaking community is by far the > largest > > > and > > > > > > most > > > > > > > active. There are some tiny configuration differences

between

the

> > wikis > > > > > (e.g. the value of $wgJobRunRate, the specific extensions

loaded) > > but > > > > > > nothing very significant I don't believe. > > > > > > > > > > > > I should also add that all four of these wikis (we have a

5th,

for > 7 > > > > > total, not 6 as I'd originally said) also use Semantic

MediaWiki > > > > > > extensively. I believe the other three wikis would run into

the

> > same > > > > > > problem if they had same amount of traffic as the English

one.

> > > However, > > > > > > since they all are vhosts within the same Apache instances,

the > > > > English > > > > > > > one's problems affect all of them. > > > > > > > > > > > > > > Justin > > > > > > > > > > > > > > > > > > > > > On Thu, Nov 12, 2015 at 1:42 AM, Marcin Cieslak < > > saper(a)saper.info> > > > > > > wrote: > > > > > > > > > > > > > >> On 2015-11-12, Justin Lloyd <jlloyd.wiki(a)gmail.com>

wrote:

> > > > >> > * Six wikis are configured as Vhosts in Apache, load

one

> of > > > the > > > > > >> public > > > > > >> > wikis dwarfs the rest and it's the wiki giving me

problems.

> > > > >> > > > > >> (...) > > > > >> > > > > >> > I'm mainly looking right now for how to troubleshoot the

stuck > > > > > >> processes, > > > > > >> > but any advice regarding this architecture is also

welcome, > > as I > > > > > feel > > > > > > it > > > > > > >> > could use some improvement but I'm not sure how just

yet.

> > > > > >> > > > > > >> The question that immediately comes to my mind before I

start > > > > digging > > > > > > >> any further - how is the wiki making problems special? Is

> > just > > > > > > getting > > > > > > >> most of the traffic (it is the "most interesting" one) or

its

> > > > >> configuration slightly different? > > > > >> > > > > >> Marcin Cieślak > > > > >> https://www.mediawiki.org/wiki/User:Saper > > > > >> > > > > >> > > > > >> _______________________________________________ > > > > >> MediaWiki-l mailing list > > > > >> To unsubscribe, go to: > > > > >> https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > > > >> > > > > > > > > > > > > > > _______________________________________________ > > > > MediaWiki-l mailing list > > > > To unsubscribe, go to: > > > > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > > > > > > > > > > > > > > > -- > > > Dave Humphrey -- dave(a)uesp.net > > > Founder/Server Admin of the Unofficial Elder Scrolls Pages -- > > www.uesp.net > > > www.viud.net - Building the world's toughest USB drive > > > _______________________________________________ > > > MediaWiki-l mailing list > > > To unsubscribe, go to: > > > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > > > > _______________________________________________ > > MediaWiki-l mailing list > > To unsubscribe, go to: > > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > > > > > -- > Dave Humphrey -- dave(a)uesp.net > Founder/Server Admin of the Unofficial Elder Scrolls Pages -- www.uesp.net

_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

-- Dave Humphrey -- dave(a)uesp.net Founder/Server Admin of the Unofficial Elder Scrolls Pages --

www.uesp.net

_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

Bartosz Dziewoński

8:49 p.m.

On 2015-11-19 19:11, Dave Humphrey wrote:

...

You could also only trigger it in production by some special URL parameter. (And get rid of it after you debug the problem, so that people don't DOS your wiki by forcing it to profile things.) In LocalSettings: if ( isset( $_GET['secretprofileparameter'] ) ) { $wgProfiler = ... } Then visit https://yourwiki.example/.../?secretprofileparameter=1 -- Bartosz Dziewoński

Justin Lloyd

20 Nov 20 Nov

7:48 p.m.

Even with a check profiling still caused me performance problems for some reason, even for a bit after commenting it all out and restarting Apache. Kinda scary. For now I've fallen back to generating a daily list of the URLs with the highest response times (per the %D LogFormat option) to hopefully narrow down the problematic (type of?) pages. On Thu, Nov 19, 2015 at 12:49 PM, Bartosz Dziewoński <matma.rex(a)gmail.com> wrote:

...

On 2015-11-19 19:11, Dave Humphrey wrote:

3113

days inactive

3121

days old

mediawiki-l@lists.wikimedia.org

Manage subscription

15 comments

5 participants

tags (0)

participants (5)

Bartosz Dziewoński
Dave Humphrey
Jurij Byrda
Justin Lloyd
Marcin Cieslak