Do we process squid logs? On Aug 26, 2013, at 12:31 PM, Diederik van Liere <dvanliere(a)wikimedia.org> wrote:

Yes, we get log lines from Varnish, Squid and Nginx servers. D On Mon, Aug 26, 2013 at 3:32 PM, Toby Negrin <tnegrin(a)wikimedia.org> wrote:

Correct. On Aug 26, 2013, at 3:52 PM, Toby Negrin <tnegrin(a)wikimedia.org> wrote: > > On Aug 26, 2013, at 12:49 PM, Diederik van Liere <dvanliere(a)wikimedia.org> wrote: > >> Yes, we get log lines from Varnish, Squid and Nginx servers. >> D >> >> >> On Mon, Aug 26, 2013 at 3:32 PM, Toby Negrin <tnegrin(a)wikimedia.org> wrote: >> Do we process squid logs? >> >> On Aug 26, 2013, at 12:31 PM, Diederik van Liere <dvanliere(a)wikimedia.org> wrote: >> >>> >>> >>> >>> On Mon, Aug 26, 2013 at 3:23 PM, Matthew Walker <mwalker(a)wikimedia.org> wrote: >>> Whist investigating an orthogonal logging issue I encountered a couple of differences between squid/varnish which I didn't know about: >>> >>> * Varnish does not give subsecond request time information >>> Yes that's correct >>> @ottomata: maybe we can add subsecond request time information to varnish requests? >>> * Varnish does give subsecond request processing time info >>> Yes that's correct. >>> * Varnish calls it 'hit/200' or 'miss/302' instead of 'TCP_MEM_HIT/200' or 'TCP_MISS/302' >>> Yes that's correct. >>> * Varnish does not URL encode the user agent field >>> We never URL encoded the user agent field as far as I know, we did replace space with %20 but have stopped doing that as we are using the tab as separator now. Maybe that patch has not been removed from the squids? >>> >>> Example log lines: >>> amssq41.esams.wikimedia.org 1013692039 2013-07-31T23:00:02.331 0 XXX TCP_MEM_HIT/200 614 GET http://meta.wikimedia.org/XXX NONE/- image/png http://en.wikipedia.org/XXX - Mozilla/5.0%20(Wind... en-US en;q=0.8 - >>> cp1006.eqiad.wmnet 1442176851 2013-07-31T23:00:02.452 0 XXX TCP_MISS/302 406 GET http://meta.wikimedia.org/XXX NONE/- - http://en.m.wikipedia.org/XXX - Mozilla/5.0%20(i.... en-us - >>> cp3012.esams.wikimedia.org 823553992 2013-07-31T23:00:02 0.000119448 XXX hit/200 20 GET http://meta.m.wikimedia.org/XXX - image/png http://de.m.wikipedia.org/XXX XXX Mozilla/5.0 (iPho... de-de - >>> >>> ~Matt Walker >>> Wikimedia Foundation >>> Fundraising Technology Team >>> >>> _______________________________________________ >>> Analytics mailing list >>> Analytics(a)lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/analytics >>> >>> >>> _______________________________________________ >>> Analytics mailing list >>> Analytics(a)lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/analytics >> >> _______________________________________________ >> Analytics mailing list >> Analytics(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/analytics >> >> >> _______________________________________________ >> Analytics mailing list >> Analytics(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/analytics > _______________________________________________ > Analytics mailing list > Analytics(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/analytics

On Mon, Aug 26, 2013 at 12:31 PM, Diederik van Liere < dvanliere(a)wikimedia.org&gt; wrote: It's not required for fundraising's purposes -- so don't add it just on our behalf. I have updated https://wikitech.wikimedia.org/wiki/Squid_log_format to reflect all of this though. ~Walker