On Wed, Jun 19, 2013 at 1:36 PM, Brian Wolff &lt;bawolff(a)gmail.com&gt; wrote: Maybe, e.g. the current one I see if I tail -f fluorine:/a/mw-logs/fatals.log is: [20-Jun-2013 18:54:45] Fatal error: Call to a member function getCode() on a non-object at /usr/local/apache/common-local/php-1.22wmf8/includes/GlobalFunctions.php on line 1288 Seems OK to display, but meanwhile in exceptions.log: 2013-06-20 18:30:45 mw1076 bswiki: [6d110124] /wiki/[redacted] Exception from line 3303 of /usr/local/apache/common-local/php-1.22wmf7/includes/User.php: User::addToDatabase: hit a key conflict attempting to insert user '[redacted], but it was not present in select! So the exception/error alone can reveal stuff. And I guess it could hint at an exploit (I hope neither of those do :-/ ). If there's a problem on a WMF site, unless it's reproduceable on a stock test wiki, I think it'll need someone with access to the fluorine logs machine. For those that have access, < https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Test_and_monitor_you… and <https://wikitech.wikimedia.org/wiki/Logs> have advice about monitoring logs and graphs. -- =S Page engineer on Editor Engagement Experiments

Well, the Ganglia graphs distinguish different types of errors (out-of-memory fatals, time limit fatals, miscellaneous fatals, exceptions, catchable fatals, and query errors). At present there is nothing that is more granular than that, private or public. The error log we consult is an undifferentiated stream of text. However, it is an area of our code that could easily welcome contributions from the community. Hashar enabled error logging for the beta cluster, so labs is now a viable development environment for a generic error-processing solution. Relevant code exists in two locations: https://git.wikimedia.org/blob/operations%2Fpuppet.git/9792c164d10f9f9f2092… (this is the script that is emitting stats to Ganglia) and https://git.wikimedia.org/tree/mediawiki%2Ftools%2Ffluoride.git (set of regexps to parse the data even further; not currently used anywhere.) I've been working on this in my spare time, but I'd be happy to provide mentorship, code review & deployment from interested contributors. If someone competent (a category which explicitly includes you, Brian!) wants to take over and "own" this problem, that's cool with me too. There's a lot we could do in this area. It should be possible to probabilistically trace an error to the commit(s) that introduced it.