We've had a couple odd errors cropping up in the last couple days:
* something tried to update a record in the 'recemtchanges' table
* something tried to run a 'peplaceinternallinks' function
These are both single-bit changes in strings, which is rather suspicious.
The 'recemtchanges' was on srv68, as recorded in the dberrorlog a number of
times over a couple days:
Sun Jan 8 16:56:02 UTC 2006 srv68 RecentChange::markPatrolled
10.0.0.101 1146 Table 'nlwiki.recemtchanges' doesn't exist
(10.0.0.101)
UPDATE `recemtchanges` SET rc_patrolled = '1' WHERE rc_id = '2429501'
When I checked it, the source code looked correct on that system; so it could be
either a bit flip in main memory (bad!), or some mysterious corrupting bug
somewhere else along the way...
There's no hits in /var/log/mcelog on srv68, which _should_ have listed any
detected memory parity errors, allegedly.
The 'pelplaceinternallinks' one I'm not sure where it happened; I don't
think
our PHP error log files were working properly, and we somewhere along the line
stopped patching PHP to include the hostname in error output to clients. But it
was reported multiple times until we restarted the web servers to clear whatever
held the bogus cached code in memory.
I've set PHP to log errors to the syslog now; these then go to the log server on
zwinger and so can be collected and listed per-server in the syslog there.
-- brion vibber (brion @
pobox.com)