[Wikitech-l] Re: Enabling canary events for all MediaWiki event streams

tl;dr Ignore this email if you do not use MediaWiki event streams. On Monday December 11 2023, all MediaWiki related event streams will have artificial canary events <https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams#Canary_Events> injected into them. If you use any of these streams, you should discard thesee canary events. Add code to your consumers that discards events where meta.domain == "canary". Canary Events At WMF, we use artificial 'canary' AKA 'heartbeat' events <https://wikitech.wikimedia.org/wiki/Event_Platform/Stream_Configuration#canary_events_enabled> to differentiate between a broken event stream and an empty event stream. Canary events should be produced at least once an hour. If there are no events in a stream for an hour, then something is likely broken with that stream. These artificial canary events can be identified by the fact that their meta.domain field is set to "canary". If you use any of the streams listed below, you will need to add code that discards any events where meta.domain == "canary". Back in 2020, we began producing canary events into all new streams, but we never got around to enabling these for streams that already existed. We needed to ensure that all consumers of these streams filtered out the canary events. We're just finally getting around to enabling canary events for all streams. We will enable canary event production <https://phabricator.wikimedia.org/T266798> for the following streams on Monday, December 11th, 2023: - mediawiki.recentchange - mediawiki.page-create - mediawiki.page-delete - mediawiki.page-links-change - mediawiki.page-move - mediawiki.page-properties-change - mediawiki.page-restrictions-change - mediawiki.page-suppress - mediawiki.page-undelete - mediawiki.revision-create - mediawiki.revision-visibility-change - mediawiki.user-blocks-change - mediawiki.centralnotice.campaign-change - mediawiki.centralnotice.campaign-create - mediawiki.centralnotice.campaign-delete If you consume any of these streams, either external to WMF networks using EventStreams, or internally using Kafka, please ensure that your consumer logic discards events where meta.domain == "canary" before this date. (Note that not all of these streams are exposed publicly at stream.wikimedia.org <https://stream.wikimedia.org/?doc#/streams>.) Thank you, -Andrew Otto & the WMF Data Engineering team <https://wikitech.wikimedia.org/wiki/Data_Engineering> References - T266798 - Enable canary events for all MediaWiki streams <https://phabricator.wikimedia.org/T266798> - T251609 - Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events <https://phabricator.wikimedia.org/T251609>