Hi all you EventLogging users out there!
tl;dr
We will switch EventLogging MySQL ingestion to be based on a schema
whitelist rather than blacklist.
As you know, we currently import EventLogging events into two locations for
analysis: The MySQL ‘log’ database, and the Hive ‘event’ database. MySQL
is not able to handle high volume events. We currently blacklist
<https://github.com/wikimedia/puppet/blob/production/hieradata/common.yaml#L362>
any schemas that we know have high volumes from being ingested into MySQL.
This can cause problems when a new high volume schema is deployed, as it
requires knowledge and communication from the schema owners to the
Analytics team, and it requires an Analytics Operations engineer to make a
Puppet commit to blacklist the schema. To address this problem, we will
switch the EventLogging MySQL schema blacklist to a whitelist. All schemas
that are actively being ingested into MySQL today will be whitelisted. In
the future, if you want an event schema to be ingested into MySQL, you’ll
need to ask the Analytics team to whitelist it.
Hive has been working for EventLogging analysis for a while now. It has
almost all of the schemas that MySQL does, plus the high volume ones. One
day in the (distant?) future, we’d like to decommission MySQL storage of
events. (Don’t worry yet, MySQL decommissioning has a lot of blockers and
this work is not planned.) By not ingesting events into MySQL by default,
we hope to encourage more users to switch to Hive.
This switch to a whitelist will happen this week. If you are deploying a
new schema and expect it to show up in MySQL, let us know so we can
whitelist it!
Thanks!
-Andrew + the Analytics Engineering team
https://phabricator.wikimedia.org/T203596