[Engineering] Important news about Analytics databases

Wed Nov 22 22:10:10 UTC 2017

>The log database is scheduled to be dropped from dbstore1002 on Tuesday
28th. After that, the log database will be available only on db1108
(analytics-slave.eqiad.wmnet).
To make sure everyone is on the same page this means that you need to
connect to analytics-slave.eqiad.wmnet if you wish to look at eventlogging
data on MySQL

On Tue, Nov 21, 2017 at 6:28 AM, Luca Toscano <ltoscano at wikimedia.org>
wrote:

> Hi everybody,
>
> some updates about the status of the Analytics databases refactoring:
>
> 1) analytics-slave.eqiad.wmnet's CNAME now points to db1108, the new host.
> The staging database that was on db1047 (the old CNAME) has been copied to
> db1108 so all the previous data is there. We are working on the owners of
> the remaining databases to figure out what to backup and what not, after
> that we'll be able to finally decommission db1047.
>
> 2) s[12]-analytics.eqiad.wmnet now point to dbstore1002.eqiad.wmnet
> (analytics-store). Previously they were pointing to db1047 (old CNAME for
> analytics-slave).
>
> 3) the analytics-store.eqiad.wmnet log database (dbstore1002.eqiad.wmnet)
> is going to be dropped soon (it was scheduled for the 20th but we thought
> to wait a bit more). Some people already followed up in
> https://phabricator.wikimedia.org/T156844 and as far as I can see there
> is no opposition to proceed, please ping us otherwise.
> The log database is scheduled to be dropped from dbstore1002 on Tuesday
> 28th. After that, the log database will be available only on db1108
> (analytics-slave.eqiad.wmnet).
>
> Thanks a lot!
>
> Luca
>
> 2017-11-08 12:02 GMT+01:00 Luca Toscano <ltoscano at wikimedia.org>:
>
>> Hi everybody,
>>
>> the Analytics team needs to make some changes to the current
>> configuration and deployment of the Analytics databases. Before starting a
>> little refresh to be on the same page:
>>
>> - db1046 - eventlogging master database
>> - db1047 - also known as analytics-slave.eqiad.wmnet - replicates via
>> mysql s1/s2 and the log database (on db1046) using a custom replication
>> script.
>> - dbstore1002 - also known as analytics-store.eqiad.wmnet and
>> x1-analytics-slave.eqiad.wmnet - replicates most of the S shards and X1 via
>> mysql, and the log database using a custom replication script.
>> - db1108 (brand new host) - replicates the log database using a custom
>> replication script.
>>
>> We have been suffering during the past months some space and performance
>> issues on dbstore1002 (https://phabricator.wikimedia.org/T168303), so we
>> came up with the following plan:
>>
>> - db1108, a brand new host with SSD disks, replaces db1047 and becomes
>> the CNAME of analytics-slave.eqiad.wmnet. This new host will be a replica
>> of the log database only, no other database will be replicated.
>> - dbstore1002 will loose the support of the log database, that will be
>> dropped from the host.
>> - db1047 will eventually be decommissioned (after backing up data and
>> alert people beforehand - T156844).
>>
>> This will allow us to:
>> 1) Reduce the load on dbstore1002 and free a lot of space on the host.
>> 2) Offer a more performant way to query eventlogging analytics data.
>> 3) Reduce the current performance issues that we have been experiencing
>> while trying to sanitize/purge old event-logging data (
>> https://phabricator.wikimedia.org/T156933)
>>
>> The plan is the following:
>>
>> - November 13th: the analytics-slave CNAME moves from db1047 to db1108
>> - November 20th: the log database will be dropped from
>> dbstore1002/analytics-store together with the event-logging replication
>> script
>> - December 4th: shutdown of db1047 (prior backup of non-log database
>> tables)
>>
>> More info in https://phabricator.wikimedia.org/T156844
>>
>> To summarize what will change from the users perspective:
>>
>> - dbstore1002 (analytics-store) will offer all the S/X shards replication
>> (wikis) and all the databases like staging that everybody is used to work
>> with. It will only loose the support of the log database.
>> - db1108 will offer the log database replication and a staging database.
>> - the db1047's (analytics-slave) staging database will be moved or copied
>> with a different name (like staging_db1047) to dbstore1002.
>>
>> Please let us know in the task your opinion in T156844, we'd love to hear
>> some feedback before proceeding, especially about extra requirements that
>> we haven't thought of.
>>
>> Thanks!
>>
>> Luca (on behalf of the Analytics team)
>>
>>
>
> _______________________________________________
> Engineering mailing list
> Engineering at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/engineering
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/engineering/attachments/20171122/55b494db/attachment.html>