[Wikitech-l] Recent authentication session instability (was Re: Update from the Wikimedia Performance Team)

On Thu, Feb 4, 2016 at 8:20 AM, MZMcBride <z(a)mzmcbride.com> wrote: The origin of the unexpected logouts falls on the AuthManager project and specifically the SessionManager component that rolled out in 1.27.0-wmf.11 [0]. We had various issues related to the session handling changes including a bug that was overloading the storage capacity of the Redis servers that store session data [1] and two other issues which required rolling the wikis back to 1.27.0-wmf.10 [2][3]. Both rollbacks were accompanied by a run of the "resetGlobalUserTokens.php" maintenance script which updates each user's CentralAuth records in such a way that their authentication session will be considered invalid the next time it is used on a wiki. This was done from an abundance of caution point of view concerning possible issues with sessions that had been issued by the SessionManager software. The reset script is not fast [4], so session invalidation has slowly worked its way across the CentralAuth user table. Part of the enhancements that are being applied in order to bring SessionManager back to production with 1.27.0-wmf.13 is a new config setting that can be used to give us a nearly instant switch to throw to invalidate all active sessions. This setting is actually included in 1.27.0-wmf.12, but the configuration on the Wikimedia cluster has not been changed to make use of it yet. Invalidating all user sessions is not something we plan to do for fun certainly, but there have been in the past (and likely will be in the future) software and configuration issues that necessitate the use of that heavy hammer approach. [0]: https://phabricator.wikimedia.org/T123451 [1]: https://phabricator.wikimedia.org/T125267 [2]: https://wikitech.wikimedia.org/wiki/Incident_documentation/20160123-Session… [3]: https://tools.wmflabs.org/sal/log/AVKZtfQXW8txF7J0uNE2 [4]: https://phabricator.wikimedia.org/T124861 Bryan -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855