On Tue, Sep 15, 2020 at 5:24 AM Derk-Jan Hartman <d.j.hartman+wmf_ml(a)gmail.com> wrote: The tasks that I mentioned in my original message are, likewise, tasks that I'm not sure belong to any team or any particular person. I have been using the phab tag/milestone "Release Engineering (Logspam)" to ensure that we don't lose track of tasks that are: 1. problems in production 2. tagged in phabricator with a team or component (in contrast to problems with unknown components/team tags) 3. no longer resourced or maintained in a discernible way Feel free to apply that tag if those 3 conditions apply to these tasks. Tracking these will make it easier to raise awareness later. Thanks! -- Tyler

On 9/15/20 9:43 AM, Alex Ezell wrote: To an extent, yes. We have separate channels for PHP errors and exceptions, for example, and although I don't think we currently differentiate in logstash, maybe we could plausibly draw a further distinction between PHP error levels. Intuitively, a low number of PHP notices probably indicates something of lower severity than a high number of fatals, and so forth. Teasing out more detail about reported error severity could be a useful exercise, but I'm not sure it would result in much more meaningful signals than we currently have about production health. Serious problems can manifest as trivial-seeming notices, some issues start out that way and cascade over time, and generally any form of recurring logspam needs human evaluation before we can easily say much more than "this is a problem". Yeah. There are doubtless a lot of ways to improve the tooling we use for that process, but right now I think it would be most helpful if we just had more eyes _routinely_ on the logs and the workboard. (See Tyler's earlier and much more detailed/thoughtful response to this thread.) -- Brennen Bearnes Release Engineering

Hi! Thanks for the feedback, this is useful information. On Tue, Sep 15, 2020 at 3:00 AM Niklas Laxström &lt;niklas.laxstrom(a)gmail.com&gt; wrote: Indeed, a trendline for production quality is difficult to compare if a large backlog is being added. +100 to checking for production errors. It's my hope that folks who have code that is going out on a train are: 1. Aware their code is going to production that week 2. Watching for related logs and alerts (where possible) 3. Performing other software quality assurance activities on their code as it rolls out (manual testing, for example) My assessment of risk as a person deploying software to production is necessarily linked to my view into quality assurance activities. If production errors are growing, I worry about sustainability. The production error dashboard's past stability has provided assurances about shared awareness and priority of a given week's deployment. That is, I know there are software quality activities that take place sometime after code hits group0 or group1 or group2; however, much of that activity remains opaque. This is why this dashboard is crucial for deployment. Having the explicit assurances of folks whose code is going to production that week would be preferable to any inference I can make from this dashboard. It's my hope that maintainers and teams triaging and grooming this dashboard will create an emergent process that can be used to provide real insight. That is, if we all are keeping this dashboard up-to-date collectively, it will be easier to see when quality assurance activities have taken place. Further, if we collectively fret over this dashboard then we'll share a collective awareness of anomalies. That's a good question. I think that having a single workboard is nice as there are reporting features[0] that provide some insights about the overall health of production. Those insights are, as evidenced, only as good as their inputs, but they remain valuable to me. Additionally, a single tag may be used in saved searches and custom dashboards to make it easy to stay on top of issues seen in production (is my hope which may not align with how folks triage in practice). Thanks for the feedback. This anomaly makes more sense to me than it did :) -- Tyler [0]: <https://phabricator.wikimedia.org/project/reports/1055/>

This feels like it needs some higher level coordination. Like perhaps managers getting together and deciding production issues are a priority and diverting resources dynamically to address them. Building an awesome new feature will have a lot less impact if the users are hurting from growing disrepair. It seems to me like if individual contributors and maintainers could have solved this problem, they would have by now. I'm a little worried that the only viable solution right now seems like heroes stepping up to fix these bugs. Concretely, I think expanding something like the Core Platform Team's clinic duty might work. Does anyone have a very rough idea of the time it would take to tackle 293 (wow we went up by a dozen since this thread started) tasks?

ACN -- for what it's worth, I've been working for the foundation for a while now, and I can report from the inside that the trend is definitely in a positive direction. There is a lot more internal focus on addressing code debt and giving maintenance a fair spot at the table. (In fact, my entire team is now sitting inside 'maintenance' now, apparently; we used to be 'platform evolution'.) This email thread is one visible aspect of that focus on code quality, not just features. That said, the one aspect which hasn't improved much in my time at the foundation has been the tendency of teams to work in silos. This thread also seems to be a symptom of that: a bunch of production issues are being dropped on the floor ('not resolved in over a month') because they are falling between the silos and nobody knows who is best able to fix them. There are knowledge/expertise gaps among the silos as well: someone qualified to fix a DB issue might be at sea trying to track down a front end bug, and vice-versa---a number of generalists in the org could technically tackle a bug no matter where it lies, but it will take them much longer to grok an unfamiliar codebase than it would for someone more familiar with that silo. So bug triage is an increasingly technical task in its own right. This thread, as I read it sitting inside the org, isn't so much asking for more attention to be paid to maintenance -- we're winning that battle, internally -- as it is a plea for those folks on the edges of their silos to keep an eye out for these things which are currently falling between them and help with the triage. --scott, speaking only for myself and my view here On Wed, Sep 16, 2020 at 11:25 PM AntiCompositeNumber < anticompositenumber(a)gmail.com&gt; wrote: -- (http://cscott.net)

Id be careful about using numbers in triage right now. The numbers are a little misleading as the error logging is only enabled on smaller wikis. Also if an error results in data loss but only impacts a small amount of people I would say that's worse than a benign error that occurs for lots. We rolled out to Spanish, German and Japanese wikipedia yesterday so these numbers will start becoming more useful, but English Wikipedia will severely skew these numbers when we finally enable it. On Tue, Sep 22, 2020, 9:59 AM Ed Sanders &lt;esanders(a)wikimedia.org&gt; wrote: