This should have an effect on both ZRR and relevance, but I think the
effect will be more on the relevance side than the ZRR side. We could do a
manual run through against our labs cluster to check for ZRR changes.
I hadn't thought about it but i suppose if we want to measure relevance
that has to be done on the client side to integrate with our satisfaction
metrics. The common terms patch supports this use case as well (feature
can be enabled via query string) if we want to go that route.
On Mon, Oct 5, 2015 at 9:25 AM, Trey Jones <tjones(a)wikimedia.org> wrote:
David or Erik can correct me if need be, but I think
Common Terms is going
to improve results quality more than ZRR.
Also, we don't have anything that quite lives up to the label of "system"
for re-running queries. We have "methods"—hacks, scripts, and other
non-systematic ways of achieving that goal. [Insert plug for Relevance Lab
here.]
Trey Jones
Software Engineer, Discovery
Wikimedia Foundation
On Mon, Oct 5, 2015 at 12:18 PM, Mikhail Popov <mpopov(a)wikimedia.org>
wrote:
1. Is the common terms query work supposed to
address zero results rate
or…? Don't we have a system now for running old queries through the engine
and seeing whether the change affected the ZRR? If the test is not
dependent on the user's actions, we don't need to do an A/B test that
involves users.
2. We finally have a couple of ways of assessing user engagement with our
search results (more on that in an announcement later today). If this work
is meant to address that, then we need to run an A/B test, but include the
TestSatisfaction2 schema (or a variant of it that also tracks the grouping).
On Mon, Oct 5, 2015 at 8:59 AM, Erik Bernhardson <
ebernhardson(a)wikimedia.org> wrote:
David's work on the common terms query is
merged and will be deploying
this week. It's hidden behind a feature flag so no user visible change
will occur.
I thought we had decided on an A/B test to check the usefulness of this
approach but I can't find anything in phab. So the question is, how do we
want to AB test this? It will be easiest to run this test through our
backend AB testing infrastructure, i think we just need to choose a
sampling rate.
_______________________________________________
discovery mailing list
discovery(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/discovery
--
*Mikhail Popov* // Data Analyst, Discovery
<https://www.mediawiki.org/wiki/Wikimedia_Discovery>
https://wikimediafoundation.org/
*Imagine a world in which every single human being can freely share in
the **sum of all knowledge. That's our commitment.* Donate
<https://donate.wikimedia.org/>.
_______________________________________________
discovery mailing list
discovery(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/discovery
_______________________________________________
discovery mailing list
discovery(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/discovery