Hey Folks (and Florian in particular),
We created an internal web list to discuss admin stuff, and I failed by
posting the email to the internal list. I wanted to catch you (and the
rest of the team up on the latest link preview data).
[image: Inline image 1]
*Link preview rationale: *
the hypothesis is that if blue-links are more effective overall for users
(benefit/cost is higher on average) with link preview, people will click
them more often.
phab ticket explains the reasoning more:
https://phabricator.wikimedia.org/T111329
*Android results:*
Link preview increased blue-link clicks/user by almost 30% in beta, but in
stable it looks flat to negative. We tried a new version and it was
slightly better, but still no improvement over 'no link-preview' condition.
*Web decision*
One of the reasons we have independent teams on each platform is to allow
us to not make the same mistake on more than one platform. Based on the
above Android results, we made a late-quarter decision not to push link
preview until more analysis could be done. This is what email below kicked
off.
*Update*
It occurred to us yesterday that our current metric is probably
incomplete. We were looking at clicks/user on Android, when we should have
been looking at clicks/pageview. We treated them as equivalent, but what
actually happens is that pageviews drop since not all users continue onto
the article. So clicks/user sees no improvement, but clicks/page looks to
have an increase of ~10%!![?] So this suggests that link preview is
arguably an improvement. I say arguable because, the drop in overall
clicks comes from the fact that before, a user would click on a link, go to
an article and then click on a link from that article (3 articles total).
Now, the user will sometimes decide they have enough information and not
visit the second article. We are giving more targeted information, but are
losing some 'serendipity'.
As Kaity put it, "
*The link preview UX is encouraging people to stay on one article and check
out lots of links but not click through to any of them.*
*Without link preview, a user is more likely to get completely off-topic,
and forget the original article. *
*Which of these behaviors do we want to encourage? We should balance a
focused reading experience with one that encourages the wiki rabbit hole. *
*If a user is encouraged to explore in other ways, like the feed and read
more, than a UX that encourages them to stay focused on 1 article is
justified. *
*Next steps:*
We made the call not to work on link preview on the mobile web (or
hovercards on desktop web [1]) in the remainder of Q2 based on our earlier
analysis. I am tempted to say that this secondary analysis justifies
implementing link preview on the web, but think it is at least worth a
discussion for mobile web, due to the potential trade-off. Do we try again
in Q3? I know Florian is already doing a great deal of the work.
-J
[?] Dmitry on the hook for further analysis
[1] The survey results for hovercards when rolled out as'on-by-default' to
specific wikis were astoundingly positive...I don't have them handy, but if
you're curious, please ping me.
---------- Forwarded message ----------
From: Jon Katz <jkatz(a)wikimedia.org>
Date: Fri, Nov 20, 2015 at 5:05 PM
Subject: Next sprint
To: Joaquin Oltra Hernandez <jhernandez(a)wikimedia.org>rg>, Kristen Lans <
klans(a)wikimedia.org>gt;, Jon Robson <jrobson(a)wikimedia.org>
Hi,
I want to give you a head's up on some recent developments that could
impact next sprint. All of this came in either last night or this
afternoon--we can discuss on Monday at kickoff, but asynch before then will
obviously save meeting time.
*Link Preview:*
Just under the wire, the first couple days of data are in for the new link
preview on Android and they do not show a significant marked increase in
links clicked per user. In other words, it looks like link preview is still
not generating any measurable improvement in user engagement on Android. I
am not a good judge of how much work it would take to push link preview to
mobile beta for further testing (an actual a/b test), but think that is
probably not a great use of your time (unless it is really small). On the
one hand, I believe in the feature and want to give it a chance on web, but
I also believe in staggering development by platform so that we don't make
the same mistakes twice. If anything, the bar is set higher on the mobile
web then on Android, since we are hijacking link clicks.
If we don't launch link preview we could (and should) continue to move
hovercards along as they do not hijack the link and have rave qualitative
reviews. This might still represent enough work to see us through the rest
of the quarter, but I am not sure if you have what you need lined up. How
do you folks feel about it?
*Added to sprint board since planning:*
1. This bug, if it is valid, will prevent us from gathering data from
our surveys until it is fixed:
https://phabricator.wikimedia.org/T119152
2. Added another story to account for the 2 surveys that we hope to run
next week before the fundraiser block:
https://phabricator.wikimedia.org/T119267
3. Might not make it:
1. After Toby and I met with community members yesterday, he is
making the executive decision that we should get rid of the special user
profile to appease our contributors:
https://phabricator.wikimedia.org/T85929. I think we have stalled
long enough, but I think we should only remove it if a user
doesn't already
have a page. Otherwise a new user who clicks on their user name
gets taken
to a blank edit screen. If a user doesn't have a page, we should add a
message to specialuserprofile saying --click here to create your own user
profile. A CTA on the link click could also work. Nirzar can
you jump on
this?
-J