New subject: [Wikimedia-search] finding related pages [was Re: Reading tech sessions at hackathon]

1 Jun 2015

Summary:
* CirrusSearch has "morelike:*PageName*", who knew?
* I sense a developer article brewing, "Finding related content"

AIUI, the Wikipedia mobile apps' "Read more" section just performs a
full-text search (API [1] ) for the current page title (Android source [2]).

Joaquin's nfity demo http://chimeces.com/webkipedia/ 's "Related pages"
section calls the GettingStarted extension's gettingstartedgetpages API
module [3] with gsgptaskname=morelike . This is implemented by
GettingStarted/MoreLikePageSuggester.php... and it seems this just makes a
search query for srsearch=morelike:Australia .  Who knew Cirrus search had
a "morelike:" keyword?  It's not in the enwiki search help, but it is in
the Cirrus search help [4].

I'm not sure if there's any reason to interpose gettingstartedgetpages
instead of querying search directly for morelike:*pagetitle*, it might
cache stuff in Redis. The mobile apps might get better "Read more"
suggestions using one of these.

There's also a srwhat=suggestion, I don't know if that helps getting
related pages.

I'll be updating https://www.mediawiki.org/wiki/API:Search_and_discovery
with this, and it seems article-worthy.

Cheers, hope this helps someone.

[1] https://en.wikipedia.org/w/api.php?action=help&modules=query%2Bsearch

[2]
https://github.com/wikimedia/apps-android-wikipedia/blob/de0b8b579f5030f684…

[3]
https://en.wikipedia.org/w/api.php?action=help&modules=query%2Bgettings…

[4] https://www.mediawiki.org/wiki/Help:CirrusSearch#Special_prefixes

On Fri, May 29, 2015 at 2:18 AM, Joaquin Oltra Hernandez <
jhernandez(a)wikimedia.org&gt; wrote:

...
  Sorry forgot to link it:
https://github.com/joakin/webkipedia

 Matt Flaschen told me about the gettingstarted 'morelike' mode for other
 purposes, but it fitted perfectly my purposes for this reading app. They
 developed it on the Growth team about a year ago, but the experiment wasn't
 successful so the API has been there dormant and unused for a lot of time
 (works pretty well!).

 About the content I'm fetching for the articles, I'm using the extracts
 with the exintro option, and embedding the html (
 https://github.com/joakin/webkipedia/blob/master/lib/api/article.js). The
 idea would be to have a 'Read more' that would show the full article I
 guess.

 The president's chest is pretty good too :D
 http://chimeces.com/webkipedia/#/wiki/Barack_Obama

 On Fri, May 29, 2015 at 4:35 AM, S Page &lt;spage(a)wikimedia.org&gt; wrote:

  (Cc'ing James Douglas, who's also
developing API playground code.)

 On Thu, May 28, 2015 at 2:52 AM, Joaquin Oltra Hernandez <
 jhernandez(a)wikimedia.org&gt; wrote:

  S, for getting started quickly, I set up a JS web
app completely
 standalone with some basic infrastructure (libraries for calling the api,
 rendering pipeline of JS views, url routing) so that interested people
 could just get quickly to render a view within the app and do interesting
 stuff querying the API. We were also open to just doing a plain html file
 with some JS and CSS, or a codepen/jsbin style would have worked too.

 Here's the demo of the lite wikipedia webapp I worked on:
 http://chimeces.com/webkipedia/

 That's lovely!  It's what API developers develop when they develop.
 *  Where's the source?
 * I had no idea the gettingstartedgetpages would give you related pages,
 so obscure!
 * I guess RESTBase has no mode that strips the citations and such, or
 gives you just the opening section (prop=extracts & exintro=)
 * Nice closeup of the great man's chest  :-),
 http://chimeces.com/webkipedia/#/wiki/Albert_Einstein

 --
 =S Page  WMF Tech writer

-- 
=S Page  WMF Tech writer