On Wed, Jun 8, 2016 at 4:14 PM Erik Bernhardson <ebernhardson(a)wikimedia.org>
wrote:
On Wed, Jun 8, 2016 at 3:28 PM, Chad
<innocentkiller(a)gmail.com> wrote:
On Wed, Jun 8, 2016 at 3:18 PM Deborah Tankersley
<
dtankersley(a)wikimedia.org>
wrote:
Hi Romaine,
Yes, the search box on the
wikipedia.org portal only shows results for
the
language that you're currently searching in
(which can be changed by
selecting a new language in the dropdown located in the search box). We
don't yet have the ability to search in all wikis at once but we'd like
to
have that capability in the future.
Actually we do have the ability to search all wikis at once, it's just
really
really slow because of how Nik and I chose to setup the indexes ~4 years
ago. This could be fixed though :)
Technically possible, yes. Doable with the current architecture and query
load though, not really feasible. The current architecture (as i'm sure
you're aware) would mean having the elasticsearch servers convert what is
currently a search against 1 index into a search against 900 indexes. Many
of these indexes are small, but a 900x increase in volume is nothing to
scoff at. We do +- 31M full text searches a day, so it's not a tiny amount
of work. It would be asking our servers to process an extra 28 billion
index searches per day.
We are doing some initial work on figuring out how to reduce this count.
Over the next two quarters discovery will be looking into how to merge
indices of the same language (eswiki, esnews, eswikivoyage, eswiktionary,
eswikiversity, eswikisource, eswikiquote) into a single index per language,
which will make searching between projects in the same language performant,
and will hopefully pave the way for future architecture changes that will
allow more native cross-project searches. Look for this in the coming
months :)
Awesome! This was always our goal but some early architecture decisions
we made has made it harder than it needed to be for the scaling reasons
you point out.
Great to see work happening towards resolving this and making real cross-
wiki searches much closer to reality :)
-Chad