GerardM post triggered my interest to post to the mailing list. As you
might know I am working on functional quadstore that is quadstore that
keeps around old version of data, like a wiki but in direct-acyclic-graph.
It only stores differences between commits. It rely on snapshot of the
latest version for fast reads. My ultimate goal is to build somekind of
portable knowlege base. That is something like WikiBase + blazegraph but
that you spinup on regular machine with the press of button.
Enought brag about me. I wont't reply to all the message of the threads one
by one but:
Here is what SHOULD BE possible:
- incremental dumps
- time traveling queries
- full dumps
- The federation of wikibase SHOULD BE possible since it stored in a
history like GIT and git pull git push are planned in the ROADMAP
And online edition of the quadstore.
Access Control List are not designed yet, I except that this should be
enforced by the application layer.
I planned start working on Data Management System (something like CKAN)
with search featrure. But I would gadly work with wikimedia instead.
Also, given it modeled after git, one can do merge-request like features,
ie. exist the massive import that is crippled.
What I would need is logs possibly with timing of queries (read and write)
to do benchmarks.
Maybe I should ask for fund at mediawiki?
FWIW, I got 2 times faster than blazegraph on microbenchmark.
Hoi,
Wikidata grows like mad. This is something we all experience in the really
bad response times we are suffering. It is so bad that people are asked
what kind of updates they are running because it makes a difference in the
lag times there are.
Given that Wikidata is growing like a weed, it follows that there are two
issues. Technical - what is the maximum that the current approach supports
- how long will this last us. Fundamental - what funding is available to
sustain Wikidata.
For the financial guys, growth like Wikidata is experiencing is not
something you can reliably forecast. As an organisation we have more money
than we need to spend, so there is no credible reason to be stingy.
For the technical guys, consider our growth and plan for at least one
year. When the impression exists that the current architecture will not
scale beyond two years, start a project to future proof Wikidata.
It will grow and the situation will get worse before it gets better.
Thanks,
GerardM
PS I know about phabricator tickets, they do not give the answers to the
questions we need to address.