Would it be an idea if HDT remains unfeasible to place the journal file of blazegraph online? Yes, people need to use blazegraph if they want to access the files and query it but it could be an extra next to turtle dump?

is not needed anymore. How much larger would this be compared to the Turtle file? _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

2017-10-27 18:51 GMT+02:00 Luigi Assom <itsawesome.yes(a)gmail.com>om>: The link to the Wikidata dump seems dead, unfortunately :'(

On 27 Oct 2017, at 18:58, Jérémie Roquet <jroquet(a)arkanosis.net> wrote: 2017-10-27 18:56 GMT+02:00 Jérémie Roquet <jroquet(a)arkanosis.net>et>: … but there's a file on the server: http://gaia.infor.uva.es/hdt/wikidata-20170313-all-BETA.hdt.gz (ie. the link was missing the “.gz”) -- Jérémie _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Also, for avoiding your users to re-create the models, you can pre-load "models" from LOV catalog.

Thanks to report that. I remember one issue that I added here https://github.com/pyvandenbussche/lov/issues/66

@Laura : you mean this list http://lov.okfn.org/lov.nq.gz ? I can download it !! Which one ? Please send me the URL and I can fix it !!

No, the idea is that each organization will have its own KNS, so users can add the KNS that they want. 

KBox is an alternative to other existing architectures for publishing KB such as SPARQL endpoints (e.g. LDFragments, Virtuoso), and Dump files. I should add that you can do federated query with KBox as as easier as you can do with SPARQL endpoints.

I will look into the size of the jnl file but should that not be located where the blazegraph is running from the sparql endpoint or is this a special flavour? Was also thinking of looking into a gitlab runner which occasionally could generate a HDT file from the ttl dump if our server can handle it but for this an md5 sum file would be preferable or should a timestamp be sufficient?

2017-10-27 18:51 GMT+02:00 Luigi Assom <itsawesome.yes(a)gmail.com>om>: The link to the Wikidata dump seems dead, unfortunately :'(

is it possible to store a weighted adjacency matrix as an HDT instead of an RDF? Something like a list of entities for each entity, or even better a list of tuples for each entity. So that a tuple could be generalised with propoerties.

Dear Laura, others, If somebody points me to the RDF datadump of Wikidata I can deliver an HDT version for it, no problem. (Given the current cost of memory I do not believe that the memory consumption for HDT creation is a blocker.) --- Cheers, Wouter Beek. Email: wouter(a)triply.cc WWW: http://triply.cc Tel: +31647674624 <+31%206%2047674624> On Fri, Oct 27, 2017 at 5:08 PM, Laura Morales &lt;lauretas(a)mail.com&gt; wrote: the already available Turtle dump [2]. HDT is a binary format to store RDF data, which is pretty useful because it can be queried from command line, it can be used as a Jena/Fuseki source, and it also uses orders-of-magnitude less space to store the same data. The problem is that it's very impractical to generate a HDT, because the current implementation requires a lot of RAM processing to convert a file. For Wikidata it will probably require a machine with 100-200GB of RAM. This is unfeasible for me because I don't have such a machine, but if you guys have one to share, I can help setup the rdf2hdt software required to convert Wikidata Turtle to HDT. _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

The first part of the Turtle data stream seems to contain syntax errors for some of the XSD decimal literals.  The first one appears on line 13,291: Notice that scientific notation is not allowed in the lexical form of decimals according to XML > Schema Part 2: Datatypes[https://www.w3.org/TR/xmlschema11-2/#decimal].  (It is allowed in floats and doubles.)  Is this a known issue or should I report this somewhere?

Le 28 oct. 2017 à 08:50, Stas Malyshev &lt;smalyshev(a)wikimedia.org&gt; a écrit : Hi! Float/double are range-limited and have limited precision. Decimals are not. Whether it is important for geo precision, needs to be checked, but we could be hitting the limits of precision pretty quickly. -- Stas Malyshev smalyshev(a)wikimedia.org _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

The first part of the Turtle data stream seems to contain syntax errors for some of the XSD decimal literals.  The first one appears on line 13,291: ```text/turtle <http://www.wikidata.org/value/ec714e2ba0fd71ec7256d3f7f7606c35> <http://wikiba.se/ontology-beta#geoPrecision> "1.0E-6"^^<http://www.w3.org/2001/XMLSchema#decimal> . ```

Thanks for the pointer!  I'm downloading from https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.ttl.gz now.

@Laura: I suspect Wouter wants to know if he "ignores" the previous errors and proposes a rather incomplete dump (just for you) or waits for Stas' feedback.

Btw why don't you use the oldest version in HDT website?

Let me try to understand ... You are a "data consumer" with the following needs: - Latest version of the data - Quick access to the data - You don't want to use the current ways to access the data by the publisher (endpoint, ttl dumps, LDFragments) However, you ask for a binary format (HDT), but you don't have enough memory to set up your own environment/endpoint due to lack of memory. For that reason, you are asking the publisher to support both .hdt and .hdt.index files. Do you think there are many users with your current UC?

Le 31 oct. 2017 à 20:44, Luigi Assom &lt;itsawesome.yes(a)gmail.com&gt; a écrit : Doh what's wrong with asking for supporting own user case "UC" ? I think it is a totally legit question to ask, and that's why this thread exists. Also, I do support for possibility to help access to data that would be hard to process from "common" hardware. Especially in the case of open data. They exists to allow someone take them and build them - amazing if can prototype locally, right? I don't like the use case where a data-scientist-or-IT show to the other data-scientist-or-IT own work looking for emotional support or praise. I've seen that, not here, and I hope this attitude stays indeed out from here.. I do like when the work of data-scientist-or-IT ignites someone else's creativity - someone who is completely external - , to say: hey your work is cool and I wanna use it for... my use case! That's how ideas go around and help other people build complexity over them, without constructing not necessary borders. About a local version of compressed, index RDF - I think that if was available, more people yes probably would use it. _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

I've just loaded the provided hdt file on a big machine (32 GiB wasn't

I'll try to run a few queries to see how it behaves.

It's a machine with 378 GiB of RAM and 64 threads running Scientific Linux 7.2, that we use mainly for benchmarks. Building the index was really all about memory because the CPUs have actually a lower per-thread performance (2.30 GHz vs 3.5 GHz) compared to those of my regular workstation, which was unable to build it.

You're right. The limited query language of hdtSearch is closer to grep than to SPARQL. Thank you for pointing out Fuseki, I'll have a look at it.

OK. I wonder though, if it would be possible to setup a regular HDT dump alongside the already regular dumps. Looking at the dumps page, https://dumps.wikimedia.org/wikidatawiki/entities/, it looks like a new dump is generated once a week more or less. So if a HDT dump could

Please take me out from these conversations.

I am currently downloading the latest ttl file. On a 250gig ram machine. I will see if that is sufficient to run the conversion Otherwise we have another busy one with  around 310 gig.

For querying I use the Jena query engine. I have created a module called HDTQuery located http://download.systemsbiology.nl/sapp/ which is a simple program and under development that should be able to use the full power of SPARQL and be more advanced than grep… ;)

If this all works out I will see with our department if we can set up if it is still needed a weekly cron job to convert the TTL file. But as it is growing rapidly we might run into memory issues later?

Does this tool allow to query HDT files from command-line, with SPARQL, and without the need to setup a Fuseki endpoint?

command line tools reside), since it uses parts of Jena (e.g. ARQ). proper Java environment. Does this tool work nicely with large HDT files such as wikidata? Or does it need to load the whole graph+index into memory?

Hello list, a very kind person from this list has generated the .hdt.index file for me, using the 1-year old wikidata HDT file available at the rdfhdt website. So I was finally able to setup a working local endpoint using HDT+Fuseki. Set up was easy, launch time (for Fuseki) also was quick (a few seconds), the only change I made was to replace -Xmx1024m to -Xmx4g in the Fuseki startup script (btw I'm not very proficient in Java, so I hope this is the correct way). I've ran some queries too. Simple select or traversal queries seems fast to me (I haven't measured them but the response is almost immediate), other queries such as "select distinct ?class where { [] a ?class }" takes several seconds or a few minutes to complete, which kinda tells me the HDT indexes don't work well on all queries. But otherwise for simple queries it works perfectly! At least I'm able to query the dataset! In conclusion, I think this more or less gives some positive feedback for using HDT on a "commodity computer", which means it can be very useful for people like me who want to use the dataset locally but who can't setup a full-blown server. If others want to try as well, they can offer more (hopefully positive) feedback. For all of this, I heartwarmingly plea any wikidata dev to please consider scheduling a HDT dump (.hdt + .hdt.index) along with the other regular dumps that it creates weekly. Thank you!! _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

On 3 Nov 2017, at 10:05, Ettore RIZZA &lt;ettorerizza(a)gmail.com&gt; wrote: Thank you for this feedback, Laura. Is the hdt index you got available somewhere on the cloud? Cheers 2017-11-03 9:56 GMT+01:00 Osma Suominen &lt;osma.suominen(a)helsinki.fi <mailto:osma.suominen@helsinki.fi>>: Hi Laura, Thank you for sharing your experience! I think your example really shows the power - and limitations - of HDT technology for querying very large RDF data sets. While I don't currently have any use case for a local, queryable Wikidata dump, I can easily see that it could be very useful for doing e.g. resource-intensive, analytic queries. Having access to a recent hdt+index dump of Wikidata would make it very easy to start doing that. So I second your plea. -Osma Laura Morales kirjoitti 03.11.2017 klo 09:48: Hello list, a very kind person from this list has generated the .hdt.index file for me, using the 1-year old wikidata HDT file available at the rdfhdt website. So I was finally able to setup a working local endpoint using HDT+Fuseki. Set up was easy, launch time (for Fuseki) also was quick (a few seconds), the only change I made was to replace -Xmx1024m to -Xmx4g in the Fuseki startup script (btw I'm not very proficient in Java, so I hope this is the correct way). I've ran some queries too. Simple select or traversal queries seems fast to me (I haven't measured them but the response is almost immediate), other queries such as "select distinct ?class where { [] a ?class }" takes several seconds or a few minutes to complete, which kinda tells me the HDT indexes don't work well on all queries. But otherwise for simple queries it works perfectly! At least I'm able to query the dataset! In conclusion, I think this more or less gives some positive feedback for using HDT on a "commodity computer", which means it can be very useful for people like me who want to use the dataset locally but who can't setup a full-blown server. If others want to try as well, they can offer more (hopefully positive) feedback. For all of this, I heartwarmingly plea any wikidata dev to please consider scheduling a HDT dump (.hdt + .hdt.index) along with the other regular dumps that it creates weekly. Thank you!! _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org <mailto:Wikidata@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikidata <https://lists.wikimedia.org/mailman/listinfo/wikidata> -- Osma Suominen D.Sc. (Tech), Information Systems Specialist National Library of Finlan <https://maps.google.com/?q=y+of+Finlan&entry=gmail&source=g>d P.O. Box 26 (Kaikukatu 4) 00014 HELSINGIN YLIOPISTO Tel. +358 50 3199529 <tel:%2B358%2050%203199529> osma.suominen(a)helsinki.fi <mailto:osma.suominen@helsinki.fi> http://www.nationallibrary.fi <http://www.nationallibrary.fi/> _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org <mailto:Wikidata@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikidata <https://lists.wikimedia.org/mailman/listinfo/wikidata> _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Thank you for this feedback, Laura.  Is the hdt index you got available somewhere on the cloud?

I’ve created a Phabricator task (https://phabricator.wikimedia.org/T179681) for providing a HDT dump, let’s see if someone else (ideally from the ops team) responds to it. (I’m not familiar with the systems we currently use for the dumps, so I can’t say if they have enough resources for this.)

Hi Jeremie, Thanks for this info. In the meantime, what about making chunks of 3.5Bio triples (or any size less than 4Bio) and a script to convert the dataset? Would that be possible ? Best, Ghislain Provenance : Courrier <https://go.microsoft.com/fwlink/?LinkId=550986> pour Windows 10 *De : *Jérémie Roquet &lt;jroquet(a)arkanosis.net&gt; *Envoyé le :*mardi 7 novembre 2017 15:25 *À : *Discussion list for the Wikidata project. &lt;wikidata(a)lists.wikimedia.org&gt; *Objet :*Re: [Wikidata] Wikidata HDT dump Hi everyone, I'm afraid the current implementation of HDT is not ready to handle more than 4 billions triples as it is limited to 32 bit indexes. I've opened an issue upstream: https://github.com/rdfhdt/hdt-cpp/issues/135 Until this is addressed, don't waste your time trying to convert the entire Wikidata to HDT: it can't work. -- Jérémie _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

what computer did you use for this? IIRC it required >512GB of RAM to

http://www.rdfhdt.org/datasets/]. Thank you very much! Keep it up! Out of curiosity, what computer did you use for this? IIRC it required contains about twice the limit of 4B triples discussed above. There is a 64-bit version of HDT that doesn't have this limitation of 4B triples. format? Speaking for me at least, Fuseki with a HDT store. But I know there are also some CLI tools from the HDT folks. _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

You shouldn't have to keep anything in RAM to HDT-ize something as you could make the dictionary by sorting on disk and also do the joins to look up everything against the dictionary by sorting.

drops `a wikibase:Item` and `a wikibase:Statement` types

On 7 Nov I created an HDT file based on the then current download link from https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.ttl.gz

PS: If this resource turns out to be useful to the community we can offer an updated HDT file at a to be determined interval.

Somebody pointed me to the following issue: https://phabricator.wikimedia.org/T179681 Unfortunately I'm not able to log in there with the "Phabricator" so I cannot edit the issue directly. I'm sending this email instead.

Dear Laura, others, If somebody points me to the RDF datadump of Wikidata I can deliver an HDT version for it, no problem. (Given the current cost of memory I do not believe that the memory consumption for HDT creation is a blocker.)

Hello everyone, I'd like to ask if Wikidata could please offer a HDT [1] dump along with the already available Turtle dump [2]. HDT is a binary format to store RDF data, which is pretty useful because it can be queried from command line, it can be used as a Jena/Fuseki source, and it also uses orders-of-magnitude less space to store the same data. The problem is that it's very impractical to generate a HDT, because the current implementation requires a lot of RAM processing to convert a file. For Wikidata it will probably require a machine with 100-200GB of RAM. This is unfeasible for me because I don't have such a machine, but if you guys have one to share, I can help setup the rdf2hdt software required to convert Wikidata Turtle to HDT. Thank you. [1] http://www.rdfhdt.org/[http://www.rdfhdt.org/] [2] https://dumps.wikimedia.org/wikidatawiki/entities/[https://dumps.wikimedia.… _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata[https://lists.wikimed…