> we have been facing the same issue, because we need to upload 20 Millons
> items, and I woder if you were able to update those tables at least?,
> perhaps we could work in the others together?
>
I have now been able to insert the data directly into the database.
The process consists of these steps:
- generate the data for an item in JSON
- determine the next Q number and update the JSON item data accordingly
- insert data into the various database tables
There are several foreign keys in the database. Therefore, you can
a) use MySQL's auto-increment and read the assigned ids after an insert
b) assume that you are the only process writing to the database and
assign the key in advance
I have written a Java program to perform these steps. The process is
still terrible slow. Generating and updating the JSON item data is
blazing fast. I can process thousands of items per seconds.
The bottleneck is the MySQL database. Up to now I have used the standard
docker-compose setup. With the Java program I can write only one item
per second.
So I have tried option b) with pre-defined ids and wrote the SQL
statements into a text file. I fed these SQL statements into the MySQL
command line tool. But after two hours I lost patience.
Now I will try to optimize the MySQL database setup.
Jesper
*Apologies for cross-posting!*
Wikimedia Deutschland is interested in having an overview of what’s
happening with Wikibase users and thus, we are running a survey to learn
how we can advance engagement between the software development team
and the people
who are using or are considering using Wikibase.
If you would like to participate, please use this link
<https://forms.gle/McgkdKzDc4BgWokG9> (Google Forms, estimated fill-in time
6min). If you don’t want to use Google Forms, you can also send me a
private email with your answers. We would love to get as many answers as
possible before Saturday, June 6th.
The participation is anonymous and there are no required answers. The data
will only be shared in an aggregated form.
If you have any questions, feel free to reach out to me directly.
Cheers,
--
Mohammed Sadat
*Community Communications Manager for Wikidata/Wikibase*
Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de
Thanks for your feedback. The moderation of the Wikibaseug list
<https://lists.wikimedia.org/mailman/listinfo/wikibaseug> is taken care of
by volunteers, and we don’t have direct access to the moderation tools. I'm
copying the Admins of the list who may have some solutions to your request.
Cheers,
Mohammed
On Wed, Jun 3, 2020 at 6:23 AM Luis Enrique Ramos García <
luisenriqueramos1977(a)googlemail.com> wrote:
> Dear Mohammed Sadat
>
>
> I just answered the survey, but i wanted to make a special remark about an
> issue I I have had with the mailing list. The issue is that beside of being
> register in the mailing list with my personal an corporative emails, every
> time I try to send an email to the list, the system rejects the message,
> saying that I am not registered in the list, and when I try to register in
> the list, it says I can not because I am just registered, which is true.
>
> I wonder if somebody else is having this kind of issue, but in this
> scenario I have had difficulties to participate, to provide support ofr to
> request it.
>
> Hope somebody in your team could check this.
>
>
> Best regards
>
>
> Luis Ramos
>
>
> El mar., 2 jun. 2020 a las 23:54, Mohammed Sadat Abdulai (<
> mohammed.sadat_ext(a)wikimedia.de>) escribió:
>
>> *Apologies for cross-posting!*
>>
>> Wikimedia Deutschland is interested in having an overview of what’s
>> happening with Wikibase users and thus, we are running a survey to learn
>> how we can advance engagement between the software development team and the people
>> who are using or are considering using Wikibase.
>>
>> If you would like to participate, please use this link
>> <https://forms.gle/McgkdKzDc4BgWokG9> (Google Forms, estimated fill-in
>> time 6min). If you don’t want to use Google Forms, you can also send me a
>> private email with your answers. We would love to get as many answers as
>> possible before Saturday, June 6th.
>>
>> The participation is anonymous and there are no required answers. The
>> data will only be shared in an aggregated form.
>>
>> If you have any questions, feel free to reach out to me directly.
>>
>> Cheers,
>> --
>>
>> Mohammed Sadat
>>
>> *Community Communications Manager for Wikidata/Wikibase*
>>
>> Wikimedia Deutschland e.V.
>> Tempelhofer Ufer 23-24
>> 10963 Berlin
>> www.wikimedia.de
>> _______________________________________________
>> Wikibaseug mailing list
>> Wikibaseug(a)lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikibaseug
>>
>
--
Mohammed Sadat Abdulai
*Community Communications Manager for Wikidata/Wikibase*
Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de