We had a discussion today at the MediaWiki Stakeholders' Group membership meeting about the current status of development on the new WikiApiary as well as future directions. Below is a brief summary as well as a request for feedback.
We have created a new database schema to support WikiApiary, scripts to populate the data, as well as a MediaWiki extension to expose the data to a wiki. We will be making that extension available on gerrit very soon. It currently temporarily resides at https://github.com/cicalese/NewWikiApiary.
We exported the wiki URLs found in the old WikiApiary and imported them into a new wiki, which resides at https://wikiapiary.wmcloud.org hosted by Wikimedia Cloud Services. There were approximately 96,000 wiki URLs retrieved from the old WikiApiary. Or those, about 90% are Fandom wikis. There may have been URLs that were missed in the export, despite the fact that we found far more URLs than the approximately 46,000 that the old WikiApiary reported as active. We will be making available a mechanism to contribute wiki URLs both through the wiki user interface as well as through an API.
We created pages for the imported wikis in the new wiki with page titles that consist of the wiki sitename followed by its language code in parenthesis. In the approximately 500 cases where this resulted in naming collisions, alternative page titles were used. Because of this naming scheme, page titles differ between the old WikiApiary and the new WikiApiary.
We have done an initial scrape of all of those wikis to get statistics and configuration information through the MediaWiki action API, as was done in the old WikiApiary, storing the data in the database rather than in subpages in the wiki. It takes roughly 24 hours to scrape approximately 96000 wikis. We have begun work on the front end of the new wiki to render the data. We are taking advantage of the extension information available at mediawiki.org in Module:ExtensionJson as well.
The old WikiApiary is still accessible in a read only mode at https://wikiapiary.com. It is hosted on an Azure instance but will be moving as-is to a Wikimedia Cloud Services instance. It will be maintained for the near future to allow the community to identify features that should be made available in the new WikiApiary. At some point, when it has enough capabilities, the new WikiApiary will become available at https://wikiapiary.com, while the old WikiApiary will continue to be available at another URL. We anticipate that the old WikiApiary will be retired at some point, but not before we have had plenty of time to migrate useful functionality.
This is very much a work in progress, but we have gotten far enough along in the development to be reassured that this new data model will be much more performant that that in the old WikiApiary. At this point it would be helpful to have feedback from the WikiApiary community, especially on a few key upcoming decisions. In addition, we welcome volunteers to contribute to the effort. Some items we will need to decide:
1. We currently have account creation on the new wiki disabled to avoid being besieged by spam account creations. We will need to decide what approach to take to account creation and spam prevention. We would very much like to enable account creation so other can contribute.
2. At this point, the main effort has been to gather statistics on wikis and their extensions/skins through MediaWiki action API queries. There is a wealth of information in the old WikiApiary that is not yet incorporated, such as Internet archive information. We welcome volunteers to contribute to any migration that is determined useful.
3. We do not yet have an extension enabled on the wiki for semantic tagging. We are hesitant to enable Semantic MediaWiki at this point, as there appears to be a bug that is preventing properties from being reliably saved. That needs more investigation. We may also consider Cargo or other mechanisms, such as investigating approaches to indexing the data in Elasticsearch. And, we may investigate the relationship to WikiData.
4. We currently have less than 10 extensions/skins enabled on the new wiki. We will need some decisions on what other extensions would be useful.
5. The user interface of the new WikiApiary isn't very spiffy. We welcome volunteers who have a passion for creating appealing, responsive user interfaces to give it a more modern look.
OK, maybe this wasn't such a brief status update. But, hopefully it gives you a sense of where we're at and what opportunities there are to contribute. Several of us will be at the Wikimedia Hackathon in May working on this project, and we would welcome others to collaborate with us there or remotely. In the meantime, we will continue to volunteer what time we can to move this project forward.
A hearty thank you to Mark Hershberger and Charly Cobben who have worked with me on this project to get it this far.
Dear fans of WikiApiary,
After a year or more of limping along, we're taking WikiApiary offline
to rebuild the site.
If you use Wikiapiary or have used Wikiapiary, or have written a bot
that uses Wikiapiary, we need your input on the design.
We need the help of volunteers. We have ideas around how to construct
the site so that it is robust, but we'd like input from the community
for its design.
The reimplementation of Wikiapiary is mainly happening because we
couldn't resolve some SMW issues and its query limitations as well as
just the need to update the wiki.
The site has become unmaintainable and those of us who have been
responsible for the back-end (mainly Cindy and myself) haven't had time
to dedicate to resolving the problems
A major contributing factor to this decision is the ongoing cost of
hosting. Previously, we started to move it back to Wikimedia Cloud
Services, but attempted to upgrade it before moving it while keeping the
design the same.
That ended up causing problems which blocked our progress.
As a result, we've decided we need to rethink Wikiapiary's design
completely and rebuild it from scratch.
I'm interested in your feedback and any ideas you may have.
Thanks,
Mark.
--
Mark A. Hershberger
NicheWork LLC
717-271-1084
Hi, WikiApiary users!
WikiApiary has been in write-only mode and, since I haven't posted any
information, people have been asking what's up.
I would like to bring the site back online as soon as possible, but my
time is quite limited and I need some help. If you can help with any of
the steps under the "A solution" section, please let me know.
The problem
=== =======
The problem seems similar to the one that Justin Lloyd reported[1]
early this year. Cindy left a comment[2] on that issue that gives some
insight into the problems we're seeing on WikiApiary.
There has already been some discussion tonight about WikiApiary and SMW
on the Semantic MediaWiki discussion channel[3].
A solution
= ========
In the immediate term, though, we need to do something to get the site
back online. What we've been doing hasn't been working, so another
approach is needed.
The first idea that I have for doing that is taking an XML dump of the
site and importing it into an older version of SMW. I do not have the
resources to do this, so I need to ask for help from you all.
I *think* the immediate steps are:
1. Create an XML dump of the site.
2. Set up a wiki with an older version of SMW (3.x?) on WMCS and import
the site there.
3. Run rebuildData.
4. Turn the site and the bots back on, pointing to the new WMCS site.
Those steps would get WikiApiary back online.
The Future
=== ======
After that, we could start to think about the long-term fix is to
address the scalability problem with WikiApiary. Doing that means
either fixing the bug in SMW or re-architecting the site so that SMW is
no longer used.
I know this is a lot to digest, but I'm hoping that we can, as a
community, figure out a way to get the project that Jamie Thingelstad
started all those years ago back online.
Mark.
Footnotes:
[1] https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/5392
[2] https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/5392#issuecom…
[3] https://matrix.to/#/!buSEZBEvVNBUdwhvvn:matrix.org/$Xl86P6MWjCE8-4lEzKQdUgy…
--
Mark A. Hershberger
NicheWork LLC
717-271-1084
Hello, fellow Apiary lovers!
Cindy has done a lot of work behind the scenes to get WikiApiary
upgraded to 1.39 and we've moved the worker bees back to Wikimedia Cloud
Services so that they don't compete for CPU and bandwidth with the
website itself.
As a result, I've brought wikiapiary.com back online.
We'd like to ask for help from other people now.
Since a big part of the problem with WikiApiary is database timeouts,
Cindy and I are wondering if those of you with experience with SMW and
ElasticStore would help think changing the back end from SQLStore to
ElasticStore would help.
Of course, we would need some help with this. Ideally, some brave soul
would volunteer to do the conversion, but, if that isn't possible, a
step by step cookbook would be nice.
Finally, we think that the site's appearance could use some work.
Cindy suggested switching to Chameleon (and has it installed as an
option), but there is some general tidying that needs to be done.
Have a beautiful day!
Mark.
--
Mark A. Hershberger
NicheWork LLC
717-271-1084
Hello, I recently translated all the translatable pages on the WikiApiary website to Swedish and realised that the software didn't allow me to publish external links. I got around this by simply not adding the needed external link. If anyone could add these external links for me or fix the bug that prevents me from adding external links that'd be appreciated.
This affected the pages Main Page, User:Bumble Bee and WikiApiary:About.
/Sabelöga