Getting started hacking parsoid

List overview All Threads
Download

newer

older

Parsoid blog post & job opening

A grab bag of parsoid issues

Andrew Dunbar

25 Feb 2013 25 Feb '13

6:03 p.m.

I'd like to get involved in the parsoid effort. I've been hanging out in the IRC channel on freenode but there's usually just a dozen lurkers and no action. In the mailing lists parsoid seems to be mentioned about as often in wikitext-l and wikitech-l - which one is best to ask questions of this nature? I'd like to scratch my own itch rather than necessarily go after things on the todo list and roadmap. Basically I'm interested in what parsoid can do for parsing wikitext markup into HTML (or other formats). I want to use it without a mediawiki install and without an internet connection. I see there is already some kind of support for reading in articles from compressed dump files. Any suggestions where I should start or where I can hang out to chat live with people who could help getting me involved? Andrew Dunbar (hippietrail)

Attachments:

attachment.htm (text/html — 1.1 KB)

Show replies by date

Subramanya Sastry

25 Feb 25 Feb

9:46 p.m.

Hi Andrew, On 02/25/2013 06:03 AM, Andrew Dunbar wrote:

...

I'd like to get involved in the parsoid effort.

That is great. We would love to have more people involved.

...

I've been hanging out in the IRC channel on freenode but there's usually just a dozen lurkers and no action.

We are all around on #mediawiki-parsoid (subbu -- me, gwicke -- Gabriel Wicke, marktraceur -- Mark Holmquist). Over the last couple weeks, we've got another active contributor cscott (C.Scott Ananian). There are visualeditor folks also hanging around who participate in discussions once in a while. On most weekdays, there is a fair bit of activity, except for occasional periods of lull. I assume you probably caught a period of lull. So, come around and say hi.

...

In the mailing lists parsoid seems to be mentioned about as often in wikitext-l and wikitech-l - which one is best to ask questions of this nature? I'd like to scratch my own itch rather than necessarily go after things on the todo list and roadmap. Basically I'm interested in what parsoid can do for parsing wikitext markup into HTML (or other formats). I want to use it without a mediawiki install and without an internet connection. I see there is already some kind of support for reading in articles from compressed dump files.

You and Scott have similar goals then since he is interested in using Parsoid in situations without internet or spotty internet as well, if I understand it correctly. In this context, he has been doing some code cleanup, handling some edge cases, and contributing patches.

...

Any suggestions where I should start or where I can hang out to chat live with people who could help getting me involved?

It is defintely faster on IRC, but wikitext-l is good for Parsoid-specific technical questions. Subbu.

...

Andrew Dunbar (hippietrail) _______________________________________________ Wikitext-l mailing list Wikitext-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitext-l

C. Scott Ananian

10:46 p.m.

On Mon, Feb 25, 2013 at 10:46 AM, Subramanya Sastry <ssastry(a)wikimedia.org>wrote;wrote:

...

On most weekdays, there is a fair bit of activity, except for occasional periods of lull. I assume you probably caught a period of lull.

The parsoid team is mostly based in California, and tend to be online PST working hours, ie noon-8pm EST on weekdays. That's the best time to contact people. Some members of the team use an IRC bot to let them catch up on backlog when they're not online, but not all, so if you don't get a response it's usually best to either send an email to wikitext or else just wait until the next 'working day'. You and Scott have similar goals then since he is interested in using

...

Parsoid in situations without internet or spotty internet as well, if I understand it correctly. In this context, he has been doing some code cleanup, handling some edge cases, and contributing patches.

There are instructions on the wiki for cloning the git repo of parsoid and running your own instance. I've also successfully set up an instance on a free appfog server; let me know if you're interested in that (and maybe I should think about merging those patches to make it easier for others). My embryonic offline wikibrowser is at http://github.com/cscott/nell-wikipedia ; you can try it out at http://nell-wikipedia.github.cscott.net, although the offline stuff isn't implemented yet, it's mostly a showcase for how well (or not) Parsoid renders the wikitext. If you're interested in offline wikibrowser tech, I'd love to have help. The novel thing I'd like to do here is allow offline wiki editing, snarfing in some of the visual editor codebase to do so with panache. One Laptop per Child has over two million laptops in the field with offline copies of spanish and english wikipedia, but the current app is read-only -- which defeats the purpose of a wiki, really. The current app's snapshot is also out of date, because the original version required a lot of offline tools to grunge over a wikipedia xml dump to make a new snapshot; my goal is for this next version of the app to be able to update its snapshot itself whenever it's online. --scott

Andrew Dunbar

26 Feb 26 Feb

6:19 a.m.

Here's some feedback from the point of view of sombody trying to use Parsoid for the first time. Hopefully it can be taken as constructive to help make it easier for new people to get involved, sorry if some of it will seem nitpicky ... - MediaWiki and node.js are cross-platform. Some of us are crazy enough to use it on Windows but the docs are Unix-centric - npm works great on Windows but stuff like "curlhttps:// npmjs.org/install.sh | sudo sh" does not - Can parsoid currently be used without a local mediawiki install? this instruction makes it seem that it cannot yet talking to people make it seem like it should work. Confusing: Go to the Parsoid/js/api directory and create a localsettings.js file based on localsettings.js.example. In particular, make sure that you useparsoidConfig.setInterwiki to point to the MediaWiki instance(s) you want to use. - Can you point to a remote MediaWiki instance or must it be local? Can you leave it out? - export NODE_PATH=node_modules is Unix-centric - what should Windows users do? - The docs jump straight into "running the tests" - this doesn't seem very interesting to people trying out parsoid for the first time - at least without much introduction. - It seems too "internal". Why not start by showing people how to convert a wikitext string directly from a command line? That's more likely to get somebody hooked. - "npm test" does seem to work great on Windows without "export NODE_PATH=node_modules" - my test results were (with a week old repo): 1854 total passed tests, 1107 total failures - I think we should downplay stuff that makes it look like a mediawiki install is needed - I think we can promote that it should work cross-platform Andrew Dunbar (hippietrail)

Gabriel Wicke

6:45 a.m.

Andrew,] thanks for your feedback! On 02/25/2013 04:19 PM, Andrew Dunbar wrote:

...

- MediaWiki and node.js are cross-platform. Some of us are crazy enough to use it on Windows but the docs are Unix-centric - npm works great on Windows but stuff like "curlhttps://npmjs.org/install.sh <http://npmjs.org/install.sh> | sudo sh" does not

I personally have not used Windows in a while, but if you have corresponding instructions for Windows then it would be great if you could add them. Perhaps in a subpage like http://www.mediawiki.org/wiki/Parsoid/Windows?

...

- Can parsoid currently be used without a local mediawiki install? this instruction makes it seem that it cannot yet talking to people make it seem like it should work. Confusing:

these instructions were very geared to an installation with the VisualEditor for a local wiki. I have reworded it a bit to make it clear that you can run the web service without any config.

...

- Can you point to a remote MediaWiki instance or must it be local? Can you leave it out?

You can just try the built-in wikis (see example links on the API front page) or test snippets using the forms.

...

- export NODE_PATH=node_modules is Unix-centric - what should Windows users do?

This is probably no longer needed on Unix/Linux either. I have never used it and never had problems. Removed it from the documentation.

...

- The docs jump straight into "running the tests" - this doesn't seem very interesting to people trying out parsoid for the first time - at least without much introduction. - It seems too "internal". Why not start by showing people how to convert a wikitext string directly from a command line? That's more likely to get somebody hooked.

Good point. Will add something to that effect.

...

- my test results were (with a week old repo): 1854 total passed tests, 1107 total failures

That means that Windows works just as well as Linux, which is great given that we have never tested it on Windows! Gabriel

4098

days inactive

4099

days old

wikitext-l@lists.wikimedia.org

Manage subscription

4 comments

4 participants

tags (0)

participants (4)

Andrew Dunbar
C. Scott Ananian
Gabriel Wicke
Subramanya Sastry