Magnus Manske wrote:
If we'd limit ourselves to PHP, we could avoid
XML, but PHP is not very
well suited for this kind of text parsing, even with regexps, especially
when it comes to performance. Also, using XML as an itermediate at one
point opens potential for exchange with other services, either for
output (generating static dumps, statistic analyses, etc.) or input
(e.g., using other wiki markup in MediaWiki, for other projects).
I can sympathise with this line of reasoning. The lexing and parsing
needs to be done in C/C++ for efficiency, but then the generated parse
tree should be processed and compiled by PHP, so we need to "transfer"
it, and XML does that job.
The one I'm currently working on is
"manual" C++, but I see no reason
not to use an parser-generator. Question: Can a parser-generator ensure
the output (no matter the input) is valid XML?
Krzysztof Kowalczyk already tried to explain this, but I'll put it in
different words. The parser generator doesn't "ensure" the output is
valid XML; the programmer has to do that. But it's easy. A parser
generator generates the code to turn wikitext into a parse tree; to go
from a parse tree to a valid-XML representation of that parse tree is
like 1+1=2. It's trivial.
Do you know what a parse tree is? If not, let me know, and I'll try to
explain that to you too.
Can it remove potential harmful HTML tags?
Don't think of it as "removing". Don't think of the whole process as
turning one text directly into another. It's not like that.
You give a parser generator a "grammar" (see [[formal grammar]]). If
this grammar says that "<marquee>" isn't a syntax element, then it
will
be interpreted as text, and stored as such in the parse tree. Later
(much later), when the parse tree is compiled into actual (X)HTML, the
text would (obviously) be HTML-escaped. So it would become <marquee>.
Alternatively, of course, one can explicitly define <marquee> to be a
null syntax element in the grammar, so that the final output doesn't
contain it. But that shouldn't be necessary.
The chain would then be
preprocessing-parsing-XML-XHTML.
Uhm... no. See:
http://mail.wikipedia.org/pipermail/wikitech-l/2004-August/012135.html
where I describe the process.
You don't want to do anything to the text before parsing it (assuming
here that parsing includes lexing, although technically they're separate
steps). You want to do all processing after parsing.
Why? Well, because this is the purpose of parsing. We want to turn the
text into a data structure that computers can handle better than text.
It is much easier and much less error-prone to say "if this object is a
tree node representing template-inclusion, then do this" than to say
"search the string for some fuzzy pattern that looks a bit like a
template inclusion, but look out for nesting, and make sure you get the
parameters right, because they might contain pipes inside piped links,
and try not to mess things up."
The XML could be cached, as all changes influenced by
user options would
happen only in the final step.
That is correct! But this XML would still be just the parse tree for the
wiki text.
Caveat: Cache will have to be invalidated
for variables and templates that change (e.g., {{NUMBEROFARTICLES}} and
edited templates).
This is obvious. We already have to invalidate the cache for everything
that changes.
However, currently we also need to invalidate the parser cache for pages
that include a template we have edited. This should not be necessary. We
should be able to retrieve the parse tree for a page independently of
that for the included templates. To put them together at page-view time
is not costly, because we have to compile the parse tree into HTML
anyway (which is also not costly, but it means one sweep through the
parse tree).
Once the parser is basically working, we can [...]
run some benchmarks against the current parser. That
should give us some more facts to base a decision on.
Personally, I'm almost inclined to say that having a proper parser is
more important than performance. :-) But I'm confident that it will
outperform the current parser by far.
Timwi