On Wed, 2006-03-29 at 18:48 +0000, Ævar Arnfjörð Bjarmason wrote:
Thus what I'm asking you to do is go over the test
cases that were
solved with your fix(es) and add new ones that do fail if you didn't
solve the real problem they were supposed to test for (e.g. if it
tested whether the parser can balance tags and you only fixed
<i><b></i></b> or some other things like that), because having a
test
suite that passes when we still have some core parsing problems
doesn't mean that we have a decent parser, it means that the test
suite has become worthless.
Ævar,
i do not agree that the test suite has become worthless- it currently
provides 301 regression tests and a sound specification on what a future
parser needs to do. Any drop-in replacement parser will have to pass the
same tests as the current one, plus more tricky ones.
Having many failing tests to document obvious and known shortcomings
that won't be fixed without a rewrite tends to decrease the motivation
to run the test suite and distracts from important regressions. Who
cares if there are 50 or 52 failing tests? Having all tests pass and
then suddenly two fail generates far more attention.
In my opinion deep structural shortcomings are better documented in
other places (bug reports and design documents come to mind) or in
not-yet-implemented/wishlist tests that are not run by default, but can
be enabled with a switch to test new features or implementations.
In XP, unit tests are mainly a way to enable changes without having to
worry too much about unforeseen consequences- if the tests pass, you
should be fine. Having all tests pass does not mean that the product is
perfect. It only means the product is doing what the tests specify and
nothing about what they don't specify.
--
Gabriel Wicke
MediaWiki hosting, support and development
http://wikidev.net wicke(a)wikidev.net
Phone +49 (0)431-5569001, Mob +49 (0)177 2065127
Eckernförder Str.58, 24116 Kiel