It appears there is no clear consensus in the developer community how to resolve the
problem of the parserTests known-to-fail tests. Consequently, the most useful thing I can
do is provide the patches I have developed and explain how they can be used to implement
one of a number of options. These are neither the only options nor is one of them
necessarily the best. They simply correspond to suggestions made on the wikitech-1 email
list (the second option is additional). There is, of course, the possibility of developing
an option not specified here and implementing it.
[Mark Clements added another option this morning - 7/27/09 - that requires additional
development. If that is what people want, I (or others) can do the necessary
implementation. However, I thought it best to get out what I have done so far in case it
meets consensus needs.]
I have attached the patches to bug 19957 (I couldn't find an appropriate existing bug
for this issue, so I created one. Also, I attached a tar file with 7 patches. When I
looked at the bug after submitting it, it wasn't clear the tar file made it. Any
advice?). These patches can be used to implement the options specified below. Details of
the patches are found in text I have added to the bug report.
+ Install none of these patches and do nothing else:
+ Pros: parserTests behavior remains as currently defined.
+ Cons: parserTests behavior remains as currently defined.
+ Install none of these patches and define successful output as that which the Parser
currently returns (this would require changing parserTests.txt using a unprovided patch):
+ Pros: parserTests is brought into compliance with the test set.
+ Cons: It is generally bad practice to define success without understanding the logic
behind it.
+ Install none of these patches, file bugs against the 14 failing tests and then fix the
bugs:
+ Pros: parserTests is brought into compliance with the test set.
+ Cons: It has been suggested that the parserTests logic is sufficiently complex that it
may be difficult to modify it to pass the known to fail tests.
+ File bugs against the 14 failing tests and patch parserTests.txt to comment out the
known to fail tests:
+ Pros: Testing the Parser using parserTests does not return confusing test results.
Developers can concentrate on those tests that fail due to the addition of new
functionality or intefering bug fixes.
+ Cons: Since the known to fail tests no longer report their presence, it may be easy
for the community to forget they are still a problem.
+ Patch parserTests.txt so the known to fail tests are disabled. Optionally file bugs
against these tests:
+ Pros: Testing the Parser using parserTests does not return confusing test results.
Developers can concentrate on those tests that fail due to the addition of new
functionality or intefering bug fixes.
+ Cons: There are other disabled tests in parserTests.txt. These could become confused
with the known to fail tests.
+ Patch parserTests|.inc|.php to provide a --run-disabled option:
+ Pros: This would allow disabled known to fail tests to run without editing
parserTests.txt
+ Cons: Specifciation of this option runs all disabled tests, not just those known to
fail. Test results using this option might confuse developers.
+ Patch parserTests|.inc|.txt to provide a knowntofail option:
+ Pros: parserTests output and summary statistics indicate which tests succeeded, failed
and returned known-to-fail status.
+ Cons: Creates a new test status for what should be a temporary problem. This is a
questionable software architecture decision.
+ Patch parserTests.php to support a ktf-in-fail option:
+ Pros: those of the previous option. Adds the ability to accumulate ktf failure status
in failure statistics.
+ Cons: those of the previous option. Raises the question of what is the proper
statistics strategy when the known-to-fail option is missing (on runs with this option
missing, known-to-fail tests accumulate against success statistics).
The provided patches do not address whether the 'testrun' and 'testitem'
tables should be modifed to note the specification of statistics changing application
options (i.e., either --ktf-in-fail or --run-disabled) or the use of the knowntofail
option for a particular test. These tables are not modified by the patches.