33.3. Variant Comparison Files
Since some of the tests inherently produce environment-dependent results, we have provided ways to specify alternate " expected " result files. Each regression test can have several comparison files showing possible results on different platforms. There are two independent mechanisms for determining which comparison file is used for each test.
The first mechanism allows comparison files to be selected for
specific platforms. There is a mapping file,
, that defines
which comparison file to use for each platform.
To eliminate bogus test
for a particular platform,
you first choose or make a variant result file, and then add a line to the
Each line in the mapping file is of the form
The test name is just the name of the particular regression test
module. The output value indicates which output file to check. For the
standard regression tests, this is always
value corresponds to the file extension of the output file.
The platform pattern is a pattern in the style of the Unix
(that is, a regular expression with an implicit
anchor at the start). It is matched against the
platform name as printed by
The comparison file name is the base name of the substitute result
For example: some systems lack a working
for which our workaround causes rounding errors in the
Therefore, we provide a variant comparison file,
, which includes
the results to be expected on these systems. To silence the bogus
which will trigger on any machine where the output of
Other lines in
select the variant comparison
file for other platforms where it's appropriate.
The second selection mechanism for variant comparison files is
much more automatic: it simply uses the
several supplied comparison files. The regression test driver
script considers both the standard comparison file for a test,
is any single digit
). If any such file is an exact match,
the test is considered to pass; otherwise, the one that generates
the shortest diff is used to create the failure report. (If
includes an entry for the particular
test, then the base
is the substitute
name given in
For example, for the
test, the comparison file
contains results that are expected
contains results sorted as
they appear in many other locales.
The best-match mechanism was devised to cope with locale-dependent results, but it can be used in any situation where the test results cannot be predicted easily from the platform name alone. A limitation of this mechanism is that the test driver cannot tell which variant is actually " correct " for the current environment; it will just pick the variant that seems to work best. Therefore it is safest to use this mechanism only for variant results that you are willing to consider equally valid in all contexts.