Return to README CVS log

Up to [cvs.NetBSD.org] / src / regress

Annotation of src/regress/README, Revision 1.1

1.1     ! martin      1: $NetBSD$
        !             2:
        !             3: This part of the source tree contains regression tests. There are special
        !             4: make targets and rules to follow. Most of these, however, are currently not
        !             5: enforced, and most tests available are not conforming.
        !             6:
        !             7: We hope to fix this someday. If you add new tests, please try to be conforming.
        !             8:
        !             9: What is a regression test?
        !            10:
        !            11:   A regression test is run by a makefile in a test directory (see below).
        !            12:   Each makefile may run multiple tests.
        !            13:
        !            14: What is a test directory?
        !            15:
        !            16:   A directory in this part of the tree is a regression test directory. It
        !            17:   contains a Makefile which implements the additional "regress" target,
        !            18:   and runs all it's regression tests during this target.
        !            19:
        !            20: May the make progress be stopped on failures?
        !            21:
        !            22:   No, the "make regress" target should succeed, unless some regression
        !            23:   binaries could not be build, disk is full or other catastrophic failures
        !            24:   outside of the tested subsystem happen. A failing regression test should
        !            25:   log the failure (see below), but not make the target itself fail.
        !            26:
        !            27: What are the possible results of a regression test?
        !            28:
        !            29:   A test may either
        !            30:
        !            31:   - succeed, in which case it logs "PASSED" (see below for logging details)
        !            32:   - fail, in which case it logs "FAILED"
        !            33:   - not be able to run, in which case it logs "SKIPPED" and the reason
        !            34:     for the skip in the comment field (see below)
        !            35:
        !            36:   Typical reasons for tests to not being run are missing kernel options,
        !            37:   or missing privileges (test needs root, "make regress" is invoked by
        !            38:   mere mortal or vice versa). A test may not fail because of such
        !            39:   environmental issues, it must detect and properly log the problem.
        !            40:
        !            41:   If a test directory contains tests that may be skipped, it should have
        !            42:   a README file explaining the prerequisites (e.g. needed kernel options)
        !            43:
        !            44:   In future, we will mark affected makefiles and optimize run/skipped test
        !            45:   during repeated runs with differing privileges - but currently there is no
        !            46:   make framework in place to handle this.
        !            47:
        !            48: How and when does a test log results?
        !            49:
        !            50:   If the make/environment variable ${REGRESS_LOG} is defined, the final
        !            51:   results (and only those) should be logged to the file named by that
        !            52:   variable. We will, in the future, add make targets for this purpose.
        !            53:   The log format is line oriented, one line used per test. Each line
        !            54:   consists of the directory where the Makefile lives, followed by the
        !            55:   test name and the result (see above: PASSED, FAILED, SKIPPED).
        !            56:   Following this an optional comment may be added. For SKIPPED tests the
        !            57:   comment is not optional. Fields are separated by spaces.