Annotation of htdocs/docs/pkgsrc/regression.html, revision 1.26

1.18      sevan       1: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
1.1       dsieger     2: <html>
                      3: <head>
                      4: <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
1.25      rillig      5: <title>Chapter 26. Regression tests</title>
1.7       spz         6: <link rel="stylesheet" type="text/css" href="/global.css">
1.1       dsieger     7: <meta name="generator" content="DocBook XSL Stylesheets VX.X.X">
1.5       rillig      8: <link rel="home" href="index.html" title="The pkgsrc guide">
1.1       dsieger     9: <link rel="up" href="infrastructure.html" title="Part III. The pkgsrc infrastructure internals">
1.25      rillig     10: <link rel="prev" href="infr.design.html" title="Chapter 25. Design of the pkgsrc infrastructure">
                     11: <link rel="next" href="porting.html" title="Chapter 27. Porting pkgsrc">
1.1       dsieger    12: </head>
                     13: <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
                     14: <div class="navheader">
                     15: <table width="100%" summary="Navigation header">
1.25      rillig     16: <tr><th colspan="3" align="center">Chapter 26. Regression tests</th></tr>
1.1       dsieger    17: <tr>
                     18: <td width="20%" align="left">
                     19: <a accesskey="p" href="infr.design.html">Prev</a> </td>
                     20: <th width="60%" align="center">Part III. The pkgsrc infrastructure internals</th>
                     21: <td width="20%" align="right"> <a accesskey="n" href="porting.html">Next</a>
                     22: </td>
                     23: </tr>
                     24: </table>
                     25: <hr>
                     26: </div>
1.8       asau       27: <div class="chapter">
1.1       dsieger    28: <div class="titlepage"><div><div><h2 class="title">
1.25      rillig     29: <a name="regression"></a>Chapter 26. Regression tests</h2></div></div></div>
1.1       dsieger    30: <div class="toc">
                     31: <p><b>Table of Contents</b></p>
1.15      leot       32: <dl class="toc">
1.25      rillig     33: <dt><span class="sect1"><a href="regression.html#regression.run">26.1. Running the regression tests</a></span></dt>
                     34: <dt><span class="sect1"><a href="regression.html#regression.new">26.2. Adding a new regression test</a></span></dt>
1.1       dsieger    35: <dd><dl>
1.25      rillig     36: <dt><span class="sect2"><a href="regression.html#regression.fun.override">26.2.1. Overridable functions</a></span></dt>
                     37: <dt><span class="sect2"><a href="regression.html#regression.fun.helper">26.2.2. Helper functions</a></span></dt>
1.1       dsieger    38: </dl></dd>
                     39: </dl>
                     40: </div>
                     41: <p>The pkgsrc infrastructure consists of a large codebase,
                     42:        and there are many corners where every little bit of a file is
                     43:        well thought out, making pkgsrc likely to fail as soon as
                     44:        anything is changed near those parts. To prevent most changes
                     45:        from breaking anything, a suite of regression tests should go
                     46:        along with every important part of the pkgsrc infrastructure.
                     47:        This chapter describes how regression tests work in pkgsrc and
                     48:        how you can add new tests.</p>
1.8       asau       49: <div class="sect1">
1.1       dsieger    50: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
1.25      rillig     51: <a name="regression.run"></a>26.1. Running the regression tests</h2></div></div></div>
1.26    ! leot       52: <p>You first need to install the <a href="https://cdn.NetBSD.org/pub/pkgsrc/current/pkgsrc/pkgtools/pkg_regress/index.html" target="_top"><code class="filename">pkgtools/pkg_regress</code></a> package, which
1.4       hubertf    53:        provides the <span class="command"><strong>pkg_regress</strong></span> command. Then you
1.1       dsieger    54:        can simply run that command, which will run all tests in the
1.21      rillig     55:        <code class="filename">regress/</code> directory.</p>
1.1       dsieger    56: </div>
1.8       asau       57: <div class="sect1">
1.1       dsieger    58: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
1.25      rillig     59: <a name="regression.new"></a>26.2. Adding a new regression test</h2></div></div></div>
1.21      rillig     60: <p>Every directory in the <code class="filename">regress/</code>
                     61:        directory that contains a file called <code class="filename">spec</code>
1.1       dsieger    62:        is considered a regression test. This file is a shell program
1.4       hubertf    63:        that is included by the <span class="command"><strong>pkg_regress</strong></span> command.
1.1       dsieger    64:        The following functions can be overridden to suit your
                     65:        needs.</p>
1.8       asau       66: <div class="sect2">
1.1       dsieger    67: <div class="titlepage"><div><div><h3 class="title">
1.25      rillig     68: <a name="regression.fun.override"></a>26.2.1. Overridable functions</h3></div></div></div>
1.16      rillig     69: <p>These functions do not take any parameters. Although they
                     70:        are called in <span class="quote">&#8220;<span class="quote">set -e</span>&#8221;</span> mode, they don't stop at the
1.23      leot       71:        first failing command. See <a class="ulink" href="https://stackoverflow.com/q/4072984" target="_top">this Stack Overflow
1.16      rillig     72:        question</a> for details.</p>
1.8       asau       73: <div class="variablelist"><dl class="variablelist">
1.16      rillig     74: <dt><span class="term"><code class="varname">do_setup</code></span></dt>
1.1       dsieger    75: <dd><p>This function prepares the environment for the
                     76:        test. By default it does nothing.</p></dd>
1.16      rillig     77: <dt><span class="term"><code class="varname">do_test</code></span></dt>
                     78: <dd>
                     79: <p>This function runs the actual test. By default,
1.1       dsieger    80:        it calls <code class="varname">TEST_MAKE</code> with the arguments
                     81:        <code class="varname">MAKEARGS_TEST</code> and writes its output including
                     82:        error messages into the file
1.16      rillig     83:        <code class="varname">TEST_OUTFILE</code>.</p>
                     84: <p>When defining this function, make sure that all output that
                     85:        needs to be checked is written to the correct output file.
                     86:        Example:</p>
                     87: <pre class="programlisting">
                     88: do_test() {
1.17      rillig     89:         echo "Example output"
1.16      rillig     90: } 1&gt;$TEST_OUTFILE 2&gt;&amp;1
                     91: </pre>
                     92: </dd>
                     93: <dt><span class="term"><code class="varname">check_result</code></span></dt>
                     94: <dd>
                     95: <p>This function is run after the test and is
1.1       dsieger    96:        typically used to compare the actual output from the one that is
                     97:        expected. It can make use of the various helper functions from
1.16      rillig     98:        the next section. Example:</p>
                     99: <pre class="programlisting">
                    100: check_result() {
1.17      rillig    101:         exit_status 0
                    102:         output_require "Example"
                    103:         output_require "^[[:alpha:]+[[:space:]][[:alpha:]]{6}$"
                    104:         output_prohibit "no such file or directory"
1.22      rillig    105:         regress_fail "expected $expected but got $actual for input $input"
1.16      rillig    106: }
                    107: </pre>
                    108: </dd>
                    109: <dt><span class="term"><code class="varname">do_cleanup</code></span></dt>
1.1       dsieger   110: <dd><p>This function cleans everything up after the
                    111:        test has been run. By default it does nothing.</p></dd>
                    112: </dl></div>
                    113: </div>
1.8       asau      114: <div class="sect2">
1.1       dsieger   115: <div class="titlepage"><div><div><h3 class="title">
1.25      rillig    116: <a name="regression.fun.helper"></a>26.2.2. Helper functions</h3></div></div></div>
1.8       asau      117: <div class="variablelist"><dl class="variablelist">
1.21      rillig    118: <dt><span class="term"><code class="varname">regress_fail <em class="replaceable"><code>message...</code></em></code></span></dt>
                    119: <dd><p>This function makes the test fail with the given error message.</p></dd>
1.16      rillig    120: <dt><span class="term"><code class="varname">exit_status expected</code></span></dt>
1.1       dsieger   121: <dd><p>This function compares the exitcode of the
1.16      rillig    122:        <span class="command"><strong>do_test</strong></span> function with its first parameter.
1.1       dsieger   123:        If they differ, the test will fail.</p></dd>
1.16      rillig    124: <dt><span class="term"><code class="varname">output_require regex...</code></span></dt>
                    125: <dd>
                    126: <p>This function checks for each of its parameters
                    127:        if the output from <span class="command"><strong>do_test</strong></span> matches the
1.1       dsieger   128:        extended regular expression. If it does not, the test will
1.16      rillig    129:        fail. Example:</p>
                    130: <pre class="programlisting">
                    131: output_require "looks fine"
                    132: output_require "^[[:alpha:]+[[:space:]][[:alpha:]]{6}$"
                    133: </pre>
                    134: </dd>
1.24      leot      135: <dt><span class="term"><code class="varname">output_prohibit regex...</code></span></dt>
1.1       dsieger   136: <dd><p>This function checks for each of its parameters
1.4       hubertf   137:        if the output from <span class="command"><strong>do_test()</strong></span> does
1.1       dsieger   138:        <span class="emphasis"><em>not</em></span> match the extended regular expression.
                    139:        If any of the regular expressions matches, the test will
                    140:        fail.</p></dd>
                    141: </dl></div>
                    142: </div>
                    143: </div>
                    144: </div>
                    145: <div class="navfooter">
                    146: <hr>
                    147: <table width="100%" summary="Navigation footer">
                    148: <tr>
                    149: <td width="40%" align="left">
                    150: <a accesskey="p" href="infr.design.html">Prev</a> </td>
                    151: <td width="20%" align="center"><a accesskey="u" href="infrastructure.html">Up</a></td>
                    152: <td width="40%" align="right"> <a accesskey="n" href="porting.html">Next</a>
                    153: </td>
                    154: </tr>
                    155: <tr>
1.25      rillig    156: <td width="40%" align="left" valign="top">Chapter 25. Design of the pkgsrc infrastructure </td>
1.1       dsieger   157: <td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td>
1.25      rillig    158: <td width="40%" align="right" valign="top"> Chapter 27. Porting pkgsrc</td>
1.1       dsieger   159: </tr>
                    160: </table>
                    161: </div>
                    162: </body>
                    163: </html>

CVSweb <webmaster@jp.NetBSD.org>