Annotation of htdocs/docs/pkgsrc/regression.html, revision 1.2

1.1       dsieger     1: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
                      2: <html>
                      3: <head>
                      4: <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
                      5: <title>Chapter 24. Regression tests</title>
1.2     ! rillig      6: <link rel="stylesheet" href="/global.css" type="text/css">
1.1       dsieger     7: <meta name="generator" content="DocBook XSL Stylesheets VX.X.X">
                      8: <link rel="start" href="index.html" title="The pkgsrc guide">
                      9: <link rel="up" href="infrastructure.html" title="Part III. The pkgsrc infrastructure internals">
                     10: <link rel="prev" href="infr.design.html" title="Chapter 23. Design of the pkgsrc infrastructure">
                     11: <link rel="next" href="porting.html" title="Chapter 25. Porting pkgsrc">
                     12: </head>
                     13: <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
                     14: <div class="navheader">
                     15: <table width="100%" summary="Navigation header">
                     16: <tr><th colspan="3" align="center">Chapter 24. Regression tests</th></tr>
                     17: <tr>
                     18: <td width="20%" align="left">
                     19: <a accesskey="p" href="infr.design.html">Prev</a> </td>
                     20: <th width="60%" align="center">Part III. The pkgsrc infrastructure internals</th>
                     21: <td width="20%" align="right"> <a accesskey="n" href="porting.html">Next</a>
                     22: </td>
                     23: </tr>
                     24: </table>
                     25: <hr>
                     26: </div>
                     27: <div class="chapter" lang="en">
                     28: <div class="titlepage"><div><div><h2 class="title">
                     29: <a name="regression"></a>Chapter 24. Regression tests</h2></div></div></div>
                     30: <div class="toc">
                     31: <p><b>Table of Contents</b></p>
                     32: <dl>
                     33: <dt><span class="sect1"><a href="regression.html#regression.descr">24.1. The regression tests framework</a></span></dt>
                     34: <dt><span class="sect1"><a href="regression.html#regression.run">24.2. Running the regression tests</a></span></dt>
                     35: <dt><span class="sect1"><a href="regression.html#regression.new">24.3. Adding a new regression test</a></span></dt>
                     36: <dd><dl>
                     37: <dt><span class="sect2"><a href="regression.html#regression.fun.override">24.3.1. Overridable functions</a></span></dt>
                     38: <dt><span class="sect2"><a href="regression.html#regression.fun.helper">24.3.2. Helper functions</a></span></dt>
                     39: </dl></dd>
                     40: </dl>
                     41: </div>
                     42: <p>The pkgsrc infrastructure consists of a large codebase,
                     43:        and there are many corners where every little bit of a file is
                     44:        well thought out, making pkgsrc likely to fail as soon as
                     45:        anything is changed near those parts. To prevent most changes
                     46:        from breaking anything, a suite of regression tests should go
                     47:        along with every important part of the pkgsrc infrastructure.
                     48:        This chapter describes how regression tests work in pkgsrc and
                     49:        how you can add new tests.</p>
                     50: <div class="sect1" lang="en">
                     51: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
                     52: <a name="regression.descr"></a>24.1. The regression tests framework</h2></div></div></div>
                     53: <p></p>
                     54: </div>
                     55: <div class="sect1" lang="en">
                     56: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
                     57: <a name="regression.run"></a>24.2. Running the regression tests</h2></div></div></div>
                     58: <p>You first need to install the <a href="ftp://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc/pkgtools/pkg_regress/README.html" target="_top"><code class="filename">pkgtools/pkg_regress</code></a> package, which
                     59:        provides the <span><strong class="command">pkg_regress</strong></span> command. Then you
                     60:        can simply run that command, which will run all tests in the
                     61:        <code class="filename">regress</code> category.</p>
                     62: </div>
                     63: <div class="sect1" lang="en">
                     64: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
                     65: <a name="regression.new"></a>24.3. Adding a new regression test</h2></div></div></div>
                     66: <p>Every directory in the <code class="filename">regress</code>
                     67:        category that contains a file called <code class="filename">spec</code>
                     68:        is considered a regression test. This file is a shell program
                     69:        that is included by the <span><strong class="command">pkg_regress</strong></span> command.
                     70:        The following functions can be overridden to suit your
                     71:        needs.</p>
                     72: <div class="sect2" lang="en">
                     73: <div class="titlepage"><div><div><h3 class="title">
                     74: <a name="regression.fun.override"></a>24.3.1. Overridable functions</h3></div></div></div>
                     75: <p>These functions do not take any parameters. They are all
                     76:        called in &#8220;<span class="quote">set -e</span>&#8221; mode, so you should be careful
                     77:        to check the exitcodes of any commands you run in the
                     78:        test.</p>
                     79: <div class="variablelist"><dl>
                     80: <dt><span class="term"><code class="varname">do_setup()</code></span></dt>
                     81: <dd><p>This function prepares the environment for the
                     82:        test. By default it does nothing.</p></dd>
                     83: <dt><span class="term"><code class="varname">do_test()</code></span></dt>
                     84: <dd><p>This function runs the actual test. By default,
                     85:        it calls <code class="varname">TEST_MAKE</code> with the arguments
                     86:        <code class="varname">MAKEARGS_TEST</code> and writes its output including
                     87:        error messages into the file
                     88:        <code class="varname">TEST_OUTFILE</code>.</p></dd>
                     89: <dt><span class="term"><code class="varname">check_result()</code></span></dt>
                     90: <dd><p>This function is run after the test and is
                     91:        typically used to compare the actual output from the one that is
                     92:        expected. It can make use of the various helper functions from
                     93:        the next section.</p></dd>
                     94: <dt><span class="term"><code class="varname">do_cleanup()</code></span></dt>
                     95: <dd><p>This function cleans everything up after the
                     96:        test has been run. By default it does nothing.</p></dd>
                     97: </dl></div>
                     98: </div>
                     99: <div class="sect2" lang="en">
                    100: <div class="titlepage"><div><div><h3 class="title">
                    101: <a name="regression.fun.helper"></a>24.3.2. Helper functions</h3></div></div></div>
                    102: <div class="variablelist"><dl>
                    103: <dt><span class="term"><code class="varname">exit_status(expected)</code></span></dt>
                    104: <dd><p>This function compares the exitcode of the
                    105:        <span><strong class="command">do_test()</strong></span> function with its first parameter.
                    106:        If they differ, the test will fail.</p></dd>
                    107: <dt><span class="term"><code class="varname">output_require(regex...)</code></span></dt>
                    108: <dd><p>This function checks for each of its parameters
                    109:        if the output from <span><strong class="command">do_test()</strong></span> matches the
                    110:        extended regular expression. If it does not, the test will
                    111:        fail.</p></dd>
                    112: <dt><span class="term"><code class="varname">output_prohibit(regex...)</code></span></dt>
                    113: <dd><p>This function checks for each of its parameters
                    114:        if the output from <span><strong class="command">do_test()</strong></span> does
                    115:        <span class="emphasis"><em>not</em></span> match the extended regular expression.
                    116:        If any of the regular expressions matches, the test will
                    117:        fail.</p></dd>
                    118: </dl></div>
                    119: </div>
                    120: </div>
                    121: </div>
                    122: <div class="navfooter">
                    123: <hr>
                    124: <table width="100%" summary="Navigation footer">
                    125: <tr>
                    126: <td width="40%" align="left">
                    127: <a accesskey="p" href="infr.design.html">Prev</a> </td>
                    128: <td width="20%" align="center"><a accesskey="u" href="infrastructure.html">Up</a></td>
                    129: <td width="40%" align="right"> <a accesskey="n" href="porting.html">Next</a>
                    130: </td>
                    131: </tr>
                    132: <tr>
                    133: <td width="40%" align="left" valign="top">Chapter 23. Design of the pkgsrc infrastructure </td>
                    134: <td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td>
                    135: <td width="40%" align="right" valign="top"> Chapter 25. Porting pkgsrc</td>
                    136: </tr>
                    137: </table>
                    138: </div>
                    139: </body>
                    140: </html>

CVSweb <webmaster@jp.NetBSD.org>