Annotation of htdocs/docs/pkgsrc/regression.html, revision 1.7

1.1       dsieger     1: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
                      2: <html>
                      3: <head>
                      4: <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
1.3       rillig      5: <title>Chapter 25. Regression tests</title>
1.7     ! spz         6: <link rel="stylesheet" type="text/css" href="/global.css">
1.1       dsieger     7: <meta name="generator" content="DocBook XSL Stylesheets VX.X.X">
1.5       rillig      8: <link rel="home" href="index.html" title="The pkgsrc guide">
1.1       dsieger     9: <link rel="up" href="infrastructure.html" title="Part III. The pkgsrc infrastructure internals">
1.3       rillig     10: <link rel="prev" href="infr.design.html" title="Chapter 24. Design of the pkgsrc infrastructure">
                     11: <link rel="next" href="porting.html" title="Chapter 26. Porting pkgsrc">
1.1       dsieger    12: </head>
                     13: <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
                     14: <div class="navheader">
                     15: <table width="100%" summary="Navigation header">
1.3       rillig     16: <tr><th colspan="3" align="center">Chapter 25. Regression tests</th></tr>
1.1       dsieger    17: <tr>
                     18: <td width="20%" align="left">
                     19: <a accesskey="p" href="infr.design.html">Prev</a> </td>
                     20: <th width="60%" align="center">Part III. The pkgsrc infrastructure internals</th>
                     21: <td width="20%" align="right"> <a accesskey="n" href="porting.html">Next</a>
                     22: </td>
                     23: </tr>
                     24: </table>
                     25: <hr>
                     26: </div>
1.6       wiz        27: <div class="chapter" title="Chapter 25. Regression tests">
1.1       dsieger    28: <div class="titlepage"><div><div><h2 class="title">
1.3       rillig     29: <a name="regression"></a>Chapter 25. Regression tests</h2></div></div></div>
1.1       dsieger    30: <div class="toc">
                     31: <p><b>Table of Contents</b></p>
                     32: <dl>
1.3       rillig     33: <dt><span class="sect1"><a href="regression.html#regression.descr">25.1. The regression tests framework</a></span></dt>
                     34: <dt><span class="sect1"><a href="regression.html#regression.run">25.2. Running the regression tests</a></span></dt>
                     35: <dt><span class="sect1"><a href="regression.html#regression.new">25.3. Adding a new regression test</a></span></dt>
1.1       dsieger    36: <dd><dl>
1.3       rillig     37: <dt><span class="sect2"><a href="regression.html#regression.fun.override">25.3.1. Overridable functions</a></span></dt>
                     38: <dt><span class="sect2"><a href="regression.html#regression.fun.helper">25.3.2. Helper functions</a></span></dt>
1.1       dsieger    39: </dl></dd>
                     40: </dl>
                     41: </div>
                     42: <p>The pkgsrc infrastructure consists of a large codebase,
                     43:        and there are many corners where every little bit of a file is
                     44:        well thought out, making pkgsrc likely to fail as soon as
                     45:        anything is changed near those parts. To prevent most changes
                     46:        from breaking anything, a suite of regression tests should go
                     47:        along with every important part of the pkgsrc infrastructure.
                     48:        This chapter describes how regression tests work in pkgsrc and
                     49:        how you can add new tests.</p>
1.6       wiz        50: <div class="sect1" title="25.1. The regression tests framework">
1.1       dsieger    51: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
1.3       rillig     52: <a name="regression.descr"></a>25.1. The regression tests framework</h2></div></div></div>
1.1       dsieger    53: <p></p>
                     54: </div>
1.6       wiz        55: <div class="sect1" title="25.2. Running the regression tests">
1.1       dsieger    56: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
1.3       rillig     57: <a name="regression.run"></a>25.2. Running the regression tests</h2></div></div></div>
1.1       dsieger    58: <p>You first need to install the <a href="ftp://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc/pkgtools/pkg_regress/README.html" target="_top"><code class="filename">pkgtools/pkg_regress</code></a> package, which
1.4       hubertf    59:        provides the <span class="command"><strong>pkg_regress</strong></span> command. Then you
1.1       dsieger    60:        can simply run that command, which will run all tests in the
                     61:        <code class="filename">regress</code> category.</p>
                     62: </div>
1.6       wiz        63: <div class="sect1" title="25.3. Adding a new regression test">
1.1       dsieger    64: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
1.3       rillig     65: <a name="regression.new"></a>25.3. Adding a new regression test</h2></div></div></div>
1.1       dsieger    66: <p>Every directory in the <code class="filename">regress</code>
                     67:        category that contains a file called <code class="filename">spec</code>
                     68:        is considered a regression test. This file is a shell program
1.4       hubertf    69:        that is included by the <span class="command"><strong>pkg_regress</strong></span> command.
1.1       dsieger    70:        The following functions can be overridden to suit your
                     71:        needs.</p>
1.6       wiz        72: <div class="sect2" title="25.3.1. Overridable functions">
1.1       dsieger    73: <div class="titlepage"><div><div><h3 class="title">
1.3       rillig     74: <a name="regression.fun.override"></a>25.3.1. Overridable functions</h3></div></div></div>
1.1       dsieger    75: <p>These functions do not take any parameters. They are all
1.6       wiz        76:        called in <span class="quote">&#8220;<span class="quote">set -e</span>&#8221;</span> mode, so you should be careful
1.1       dsieger    77:        to check the exitcodes of any commands you run in the
                     78:        test.</p>
                     79: <div class="variablelist"><dl>
                     80: <dt><span class="term"><code class="varname">do_setup()</code></span></dt>
                     81: <dd><p>This function prepares the environment for the
                     82:        test. By default it does nothing.</p></dd>
                     83: <dt><span class="term"><code class="varname">do_test()</code></span></dt>
                     84: <dd><p>This function runs the actual test. By default,
                     85:        it calls <code class="varname">TEST_MAKE</code> with the arguments
                     86:        <code class="varname">MAKEARGS_TEST</code> and writes its output including
                     87:        error messages into the file
                     88:        <code class="varname">TEST_OUTFILE</code>.</p></dd>
                     89: <dt><span class="term"><code class="varname">check_result()</code></span></dt>
                     90: <dd><p>This function is run after the test and is
                     91:        typically used to compare the actual output from the one that is
                     92:        expected. It can make use of the various helper functions from
                     93:        the next section.</p></dd>
                     94: <dt><span class="term"><code class="varname">do_cleanup()</code></span></dt>
                     95: <dd><p>This function cleans everything up after the
                     96:        test has been run. By default it does nothing.</p></dd>
                     97: </dl></div>
                     98: </div>
1.6       wiz        99: <div class="sect2" title="25.3.2. Helper functions">
1.1       dsieger   100: <div class="titlepage"><div><div><h3 class="title">
1.3       rillig    101: <a name="regression.fun.helper"></a>25.3.2. Helper functions</h3></div></div></div>
1.1       dsieger   102: <div class="variablelist"><dl>
                    103: <dt><span class="term"><code class="varname">exit_status(expected)</code></span></dt>
                    104: <dd><p>This function compares the exitcode of the
1.4       hubertf   105:        <span class="command"><strong>do_test()</strong></span> function with its first parameter.
1.1       dsieger   106:        If they differ, the test will fail.</p></dd>
                    107: <dt><span class="term"><code class="varname">output_require(regex...)</code></span></dt>
                    108: <dd><p>This function checks for each of its parameters
1.4       hubertf   109:        if the output from <span class="command"><strong>do_test()</strong></span> matches the
1.1       dsieger   110:        extended regular expression. If it does not, the test will
                    111:        fail.</p></dd>
                    112: <dt><span class="term"><code class="varname">output_prohibit(regex...)</code></span></dt>
                    113: <dd><p>This function checks for each of its parameters
1.4       hubertf   114:        if the output from <span class="command"><strong>do_test()</strong></span> does
1.1       dsieger   115:        <span class="emphasis"><em>not</em></span> match the extended regular expression.
                    116:        If any of the regular expressions matches, the test will
                    117:        fail.</p></dd>
                    118: </dl></div>
                    119: </div>
                    120: </div>
                    121: </div>
                    122: <div class="navfooter">
                    123: <hr>
                    124: <table width="100%" summary="Navigation footer">
                    125: <tr>
                    126: <td width="40%" align="left">
                    127: <a accesskey="p" href="infr.design.html">Prev</a> </td>
                    128: <td width="20%" align="center"><a accesskey="u" href="infrastructure.html">Up</a></td>
                    129: <td width="40%" align="right"> <a accesskey="n" href="porting.html">Next</a>
                    130: </td>
                    131: </tr>
                    132: <tr>
1.3       rillig    133: <td width="40%" align="left" valign="top">Chapter 24. Design of the pkgsrc infrastructure </td>
1.1       dsieger   134: <td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td>
1.3       rillig    135: <td width="40%" align="right" valign="top"> Chapter 26. Porting pkgsrc</td>
1.1       dsieger   136: </tr>
                    137: </table>
                    138: </div>
                    139: </body>
                    140: </html>

CVSweb <webmaster@jp.NetBSD.org>