Annotation of htdocs/docs/pkgsrc/regression.html, revision 1.26
1.18 sevan 1: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
1.1 dsieger 2: <html>
3: <head>
4: <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
1.25 rillig 5: <title>Chapter 26. Regression tests</title>
1.7 spz 6: <link rel="stylesheet" type="text/css" href="/global.css">
1.1 dsieger 7: <meta name="generator" content="DocBook XSL Stylesheets VX.X.X">
1.5 rillig 8: <link rel="home" href="index.html" title="The pkgsrc guide">
1.1 dsieger 9: <link rel="up" href="infrastructure.html" title="Part III. The pkgsrc infrastructure internals">
1.25 rillig 10: <link rel="prev" href="infr.design.html" title="Chapter 25. Design of the pkgsrc infrastructure">
11: <link rel="next" href="porting.html" title="Chapter 27. Porting pkgsrc">
1.1 dsieger 12: </head>
13: <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
14: <div class="navheader">
15: <table width="100%" summary="Navigation header">
1.25 rillig 16: <tr><th colspan="3" align="center">Chapter 26. Regression tests</th></tr>
1.1 dsieger 17: <tr>
18: <td width="20%" align="left">
19: <a accesskey="p" href="infr.design.html">Prev</a> </td>
20: <th width="60%" align="center">Part III. The pkgsrc infrastructure internals</th>
21: <td width="20%" align="right"> <a accesskey="n" href="porting.html">Next</a>
22: </td>
23: </tr>
24: </table>
25: <hr>
26: </div>
1.8 asau 27: <div class="chapter">
1.1 dsieger 28: <div class="titlepage"><div><div><h2 class="title">
1.25 rillig 29: <a name="regression"></a>Chapter 26. Regression tests</h2></div></div></div>
1.1 dsieger 30: <div class="toc">
31: <p><b>Table of Contents</b></p>
1.15 leot 32: <dl class="toc">
1.25 rillig 33: <dt><span class="sect1"><a href="regression.html#regression.run">26.1. Running the regression tests</a></span></dt>
34: <dt><span class="sect1"><a href="regression.html#regression.new">26.2. Adding a new regression test</a></span></dt>
1.1 dsieger 35: <dd><dl>
1.25 rillig 36: <dt><span class="sect2"><a href="regression.html#regression.fun.override">26.2.1. Overridable functions</a></span></dt>
37: <dt><span class="sect2"><a href="regression.html#regression.fun.helper">26.2.2. Helper functions</a></span></dt>
1.1 dsieger 38: </dl></dd>
39: </dl>
40: </div>
41: <p>The pkgsrc infrastructure consists of a large codebase,
42: and there are many corners where every little bit of a file is
43: well thought out, making pkgsrc likely to fail as soon as
44: anything is changed near those parts. To prevent most changes
45: from breaking anything, a suite of regression tests should go
46: along with every important part of the pkgsrc infrastructure.
47: This chapter describes how regression tests work in pkgsrc and
48: how you can add new tests.</p>
1.8 asau 49: <div class="sect1">
1.1 dsieger 50: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
1.25 rillig 51: <a name="regression.run"></a>26.1. Running the regression tests</h2></div></div></div>
1.26 ! leot 52: <p>You first need to install the <a href="https://cdn.NetBSD.org/pub/pkgsrc/current/pkgsrc/pkgtools/pkg_regress/index.html" target="_top"><code class="filename">pkgtools/pkg_regress</code></a> package, which
1.4 hubertf 53: provides the <span class="command"><strong>pkg_regress</strong></span> command. Then you
1.1 dsieger 54: can simply run that command, which will run all tests in the
1.21 rillig 55: <code class="filename">regress/</code> directory.</p>
1.1 dsieger 56: </div>
1.8 asau 57: <div class="sect1">
1.1 dsieger 58: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
1.25 rillig 59: <a name="regression.new"></a>26.2. Adding a new regression test</h2></div></div></div>
1.21 rillig 60: <p>Every directory in the <code class="filename">regress/</code>
61: directory that contains a file called <code class="filename">spec</code>
1.1 dsieger 62: is considered a regression test. This file is a shell program
1.4 hubertf 63: that is included by the <span class="command"><strong>pkg_regress</strong></span> command.
1.1 dsieger 64: The following functions can be overridden to suit your
65: needs.</p>
1.8 asau 66: <div class="sect2">
1.1 dsieger 67: <div class="titlepage"><div><div><h3 class="title">
1.25 rillig 68: <a name="regression.fun.override"></a>26.2.1. Overridable functions</h3></div></div></div>
1.16 rillig 69: <p>These functions do not take any parameters. Although they
70: are called in <span class="quote">“<span class="quote">set -e</span>”</span> mode, they don't stop at the
1.23 leot 71: first failing command. See <a class="ulink" href="https://stackoverflow.com/q/4072984" target="_top">this Stack Overflow
1.16 rillig 72: question</a> for details.</p>
1.8 asau 73: <div class="variablelist"><dl class="variablelist">
1.16 rillig 74: <dt><span class="term"><code class="varname">do_setup</code></span></dt>
1.1 dsieger 75: <dd><p>This function prepares the environment for the
76: test. By default it does nothing.</p></dd>
1.16 rillig 77: <dt><span class="term"><code class="varname">do_test</code></span></dt>
78: <dd>
79: <p>This function runs the actual test. By default,
1.1 dsieger 80: it calls <code class="varname">TEST_MAKE</code> with the arguments
81: <code class="varname">MAKEARGS_TEST</code> and writes its output including
82: error messages into the file
1.16 rillig 83: <code class="varname">TEST_OUTFILE</code>.</p>
84: <p>When defining this function, make sure that all output that
85: needs to be checked is written to the correct output file.
86: Example:</p>
87: <pre class="programlisting">
88: do_test() {
1.17 rillig 89: echo "Example output"
1.16 rillig 90: } 1>$TEST_OUTFILE 2>&1
91: </pre>
92: </dd>
93: <dt><span class="term"><code class="varname">check_result</code></span></dt>
94: <dd>
95: <p>This function is run after the test and is
1.1 dsieger 96: typically used to compare the actual output from the one that is
97: expected. It can make use of the various helper functions from
1.16 rillig 98: the next section. Example:</p>
99: <pre class="programlisting">
100: check_result() {
1.17 rillig 101: exit_status 0
102: output_require "Example"
103: output_require "^[[:alpha:]+[[:space:]][[:alpha:]]{6}$"
104: output_prohibit "no such file or directory"
1.22 rillig 105: regress_fail "expected $expected but got $actual for input $input"
1.16 rillig 106: }
107: </pre>
108: </dd>
109: <dt><span class="term"><code class="varname">do_cleanup</code></span></dt>
1.1 dsieger 110: <dd><p>This function cleans everything up after the
111: test has been run. By default it does nothing.</p></dd>
112: </dl></div>
113: </div>
1.8 asau 114: <div class="sect2">
1.1 dsieger 115: <div class="titlepage"><div><div><h3 class="title">
1.25 rillig 116: <a name="regression.fun.helper"></a>26.2.2. Helper functions</h3></div></div></div>
1.8 asau 117: <div class="variablelist"><dl class="variablelist">
1.21 rillig 118: <dt><span class="term"><code class="varname">regress_fail <em class="replaceable"><code>message...</code></em></code></span></dt>
119: <dd><p>This function makes the test fail with the given error message.</p></dd>
1.16 rillig 120: <dt><span class="term"><code class="varname">exit_status expected</code></span></dt>
1.1 dsieger 121: <dd><p>This function compares the exitcode of the
1.16 rillig 122: <span class="command"><strong>do_test</strong></span> function with its first parameter.
1.1 dsieger 123: If they differ, the test will fail.</p></dd>
1.16 rillig 124: <dt><span class="term"><code class="varname">output_require regex...</code></span></dt>
125: <dd>
126: <p>This function checks for each of its parameters
127: if the output from <span class="command"><strong>do_test</strong></span> matches the
1.1 dsieger 128: extended regular expression. If it does not, the test will
1.16 rillig 129: fail. Example:</p>
130: <pre class="programlisting">
131: output_require "looks fine"
132: output_require "^[[:alpha:]+[[:space:]][[:alpha:]]{6}$"
133: </pre>
134: </dd>
1.24 leot 135: <dt><span class="term"><code class="varname">output_prohibit regex...</code></span></dt>
1.1 dsieger 136: <dd><p>This function checks for each of its parameters
1.4 hubertf 137: if the output from <span class="command"><strong>do_test()</strong></span> does
1.1 dsieger 138: <span class="emphasis"><em>not</em></span> match the extended regular expression.
139: If any of the regular expressions matches, the test will
140: fail.</p></dd>
141: </dl></div>
142: </div>
143: </div>
144: </div>
145: <div class="navfooter">
146: <hr>
147: <table width="100%" summary="Navigation footer">
148: <tr>
149: <td width="40%" align="left">
150: <a accesskey="p" href="infr.design.html">Prev</a> </td>
151: <td width="20%" align="center"><a accesskey="u" href="infrastructure.html">Up</a></td>
152: <td width="40%" align="right"> <a accesskey="n" href="porting.html">Next</a>
153: </td>
154: </tr>
155: <tr>
1.25 rillig 156: <td width="40%" align="left" valign="top">Chapter 25. Design of the pkgsrc infrastructure </td>
1.1 dsieger 157: <td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td>
1.25 rillig 158: <td width="40%" align="right" valign="top"> Chapter 27. Porting pkgsrc</td>
1.1 dsieger 159: </tr>
160: </table>
161: </div>
162: </body>
163: </html>
CVSweb <webmaster@jp.NetBSD.org>