Annotation of htdocs/docs/pkgsrc/regression.html, revision 1.3
1.1 dsieger 1: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
2: <html>
3: <head>
4: <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
1.3 ! rillig 5: <title>Chapter 25. Regression tests</title>
1.2 rillig 6: <link rel="stylesheet" href="/global.css" type="text/css">
1.1 dsieger 7: <meta name="generator" content="DocBook XSL Stylesheets VX.X.X">
8: <link rel="start" href="index.html" title="The pkgsrc guide">
9: <link rel="up" href="infrastructure.html" title="Part III. The pkgsrc infrastructure internals">
1.3 ! rillig 10: <link rel="prev" href="infr.design.html" title="Chapter 24. Design of the pkgsrc infrastructure">
! 11: <link rel="next" href="porting.html" title="Chapter 26. Porting pkgsrc">
1.1 dsieger 12: </head>
13: <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
14: <div class="navheader">
15: <table width="100%" summary="Navigation header">
1.3 ! rillig 16: <tr><th colspan="3" align="center">Chapter 25. Regression tests</th></tr>
1.1 dsieger 17: <tr>
18: <td width="20%" align="left">
19: <a accesskey="p" href="infr.design.html">Prev</a> </td>
20: <th width="60%" align="center">Part III. The pkgsrc infrastructure internals</th>
21: <td width="20%" align="right"> <a accesskey="n" href="porting.html">Next</a>
22: </td>
23: </tr>
24: </table>
25: <hr>
26: </div>
27: <div class="chapter" lang="en">
28: <div class="titlepage"><div><div><h2 class="title">
1.3 ! rillig 29: <a name="regression"></a>Chapter 25. Regression tests</h2></div></div></div>
1.1 dsieger 30: <div class="toc">
31: <p><b>Table of Contents</b></p>
32: <dl>
1.3 ! rillig 33: <dt><span class="sect1"><a href="regression.html#regression.descr">25.1. The regression tests framework</a></span></dt>
! 34: <dt><span class="sect1"><a href="regression.html#regression.run">25.2. Running the regression tests</a></span></dt>
! 35: <dt><span class="sect1"><a href="regression.html#regression.new">25.3. Adding a new regression test</a></span></dt>
1.1 dsieger 36: <dd><dl>
1.3 ! rillig 37: <dt><span class="sect2"><a href="regression.html#regression.fun.override">25.3.1. Overridable functions</a></span></dt>
! 38: <dt><span class="sect2"><a href="regression.html#regression.fun.helper">25.3.2. Helper functions</a></span></dt>
1.1 dsieger 39: </dl></dd>
40: </dl>
41: </div>
42: <p>The pkgsrc infrastructure consists of a large codebase,
43: and there are many corners where every little bit of a file is
44: well thought out, making pkgsrc likely to fail as soon as
45: anything is changed near those parts. To prevent most changes
46: from breaking anything, a suite of regression tests should go
47: along with every important part of the pkgsrc infrastructure.
48: This chapter describes how regression tests work in pkgsrc and
49: how you can add new tests.</p>
50: <div class="sect1" lang="en">
51: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
1.3 ! rillig 52: <a name="regression.descr"></a>25.1. The regression tests framework</h2></div></div></div>
1.1 dsieger 53: <p></p>
54: </div>
55: <div class="sect1" lang="en">
56: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
1.3 ! rillig 57: <a name="regression.run"></a>25.2. Running the regression tests</h2></div></div></div>
1.1 dsieger 58: <p>You first need to install the <a href="ftp://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc/pkgtools/pkg_regress/README.html" target="_top"><code class="filename">pkgtools/pkg_regress</code></a> package, which
59: provides the <span><strong class="command">pkg_regress</strong></span> command. Then you
60: can simply run that command, which will run all tests in the
61: <code class="filename">regress</code> category.</p>
62: </div>
63: <div class="sect1" lang="en">
64: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
1.3 ! rillig 65: <a name="regression.new"></a>25.3. Adding a new regression test</h2></div></div></div>
1.1 dsieger 66: <p>Every directory in the <code class="filename">regress</code>
67: category that contains a file called <code class="filename">spec</code>
68: is considered a regression test. This file is a shell program
69: that is included by the <span><strong class="command">pkg_regress</strong></span> command.
70: The following functions can be overridden to suit your
71: needs.</p>
72: <div class="sect2" lang="en">
73: <div class="titlepage"><div><div><h3 class="title">
1.3 ! rillig 74: <a name="regression.fun.override"></a>25.3.1. Overridable functions</h3></div></div></div>
1.1 dsieger 75: <p>These functions do not take any parameters. They are all
76: called in “<span class="quote">set -e</span>” mode, so you should be careful
77: to check the exitcodes of any commands you run in the
78: test.</p>
79: <div class="variablelist"><dl>
80: <dt><span class="term"><code class="varname">do_setup()</code></span></dt>
81: <dd><p>This function prepares the environment for the
82: test. By default it does nothing.</p></dd>
83: <dt><span class="term"><code class="varname">do_test()</code></span></dt>
84: <dd><p>This function runs the actual test. By default,
85: it calls <code class="varname">TEST_MAKE</code> with the arguments
86: <code class="varname">MAKEARGS_TEST</code> and writes its output including
87: error messages into the file
88: <code class="varname">TEST_OUTFILE</code>.</p></dd>
89: <dt><span class="term"><code class="varname">check_result()</code></span></dt>
90: <dd><p>This function is run after the test and is
91: typically used to compare the actual output from the one that is
92: expected. It can make use of the various helper functions from
93: the next section.</p></dd>
94: <dt><span class="term"><code class="varname">do_cleanup()</code></span></dt>
95: <dd><p>This function cleans everything up after the
96: test has been run. By default it does nothing.</p></dd>
97: </dl></div>
98: </div>
99: <div class="sect2" lang="en">
100: <div class="titlepage"><div><div><h3 class="title">
1.3 ! rillig 101: <a name="regression.fun.helper"></a>25.3.2. Helper functions</h3></div></div></div>
1.1 dsieger 102: <div class="variablelist"><dl>
103: <dt><span class="term"><code class="varname">exit_status(expected)</code></span></dt>
104: <dd><p>This function compares the exitcode of the
105: <span><strong class="command">do_test()</strong></span> function with its first parameter.
106: If they differ, the test will fail.</p></dd>
107: <dt><span class="term"><code class="varname">output_require(regex...)</code></span></dt>
108: <dd><p>This function checks for each of its parameters
109: if the output from <span><strong class="command">do_test()</strong></span> matches the
110: extended regular expression. If it does not, the test will
111: fail.</p></dd>
112: <dt><span class="term"><code class="varname">output_prohibit(regex...)</code></span></dt>
113: <dd><p>This function checks for each of its parameters
114: if the output from <span><strong class="command">do_test()</strong></span> does
115: <span class="emphasis"><em>not</em></span> match the extended regular expression.
116: If any of the regular expressions matches, the test will
117: fail.</p></dd>
118: </dl></div>
119: </div>
120: </div>
121: </div>
122: <div class="navfooter">
123: <hr>
124: <table width="100%" summary="Navigation footer">
125: <tr>
126: <td width="40%" align="left">
127: <a accesskey="p" href="infr.design.html">Prev</a> </td>
128: <td width="20%" align="center"><a accesskey="u" href="infrastructure.html">Up</a></td>
129: <td width="40%" align="right"> <a accesskey="n" href="porting.html">Next</a>
130: </td>
131: </tr>
132: <tr>
1.3 ! rillig 133: <td width="40%" align="left" valign="top">Chapter 24. Design of the pkgsrc infrastructure </td>
1.1 dsieger 134: <td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td>
1.3 ! rillig 135: <td width="40%" align="right" valign="top"> Chapter 26. Porting pkgsrc</td>
1.1 dsieger 136: </tr>
137: </table>
138: </div>
139: </body>
140: </html>
CVSweb <webmaster@jp.NetBSD.org>