Annotation of htdocs/developers/PR.xml, revision 1.2

1.1       hubertf     1: <?xml version="1.0"?>
                      2: <!DOCTYPE webpage
                      3:   PUBLIC "-//NetBSD//DTD Website-based NetBSD Extension//EN"
                      4:     "http://www.NetBSD.org/XML/htdocs/lang/share/xml/website-netbsd.dtd">
                      5: 
                      6: <webpage id="developers-PR">
                      7: <config param="desc" value="Resolving Problem Reports"/>
1.2     ! kano        8: <config param="cvstag" value="$NetBSD: PR.xml,v 1.1 2007/04/05 17:50:59 hubertf Exp $"/>
        !             9: <config param="rcsdate" value="$Date: 2007/04/05 17:50:59 $"/>
1.1       hubertf    10: <head>
                     11: <title>Resolving Problem Reports</title>
                     12: </head>
                     13: 
                     14: <sect1 role="toc">
                     15: 
                     16: <sect2 id="top">
                     17: <title>Resolving Problem Reports</title>
                     18: 
                     19: <sect3 id="intro">
                     20: <title>Introduction</title>
                     21: <para>The NetBSD Project uses the GNATS <quote>Problem Report</quote>
                     22:   database to accept and track bug/problem reports from all users of
                     23:   NetBSD. When used properly, this facility allows us to make sure
                     24:   that no problem with the NetBSD software goes unfixed.</para>
                     25: 
                     26: <para>The GNATS database uses Internet E-mail as its principal
                     27:   submission mechanism, and keeps the problem reports (commonly
                     28:   abbreviated <quote>PR</quote>) in Internet E-mail format, with an
                     29:   extended header format in the body of the message. The database uses
                     30:   one file per PR, and each category is a directory, in a manner
                     31:   similar to the MH mail system, and NetNews.</para>
                     32: </sect3>
                     33: 
                     34: <sect3 id="access">
                     35: <title>Accessing the Problem Report Database</title>
                     36: 
                     37: <sect4 id="webinterface">
                     38: <title>Web Interface</title>
                     39: 
                     40: <para>There is a web-based interface to the GNATS database that has
                     41:   both a <ulink url="../Gnats/">tree of database summary pages</ulink>, and a <ulink
                     42:   url="../Misc/query-pr.html">search</ulink> facility. The web interface
                     43:   has some limitations:</para>
                     44: 
                     45: <itemizedlist>
                     46:   <listitem>The web interface is several hours behind the actual
                     47:     database because it operates on a copy of the database for
                     48:     security.</listitem>
                     49:   <listitem>The web interface <emphasis>does not</emphasis> provide
                     50:     access to any PR marked <quote>confidential</quote>.</listitem>
                     51:   <listitem>PRs displayed by a web browser have been through
                     52:     sufficient transformation that <quote>cut &amp; paste</quote> is
                     53:     not likely to do what you expect (i.e. patches will not apply
                     54:     properly because of white space substitution, uuencoded text
                     55:     will not decode properly).</listitem>
                     56: </itemizedlist>
                     57: </sect4>
                     58: 
                     59: <sect4 id="netbsd-bugs">
                     60: <title>The netbsd-bugs Mailing List</title>
                     61: 
                     62: <para>All PRs, confidential or not, are sent to the <ulink
1.2     ! kano       63:   url="/MailingLists/#netbsd-bugs">netbsd-bugs</ulink> mailing list,
1.1       hubertf    64:   so that subscribers can see each PR as it is added to the
                     65:   database.</para>
                     66: </sect4>
                     67: 
                     68: <sect4 id="gnatscommands">
                     69: <title>The GNATS commands on gnats.NetBSD.org</title>
                     70: 
                     71: <para>The GNATS database lives on the host gnats.NetBSD.org.  All
                     72:   developers are given an account on that host so that they can
                     73:   directly manipulate PRs, and access confidential PRs. The commands
                     74:   <command>edit-pr</command> and <command>query-pr</command> live in
                     75:   <filename>/usr/pkg/bin</filename>; Make sure you have that directory
                     76:   in your <varname>$PATH</varname>.</para>
                     77: 
                     78: <para>There are no UNIX Manual pages for these commands.  As with most
                     79:   GNU software, there are <quote>info</quote> pages available through
                     80:   the <command>info</command> command. Also, invoking
                     81:   <command>query-pr</command> without any arguments will cause it to
                     82:   give its usage message.</para>
                     83: 
                     84: <para>The <command>query-pr</command> command is a proper database
                     85:   query interface; it has a large number of options to search the
                     86:   database. Once you know the PR number of the PR you wish to
                     87:   manipulate, you can:</para>
                     88: 
                     89: <itemizedlist>
                     90:   <listitem>
                     91:     <para><command>query-pr --full</command> &lt;number&gt;</para>
                     92:     <para>This will dump out the full PR without any transformation to
                     93:       standard output.</para>
                     94:   </listitem>
                     95:   <listitem>
                     96:     <para><command>edit-pr</command> &lt;number&gt;</para>
                     97:     <para>This command will start up a text editor
                     98:       (<application>vi</application> by default; but this can be
                     99:       overridden by the <varname>$EDITOR</varname> or
                    100:       <varname>$VISUAL</varname> environment variables), so that
                    101:       changes can be made to the PR.</para>
                    102:   </listitem>
                    103: </itemizedlist>
                    104: </sect4>
                    105: 
                    106: <sect4 id="edit-pr">
                    107: <title>The Common Reasons to <command>edit-pr</command></title>
                    108: 
                    109: <para>The most common changes made to a PR are:</para>
                    110: 
                    111: <orderedlist>
                    112:   <listitem>
                    113:     <para>Change the PR's <code>&gt;State:</code> field to one of the values
1.2     ! kano      114:       listed in <ulink url="../Misc/pr-states.html">state</ulink> as it progresses
1.1       hubertf   115:       through the process of resolving it.</para>
                    116:     <para>The person listed in the <code>&gt;Responsible:</code> field of the
                    117:       PR should be making these state changes, as it is necessary.</para>
                    118:   </listitem>
                    119:   <listitem>
                    120:     <para>Change the PR's <code>&gt;Responsible:</code> field to the account name
                    121:       of a developer who will handle the PR.
                    122:       This person becomes the PR Submitter's primary contact for
                    123:       getting the problem resolved.</para>
                    124:     <para>This field can have any username from /etc/passwd on
                    125:       <code>gnats.NetBSD.org</code> and anyone listed in the
                    126:       <filename>/usr/pkg/share/gnats/gnats-db/gnats-adm/responsible</filename>
                    127:       file.</para>
                    128:     <para>All PRs get a default Responsible Person when they are
                    129:       initially filed, appropriate to the category in which the PR
                    130:       was filed (e.g.  <quote>security-officer</quote> for PRs in
                    131:       the <quote>security</quote> category). The
                    132:       <filename>/usr/pkg/share/gnats/gnats-db/gnats-adm/categories</filename>
                    133:       file lists the default Responsible Person for each
                    134:       category.</para>
                    135:     <para>There is also a table of <ulink
                    136:       url="../Gnats/#table-responsible">developers responsible for
                    137:       current PRs</ulink>.</para>
                    138:   </listitem>
                    139:   <listitem>
                    140:     <para>Change the PR's <code>&gt;Category:</code> field.</para>
                    141:     <para>It is not unusual for a PR Submitter to have made a poor
                    142:       category choice. There is a list of <ulink
                    143:       url="../Gnats/#category-descriptions">PR categories and their
                    144:       definitions</ulink> and a table of <ulink
                    145:       url="../Gnats/#table-category">current PRs by
                    146:       category</ulink>.</para>
                    147:     <para>The
                    148:       <filename>/usr/pkg/share/gnats/gnats-db/gnats-adm/categories</filename>
                    149:       file lists the valid categories and the default Responsible
                    150:       Person.  It is usually necessary to change the
                    151:       <code>&gt;Responsible:</code> field at the same time to a more
                    152:       appropriate person. Most often, the correct Responsible Person
                    153:       is the default Responsible Person for the new category.</para>
                    154:   </listitem>
                    155:   <listitem>
                    156:     <para>Change other <ulink url="../Misc/pr-fields.html">PR
                    157:       Fields</ulink> according to the analysis of the responsible
                    158:       developer.</para>
                    159:   </listitem>
                    160: </orderedlist>
                    161: 
                    162: <para>Edit each field you think needs modifying, then save the file
                    163:   and exit the editor. The <command>edit-pr</command> will then
                    164:   prompt for a short explanation to be typed for each key field
                    165:   change (mostly <code>&gt;State:</code> and
                    166:   <code>&gt;Responsible:</code>). This text is entered one line at
                    167:   time, ending with <command>^D</command>.</para>
                    168: 
                    169: <para>This text is then sent via e-mail to the PR Submitter, the
                    170:   Responsible developer, and <email>gnats-admin@NetBSD.org</email>.
                    171:   It is also appended to the PR by <command>edit-pr</command> along
                    172:   with the user ID of the developer making the change, a timestamp,
                    173:   and the entered text.</para>
                    174: 
                    175:   <para>Unfortunately, no external editor can be invoked at this
                    176:     point; if you make a mistake, you'll have to use
                    177:     <command>edit-pr</command> to correct it.</para>
                    178: </sect4>
                    179: </sect3>
                    180: 
                    181: <sect3 id="resolve">
                    182:   <title>Resolving Problem Reports</title>
                    183: 
                    184:   <sect4 id="ideal">
                    185:     <title>The Ideal Process</title>
                    186: 
                    187:     <para>In an ideal world, the process for a problem report is as
                    188:       follows:</para>
                    189: 
                    190:     <orderedlist>
                    191:       <listitem>
                    192:         <para>A NetBSD user has a problem with NetBSD. He invokes the
                    193:          <command>send-pr</command> command on his system (assuming
                    194:          it's still stable enough to do that), and files a Problem
                    195:          Report. Hopefully, he follows all the advice found in <ulink
                    196:          url="../Misc/pr-hints.html">"What goes into a Problem
                    197:          Report."</ulink></para>
                    198:        <para>If the user's own system is not stable enough to use
                    199:          <command>send-pr</command>, there is a <ulink
                    200:          url="../Misc/send-pr.html">web interface</ulink> that can be
                    201:          used to submit problem reports.</para>
                    202:       </listitem>
                    203:       <listitem>
                    204:         <para>When the PR arrives at <code>gnats.NetBSD.org</code>,
                    205:          the GNATS database system examines it, and files it. If the
                    206:          PR is malformed, it will be filed in the
                    207:          <code>pending</code> category, marked confidential, awaiting
                    208:          manual intervention by the GNATS database
                    209:          administrator.</para>
                    210:        <para>If the format is OK, the PR is assigned a PR number,
                    211:          filed into the requested category, and E-mailed out again to
                    212:          the default responsible party for the category, and to the
                    213:          <ulink
                    214:          url="../MailingLists/#netbsd-bugs">netbsd-bugs</ulink>
                    215:          mailing list.  A notice of the PR number and default
                    216:          Responsible Person is also E-mailed back to the PR
                    217:          Submitter.</para>
                    218:       </listitem>
                    219:       <listitem>
                    220:         <para>The Responsible Person should read and analyze the
                    221:          PR. Any other person who has insight into the problem should
                    222:          also <ulink url="../Misc/send-pr.html#appending">add
                    223:          whatever information they can to the problem report</ulink>
                    224:          (this is why the report is mailed out to a mailing list; a
                    225:          wide audience increases the probability that a key insight
                    226:          needed to solve the problem will be discovered).</para>
                    227: 
                    228:        <para>If the default Responsible Person determines that
                    229:          another developer is a more appropriate Responsible Person,
                    230:          the PR should be reassigned with
                    231:          <command>edit-pr</command>. The new Responsible Person
                    232:          should read and analyze the PR.</para>
                    233:       </listitem>
                    234:       <listitem>
                    235:         <para>Once a cause and potential fix has been identified, a
                    236:          description should be added to the PR, and the <ulink
                    237:          url="../Misc/pr-states.html">state</ulink> should be changed
                    238:          to <code>analyzed</code>. At this point, implementation of
                    239:          the fix for the problem begins.</para>
                    240:        <para>This part of the process should begin as quickly as
                    241:          possible, since a user with a current problem is suffering,
                    242:          but also has his attention engaged, and his hardware
                    243:          available for testing potential fixes. If a PR is allowed to
                    244:          languish, the opportunity to reproduce the problem and test
                    245:          potential fixes may be lost.</para>
                    246:       </listitem>
                    247:       <listitem>
                    248:         <para>Once the implementation of the fix is completed and communicated
                    249:          to the PR submitter, the PR <ulink url="/Misc/pr-states.html">state</ulink>
                    250:          should be changed to <code>feedback</code>, awaiting response from the PR
                    251:          submitter that the fix really works.</para>
                    252:        <para>A PR should also be put into <code>feedback</code> state when input
                    253:          is required from the submitter to complete the analysis of the PR
                    254:          (i.e. when you ask them a question), or when you need information
                    255:          from some other source (essentially, <code>feedback</code> is a wait
                    256:          state).</para>
                    257:       </listitem>
                    258:       <listitem>
                    259:         <para>Once the PR submitter confirms that the fix works, the PR
                    260:          can be <code>closed</code>.</para>
                    261:       </listitem>
                    262:     </orderedlist>
                    263:     <para>At each step of the PR handling process, make sure that
                    264:       feedback and other analysis and commentary is <ulink
                    265:       url="../Misc/send-pr.html#appending">appended to the PR</ulink>
                    266:       by using a proper E-mail subject line and making sure that the
                    267:       messages are copied to
                    268:       <email>gnats-bugs@NetBSD.org</email>. Having a complete record
                    269:       of information about the PR is valuable both while hunting down
                    270:       the bug and for future system maintenance.</para>
                    271:     <para>If at all possible, it is important to get any fix committed
                    272:       to the CVS trunk <ulink url="releng/pullups.html">pulled
                    273:       up</ulink> by <ulink url="releng/">NetBSD Release
                    274:       Engineering</ulink> to the "-release" branch, so that people who
                    275:       are tracking that branch can get the fix right away rather than
                    276:       waiting for the next major release of NetBSD. This also makes it
                    277:       possible for the next point release of NetBSD to have the
                    278:       fix.</para>
                    279:     <para>The NetBSD Community is a whole lot of very smart, and very
                    280:       experienced people. If you're having trouble analyzing a problem
                    281:       report, ask questions in the appropriate <ulink
                    282:       url="../MailingLists/">mailing list</ulink>; more than likely,
                    283:       someone will be able to help.</para>
                    284:   </sect4>
                    285: 
                    286: <sect4 id="otherways">
                    287: <title>Other Ways Things Get Done</title>
                    288: 
                    289: <para>The Ideal Process was described above. That's not the only way
                    290:   that problem reports get handled. All of the principal people
                    291:   involved in NetBSD are pretty busy, and can't devote full attention
                    292:   to this project. As such, if you see a PR that you can solve that
                    293:   hasn't been attended to yet, go claim it by setting yourself as the
                    294:   Responsible Person with <command>edit-pr</command>, and solve
                    295:   it.</para>
                    296: 
                    297: <para>Even if you don't feel qualified to hack the code yourself, if
                    298:   you can offer a test case or other information, send it along to
                    299:   GNATS to be <ulink url="../Misc/send-pr.html#appending">appended to
                    300:   the PR</ulink>. <quote>Many hands make light work.</quote></para>
                    301: 
                    302: <para>Some problem reports are so trivial that the fix is obvious (or
                    303:   perhaps the fix was provided by the submitter), that they go directly
                    304:   from <code>open</code> to <code>closed</code> immediately after the fix is
                    305:   committed.</para>
                    306: 
                    307: <para>If for some reason you find that you're unable to finish
                    308:   handling the PR, reset the <code>&gt;Responsible:</code> field to
                    309:   whoever had responsibility before you took the PR over.  Don't
                    310:   prevent others from making progress on the PR because they think
                    311:   you're taking care of it.</para>
                    312: 
                    313: <para>As long as you're marked as the Responsible Person for a PR,
                    314:   you'll receive a monthly E-mail reminder about it. Use those
                    315:   reminders to drive you to review PRs and put them into their correct
                    316:   states as time passes.</para>
                    317: 
                    318: <para>When a PR is in <code>feedback</code> state, the PR submitter gets an
                    319:   E-mail monthly reminder at the same time as the Responsible Persons
                    320:   do, to prompt or prod them into responding. Generally, if there
                    321:   has been no response for more than three months (three reminder
                    322:   cycles), it's pretty safe to assume that the submitter is gone or
                    323:   no longer cares.  At that point, whether to close the PR becomes
                    324:   a judgement call for the Responsible Person - how serious is it?
                    325:   Should it be solved without further input from the Submitter?</para>
                    326: 
                    327: <para>The other way we use the GNATS PR database is to keep track of
                    328:   problems which are waiting for larger issues to be solved. The
                    329:   oldest PR in the database at this writing, <ulink
                    330:   url="http://www.NetBSD.org/cgi-bin/query-pr-single.pl?number=13">lib/13</ulink>
                    331:   (yes, of course it would be 13!) begs the entire
                    332:   internationalization of the NetBSD system. I18N is a hard problem
                    333:   that requires a wholesale overhaul of the system, which is why that
                    334:   PR is still open after seven years. This doesn't mean we'll never
                    335:   solve it; just that it isn't as critical as some other problems
                    336:   reported in the database.</para>
                    337: 
                    338: <para>In effect, this usage of the GNATS PR database is as a long-term
                    339:   project tracking system.</para>
                    340: </sect4>
                    341: </sect3>
                    342: 
                    343: <sect3 id="priorities">
                    344: <title> Priorities, Severities, and Releases</title>
                    345: 
                    346: <para>In an ideal world, the GNATS PR database would be empty, we'd
                    347:   release perfect software, and Microsoft would be a shadow of its
                    348:   current self.</para>
                    349: 
                    350: <para>The <code>&gt;Priority:</code> field in the PR reflects this ideal,
                    351:   in that <code>high</code> priority is supposed to be fixed immediately;
                    352:   <code>medium</code> is supposed to be resolved before the next release
                    353:   of NetBSD (major or minor?), and <code>low</code> priority gets solved
                    354:   <quote>eventually</quote>.</para>
                    355: 
                    356: <para>In practice, PR resolution is dependent on the right mix of
                    357:   submitter interest, developer interest, problem reproduceability,
                    358:   hardware availability, and good timing. If any of the required
                    359:   elements is missing from the mix, the PR will sit.</para>
                    360: 
                    361: <para>If we were really diligent about PRs, we would adjust the priority
                    362:   of each PR to reflect its actual importance, and probability of
                    363:   getting fixed according to the definitions. Unfortunately, that
                    364:   requires an overall evaluation of Release Engineering goals and
                    365:   targets and all PRs relative to each other, which is difficult for
                    366:   a dispersed group to do in an organized fashion.</para>
                    367: 
                    368: <para>In contrast, the <code>&gt;Severity:</code> field is really an
                    369:   expression of the amount of pain the user is going through with the
                    370:   problem being reported, and it's something we really shouldn't
                    371:   adjust without careful consideration.</para>
                    372: 
                    373: <para>The proper procedure would be to review all PRs in the database
                    374:   at each release point, to decide on a per-PR basis whether to
                    375:   <quote>fix now</quote>, <quote>fix later</quote>,
                    376:   <quote>suspend</quote> and adjust priorities. Perhaps one day we'll
                    377:   have the resources and manpower for that.</para>
                    378: 
                    379: </sect3>
                    380: 
                    381: <sect3 id="remotegnats">
                    382: <title>Remote GNATS Operations</title>
                    383: 
                    384: <para>For people who find logging into a remote host tedious, the
                    385:   following csh aliases might be useful:</para>
                    386: 
                    387: <programlisting>
                    388: alias query-pr  'ssh gnats.NetBSD.org query-pr --full \!* | tee pr-\!*'
                    389: alias edit-pr   ssh -t gnats.NetBSD.org edit-pr
                    390: </programlisting>
                    391: </sect3>
                    392: 
                    393: 
                    394: </sect2>
                    395: </sect1>
                    396: 
                    397: <parentsec url="./" text="&os; Developer Documentation"/>
                    398: </webpage>

CVSweb <webmaster@jp.NetBSD.org>