CVS log for src/usr.sbin/makemandb/apropos-utils.c

Up to [cvs.NetBSD.org] / src / usr.sbin / makemandb

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

makemandb: trailing whitespace

makemandb/*: fix spelling of database and consistency of SQLite

Pull up following revision(s) (requested by gutteridge in ticket #1461):

	usr.sbin/makemandb/apropos.1: revision 1.19
	usr.sbin/makemandb/apropos.c: revision 1.25
	usr.sbin/makemandb/apropos.c: revision 1.26
	usr.sbin/makemandb/apropos.1: revision 1.20
	usr.sbin/makemandb/apropos.1: revision 1.21
	usr.sbin/makemandb/apropos.1: revision 1.22
	usr.sbin/makemandb/apropos.1: revision 1.23
	usr.sbin/makemandb/apropos-utils.c: revision 1.46
	usr.sbin/makemandb/apropos-utils.c: revision 1.47
	usr.sbin/makemandb/apropos-utils.c: revision 1.49

PR/54343: Prevent NULL pointers in callback strings; use "*?*" for now to
identify them.

PR bin/54343:  We want the callback_args.machine to be NULL if it is not
present in the DB.

The previous commit fixed the problem of allowing apropos to not crash and
produce output even if the database is missing values for certain mandatory
fields, such as name, section etc. Normally we don't expect those values
to be missing in the database but in case of parsing errors it can happen.

However, the machine architecture is an optional field since not all man pages
are hardware specific so that should be allowed to be set to NULL if not
present in the database.

apropos.c: fix pager functionality

Issue reported by Rocky Hotas on NetBSD-Users, patch input from RVP on
same, adjustments by me.

apropos.1: document the PAGER environment variable

apropos(1): use proper -width

apropos(1): use proper -width for the list of options too

apropos(1): Tweak the description of -1, ... -9, and -s

-s is not for compatibility only, because section names can be
anything.  E.g. we have 3lua and 9lua in base.  We have rudiments of
3f (for FORTRAN libs).  Some packages in pkgsrc also use suffixed 1
and 3 sections.

apropos(1): Use the official spelling for "SQLite".
While here, use .Bx to refer to 3BSD.

apropos(1): improve error handling in edge cases
Patch from RVP on NetBSD-Users, with an additional comment tweak by me.

Summary from RVP:
1. Ignore SIGPIPE so that we're not killed in the middle of some
   DB operation by a botched $PAGER:
$ env PAGER=3D/non-existent apropos -p ...
2. Return proper exit status in case of write errors:
$ apropos ... >/dev/full || echo fail

apropos(1): improve error handling in edge cases

Patch from RVP on NetBSD-Users, with an additional comment tweak by me.
Summary from RVP:

1. Ignore SIGPIPE so that we're not killed in the middle of some
   DB operation by a botched $PAGER:

$ env PAGER=/non-existent apropos -p ...

2. Return proper exit status in case of write errors:

$ apropos ... >/dev/full || echo fail

usr.sbin: remove unnecessary CONSTCOND, lint no longer needs it

Since 2021-01-31, lint no longer requires a CONSTCOND comment in a
do-while-0 statement since this is a common code pattern, especially in
statement-like macros.

sed -i -E 's,} while \(/\* ?CONSTCOND ?\*/ ?0\),} while (0),' */*.[ch]

Mostly merge changes from HEAD upto 20200411

PR bin/54343:  We want the callback_args.machine to be NULL if it is not present in the DB.

The previous commit fixed the problem of allowing apropos to not crash and
produce output even if the database is missing values for certain mandatory
fields, such as name, section etc. Normally we don't expect those values
to be missing in the database but in case of parsing errors it can happen.

However, the machine architecture is an optional field since not all man pages
are hardware specific so that should be allowed to be set to NULL if not
present in the database.

PR/54343: Prevent NULL pointers in callback strings; use "*?*" for now to
identify them.

Sync with HEAD

Properly free section_clause.

PR misc/54213: Fix performance of whatis(1) when no matches are found

In revision 1.6 of whatis.c the query was modified to return matches for names found
in MLINKS of the man pages as well. However it was slow. The reason probably being that it
required a join. But more importantly the where condition on an FTS virtual table column
is very slow. To avoid the join and the expensive where condition on the virtual table,
add the name_desc column to the mandb_links table as well. This improves the performance
of whatis(1) to the original level at the expense of slight data duplication.

Bump the schema to force database rebuild to take account for the new column addition

Memory allocated by sqlite3_mprintf should be free'd by sqlite3_free

This was causing memory corruption thus making apropos(1) fail in some cases.
Specifically following options were broken and should be fixed with this commit:

-n option was causing a core dump
apropos was giving warning when using -l and any of the section numbers as options
as reported by paulg on current-users.

Set the snippet_length field of the callback_args

Because of this field not being set, apropos was failing to show snippet when piped to a pager
or when used with -p argument.

fix memory allocation problems detected by jemalloc...

Encapsulate all the arguments required by the query callback function in a struct.

If we want to add or remove arguments from the callback functions, it requires
changing the callback interface all over the place. By letting the callback simply
expect a single struct argument, it would clean things up a bit.

ok christos

Don't use the custom tokenizer when compiled with debugging on

Using the custom tokenizer means one cannot interactively query the database
through the SQLite shell, thus thwarting the purpose of the debug build option.

Thanks to leot@ for reporting it.

(While there change the debug macro from DEBUG to APROPOS_DEBUG)

Add a custom tokenizer which does not stem certain keywords.

Which keywords should not be stemmed is specified in the nostem.txt file.
(Right now I have taken all the man page names, split them if they had
underscores, removed common English words and converted everything to
lowercase.)

The tokenizer itself is based on the Porter stemming tokenizer shipped with
Sqlite. The code in custom_apropos_tokenizer.c is copy of that code with
some modifications to prevent stemming keywords specified in nostem.txt.

Additionally, it now uses underscore `_' also as a token delimiter. Therefore,
now it's possible to do query for `lwp' and all `_lwp_*' man page names
will be matched. Or the query can be `unconst' and `__UNCONST' will be matched.
This was not possible earlier, because underscore was not a delimiter and therefore
the index would have __UNCONST as a key rather than UNCONST.

The tokenizer needs fts3_tokenizer.h file, which is not shipped with the
amalgamation build of Sqlite, therefore it needs to be added here (unless
we decide there is a better place for it).

To enforce using the new tokenizer, a schema version bump is needed

Since the tokenization is done both at the indexing time (via makemandb) and
also while query time (via apropos or whatis), it will be needed to bump
the schema version everytime nostem.txt is modified. Otherwise the
index will consist of old tokens and desired changes will not be seen with
apropos.

This should also fix the issue reported in PR bin/46255. Similar suggestion was
also made on tech-userlevel@ recently:
<http://mail-index.netbsd.org/tech-userlevel/2017/06/08/msg010620.html>

Thanks to christos@ for multiple rounds of reviews of the tokenizer code.

Sync with HEAD - tag prg-localcount2-base1

Simplify handling of the section arguments in apropos(1).

Earlier, a white space separated string was generated containing all the section
numbers passed through command line arguments. Later on that would have to be
tokenized and processed. Instead of that, use a NULL terminated array of strings.

Thanks to christos@ for reviewing and suggesting further improvements.

Simplify

Instead of dereferencing the pointer passed in as function argument, use a
temporary local buffer. Saves the cost of pointer dereferencing at so many places.

Update the comment to be in sync with the code.

Use sqlite3_mprintf() to generate SQL query instead of asprintf.

Disable the database compression if DEBUG is defined.

When debugging makemandb(8), it helps to be able to view the text being
stored in the database.

Sync with HEAD

Better handle MLINKS in apropos(1).

apropos(1) only indexes the first .Nm entry from the NAME section in the full
text index. Rest of the .Nm entries are stored in a separate table: mandb_links.

Till now apropos(1) did not use the mandb_links table. So whenever a query
was being made for one of the man page links, such as realloc(3), it was showing
malloc(3) in the results but not as the first result. And, also the result would
show up as malloc(3), rather than realloc(3) (which can be confusing).

With this change, for single keyword queries, apropos(1) would now utilise the
mandb_links table as well. If the query is for one of the links of a man page,
it would show as the first result. Also, the result would show up as the name
of the link rather than the original man page name. For example, if the query
was for realloc, the output would be realloc(3), rather than malloc(3).

Following are some example queries showing difference in the output before this
change and after this change:

#Before changes
$ apropos -n 5 -M realloc
reallocarr (3)    reallocate array
reallocarray (3)  reallocate memory for an array of elements checking
for overflow
fgetwln (3)       get a line of wide characters from a stream
fgetln (3)        get a line from a stream
posix_memalign (3)        aligned memory allocation

#After changes
$ ./apropos -n 5 -M realloc
realloc (3)       general memory allocation operations
realloc (3)       general purpose memory allocation functions
realloc (9)       general-purpose kernel memory allocator
reallocarr (3)    reallocate array
reallocarray (3)  reallocate memory for an array of elements checking
for overflow

#Before changes
$ apropos -n 5 -M TAILQ_REMOVE
SLIST_HEAD (3) implementations of singly-linked lists, lists, simple
queues, tail queues, and singly-linked tail queues

#After changes
$ ./apropos -n 5 -M TAILQ_REMOVE
TAILQ_REMOVE (3)  implementations of singly-linked lists, lists,
simple queues, tail queues, and singly-linked tail queues

#Before changes
$ apropos -n 5 -M falloc
filedesc (9)      file descriptor tables and operations
file (9)  operations on file entries

#After changes
$ ./apropos -n 5 -M falloc
falloc (9)        file descriptor tables and operations
file (9)  operations on file entries

ok christos@

Sync with HEAD

Include <unistd.h> for R_OK W_OK STDOUT_FILENO access(2)

These symbols are undefined after switch to new zlib.

Sync with HEAD

Mark the section and md5_hash columns as unindexed in the FTS table, as they are not used for search

Fix an off by one issue when concatenating strings.

Fix possible buffer overflow when concatenating strings.
Patch from christos@

Refactor the function for executing the search SQL query into two parts.

One part is responsible for generating the SQL query
The other part is responsible for executing the generated query.

While there, also remove a comment which is not valid anymore.
And, don't call the snippet function when doing legacy mode search as we are
not using the full text feature there.

CID 1358675: Wrong variable test

Pull up following revision(s) (requested by christos in ticket #1142):
	usr.sbin/makemandb/apropos-utils.c: revisions 1.18, 1.19
CID 1341551: Don't bother formatting if ti == NULL
--
PR/50344: Stephen Fisher: apropos shows formatting on console with vt100 term
type. Can't print terminfo sequences directly; need to process them with
ti_puts() to handle padding. This removes the padding delays, and stricly
could break on slow terminal hardware, but they way the code is structured
makes it impossible to fix properly (since the formatting strings are
passed in the query).

PR/51062: Abhinav Upadhyay: Allow non numeric sections to be indexed and
searched by apropos(1).
Fold long lines.

PR/51038: Abhinav Upadhyay: check for access permissions to the sqlite database

PR/51025: Abhinav Upadhyay: Remove unused includes from apropos-utils.c

PR/51004: Abhinav Upadhyay: apropos html mode doesn't handle especial
characters in the short description

PR/50460: Abhinav Upadhyay: Fix legacy apropos query to match both the name
and the one line description and delete extra args.

CID 1341551: Don't bother formatting if ti == NULL

PR/50344: Stephen Fisher: apropos shows formatting on console with vt100 term
type. Can't print terminfo sequences directly; need to process them with
ti_puts() to handle padding. This removes the padding delays, and stricly
could break on slow terminal hardware, but they way the code is structured
makes it impossible to fix properly (since the formatting strings are
passed in the query).
XXX: pullup-7

src is too big these days to tolerate superfluous apostrophes.  It's
"its", people!

Rebase to HEAD as of a few days ago.

Rebase.

Fix an off by one bug in apropos.
The bug is in the html output where some garbage characters are
seen in the context match output.

From Abhinav Upadhyay in PR 49058.

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

resync from head

instead of having a format and no format flag, and exposing various formatters,
provide a format enum and expose html formatting too.

fix legacy mode in pager filter. (don't ul format if we are not formatting).

- Fix legacy mode to use like instead of match. This loses ranking.
- default to unlimited lines
- fix formatting of legacy mode

- If the stdout is not a tty, prevent formatting unless forced with -i
- Don't ever page unless asked for with -p
- Introduce "legacy mode" (-l)
  1. searches only name and name_desc, prints name(section) - name_description
  2. turns off escape formatting (can be forced on with -i)
  3. turns off context printing (can be forced on with -c)
- Parse the environment $APROPOS variable as an argument vector.

With these changes one can simply 'export APROPOS=-l' and get the old apropos
behavior.

resync with head

add -r flag to elide tty formatting

remove trailing whitespace

sync with head

- move the terminal handling in apropos-utils.c since htmp and pager are also
  handled there.
- underline the name, section, and description so that it is prettier.
- change to bold terminal the terminal highlighting to match with less

Since mdocml decided to name headers that conflict with system ones (term.h)
move the header inclusion one up.

Resync to 2012-11-19 00:00:00 UTC

sync with head

Make mandb path configurable. makemandb (and related tools) use
the path from the _mandb variable from man.conf now.

Set _mandb in man.conf to same value as was used before.

From Abhinav Upadhyay <er.abhinav.upadhyay@gmail.com>.

sync with head.

KNF

Pull up following revision(s) (requested by wiz in ticket #229):
	usr.sbin/makemandb/makemandb.c: revision 1.9
	usr.sbin/makemandb/DBSCHEMA: revision 1.2
	usr.sbin/makemandb/apropos-utils.c: revision 1.5
	usr.sbin/makemandb/apropos-utils.h: revision 1.3
PR 46419 by Abhinav Upadhyay using his updated patch:
Clean up after removing man page aliases.

PR 46419 by Abhinav Upadhyay using his updated patch:
Clean up after removing man page aliases.

Pull up following revision(s) (requested by wiz in ticket #186):
	usr.sbin/makemandb/apropos.c: revision 1.6
	usr.sbin/makemandb/apropos-utils.c: revision 1.3
	usr.sbin/makemandb/apropos-utils.c: revision 1.4
Add the result from sqlite3_errmsg() to some error messages.
Now we can get "apropos: Unable to query schema version: database is locked"
instead of just "apropos: Unable to query schema version".
Handle pages with slashes in their names better.
From Abhinav Upadhyay in private mail.

sync with head

file apropos-utils.c was added on branch yamt-pagecache on 2012-04-17 00:09:49 +0000

Handle pages with slashes in their names better.
From Abhinav Upadhyay in private mail.

Add the result from sqlite3_errmsg() to some error messages.
Now we can get "apropos: Unable to query schema version: database is locked"
instead of just "apropos: Unable to query schema version".

Fix C&P error with $NetBSD$

Import the new apropos/whatis.

This code has been developed by Abhinav Upadhyay as part of Google's Summer
of Code 2011. It uses libmandoc to parse man pages and builds a Full
Text Index in a SQLite database. The combination of indexing the full
manual page, filtering out stop words and ranking individual matches
based on the section gives a much improved user experience.

The old makewhatis and friends are kept under MKMAKEMANDB=no for now.