CVS log for src/usr.sbin/makemandb/nostem.txt

Up to [cvs.NetBSD.org] / src / usr.sbin / makemandb

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

obsolete mkstr(1) and xstr(1)

These were only used on PDP-11 for two programs we don't ship,
and have been obsolete since the VAX days.

xstr never worked in the build.sh cross-build environment (22 years), or
parallel make environment (nearly 28 years), didn't work in the orignal 386bsd
import, and has never been needed in NetBSD as we don't have the older BSD
programs (pascal, pre-nvi ex) that needed mkstr/xstr on PDP-11.

PR toolchain/35964

- Remove obsolete cltom(9) and mtocl(9).
- Add missing manual page's link of m_adj.

Register ppoll

Merge changes from current as of 20200406

Remove uyurex(4).

This is a driver for a "nonsense machine" made by the art group Maywa-Denki
in 2008. It was disabled by default.

Unfortunately even so it draws development attention (flaws found in the
code, MP-ification needs) and it is best not to continue to maintain this
driver.

Proposed without objections on tech-kern.

Revert the removal of filemon.

Retire filemon, discussed on tech-kern@.

Sync with HEAD

Add a C99 symbol to libm: nexttowardl

It's an alias for an already existing symbol nextafterl.

Patch obtained from <mgorny>

Detected by the LLVM buildbot breakage in tests.

Add a custom tokenizer which does not stem certain keywords.

Which keywords should not be stemmed is specified in the nostem.txt file.
(Right now I have taken all the man page names, split them if they had
underscores, removed common English words and converted everything to
lowercase.)

The tokenizer itself is based on the Porter stemming tokenizer shipped with
Sqlite. The code in custom_apropos_tokenizer.c is copy of that code with
some modifications to prevent stemming keywords specified in nostem.txt.

Additionally, it now uses underscore `_' also as a token delimiter. Therefore,
now it's possible to do query for `lwp' and all `_lwp_*' man page names
will be matched. Or the query can be `unconst' and `__UNCONST' will be matched.
This was not possible earlier, because underscore was not a delimiter and therefore
the index would have __UNCONST as a key rather than UNCONST.

The tokenizer needs fts3_tokenizer.h file, which is not shipped with the
amalgamation build of Sqlite, therefore it needs to be added here (unless
we decide there is a better place for it).

To enforce using the new tokenizer, a schema version bump is needed

Since the tokenization is done both at the indexing time (via makemandb) and
also while query time (via apropos or whatis), it will be needed to bump
the schema version everytime nostem.txt is modified. Otherwise the
index will consist of old tokens and desired changes will not be seen with
apropos.

This should also fix the issue reported in PR bin/46255. Similar suggestion was
also made on tech-userlevel@ recently:
<http://mail-index.netbsd.org/tech-userlevel/2017/06/08/msg010620.html>

Thanks to christos@ for multiple rounds of reviews of the tokenizer code.