CVS log for src/lib/libc/citrus/citrus_ctype_template.h

Up to [cvs.NetBSD.org] / src / lib / libc / citrus

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

fix various typos in comments.

lib: remove CONSTCOND comment

Since 2021-01-31, lint doesn't need it anymore for the common pattern of
'do ... while (0)'.

Don't overalign _RuneStatePriv, it must share the alignment of mbstate_t
it aliased with. Assert that the alignment actually used reflects the
alignment required by existing implementation and for newly build
modules assert that it is at most the guaranteed alignment.

Mostly merge changes from HEAD upto 20200411

PR/54414: Valery Ushakov: c99 requires that wcsrtombs(3) updates the source
argument on conversion error

Add a version of localedef(1), taken from FreeBSD, which works with
NetBSD's binary format.  LC_CTYPE should be mostly compatible but is
still missing pieces; the LC_COLLATE format is in a similar situation,
since it makes different assumptions than UCA/DUCET.  The other, smaller,
LC_* types should all work, but have not been extensively tested.

Make the compilation conditional on __STDC_ISO_10646__, so that applications
that need one of the East Asian locales but don't want __STDC_ISO_10646__
don't have to pay the penalty for it.

Move Unicode <-> ku/ten mapping into the individual codec modules.
Mapping is based on existing iconv data for single-byte encodings,
and included for several, but not all, multibyte encodings.

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

resync from head

Add mbsnrtowcs and wcsnrtombs. Approved by core.

sync with HEAD

u_intXX_t -> uintXX_t.

sync with HEAD

more PR/36938, restore internal-state if mbtowc() throw EILSEQ.
patch provided by Shiozaki-san, thanks.

sync with HEAD

lib/36938 mbtowc misbehaving after invalid char sequence
 - make sure to initialize mbtowc's internal state.
 - add regression test.

correct uninitialized value `err'.

Pullup rev 1.26-1.31 (requested by soda in ticket #875)

Fix 3 bugs in cirtus:
1. The return values of wcrtomb(3) and wctomb(3) are wrong,
   if second argument wc is L\0  and current locale is either
   ja_JP.ISO2022-JP or ja_JP.ISO2022-JP2.
2. There is a case that _FUNCNAME(ctype_wcrtomb) return garbage values on the
   stack
3. wctomb isnt supposed to set errno.

wctomb isn't supposed to set errno.

forgot to replace #else with #endif, thanks yamt again.

_FUNCNAME(ctype_wcrtomb):
set *nresult on error rather than using stack garbage.

oops, forgot to add "else"

corrent an error in previous commit:
don't corrupt return value on error case, pointed out by yamt

correct the return values of wcrtomb(3) and wctomb(3).
These return values should include the length of shift sequence
to restore the initial shift state.

KNF

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22280, verified by myself.

correct wcrtomb(3) - it is not conforming to standard on handling L'\0'.
This change also affects Citrus iconv engine.

Citrus iconv support (1)
add core engine of Citrus iconv and some support routines.
  - iconv core (citrus_iconv)
  - coded character set mapper (csmapper)
  - encoding scheme database (esdb)
  - standard encoding scheme handler (stdenc)
  - basic character set manipulation utilities (bcs)
  - mmap based infrastructure (citrus_lookup/_mmap/_memstream/_db)
  - some source codes of new plugins (not built yet)

Pull up revision 1.20 (requested by yamt in ticket #1202):
- handle s==NULL case of wcrtomb.
- add a comment about it.
fix rest of PR 18269 by wurlitzer.

Pull up revision 1.19 (requested by yamt in ticket #1202):
- add missing _CEI_TO_EI/_TO_CEI.
- don't use 'cl' before assert it's non NULL.

Pull up revision 1.18 (requested by yamt in ticket #1202):
- add missing _CEI_TO_EI/_TO_CEI.
- add missing _STATE_NEEDS_EXPLICIT_INIT checks.
- more comment.

Pull up revision 1.17 (requested by yamt in ticket #1202):
in _RESTART_BEGIN macro, don't use external variable directly.

Pull up revision 1.16 (requested by yamt in ticket #1202):
when s==0, wctomb should initialize its internal state and return if
encodings have state dependency or not.
a patch provided with PR 18269 by wurlitzer.
modified by me.
fix part of PR 18269.

Pull up revision 1.15 (requested by yamt in ticket #1202):
mbsrtowcs_priv template:
        n == 0 is not an error. it's a valid input.
_citrus_NONE_ctype_mbsrtowcs:
        fix a typo that causes SEGV.
while i'm here, make them similar each other.
fix PR 19620.

changes related to lib/20576 and lib/20577.
  - add wcstok() and wcswcs(),
  - add wcstok(3),
  - move btowc() and wctob() into each ctype modules,
  - bump ctype interface ABI version,
  - bump i18n modules' minor and
  - update sets.
and fix a stupid bug in _RUNE_ISCACHED macro.

Sync with HEAD.

- handle s==NULL case of wcrtomb.
- add a comment about it.

fix rest of PR 18269 by wurlitzer.

- add missing _CEI_TO_EI/_TO_CEI.
- don't use 'cl' before assert it's non NULL.

- add missing _CEI_TO_EI/_TO_CEI.
- add missing _STATE_NEEDS_EXPLICIT_INIT checks.
- more comment.

in _RESTART_BEGIN macro, don't use external variable directly.

when s==0, wctomb should initialize its internal state and return if
encodings have state dependency or not.

a patch provided with PR 18269 by wurlitzer.
modified by me.

fix part of PR 18269.

mbsrtowcs_priv template:
	n == 0 is not an error. it's a valid input.
_citrus_NONE_ctype_mbsrtowcs:
	fix a typo that causes SEGV.

while i'm here, make them similar each other.

fix PR 19620.

Catch up to -current.

1.6 pullup ticket number 11, from thorpej. Original commit message:

>Add some casts to quiet a "discards qualifier from pointer target type"
>warning from gcc 3.2 (experimental) (which is complaining about the loss
>of __restrict).
>
>Reviewed and approved by tshiozak@astec.co.jp.

Add some casts to quiet a "discards qualifier from pointer target type"
warning from gcc 3.2 (experimental) (which is complaining about the loss
of __restrict).

Reviewed and approved by tshiozak@astec.co.jp.

Catch up to -current.

- move _TO_CEI to template from each module/*.c.
- nuke _TO_EI and _TO_STATE.
- convert _TO_EI(cl) to _CEI_TO_EI(_TO_CEI(cl)).
- wrap long lines.

fix comments.
(_TO_INTERNAL_STATE -> _CEI_TO_STATE)

introduce new macro _STATE_NEEDS_EXPLICIT_INIT.
and use it for encodings that needs explicit
init_state in addition to zero-fill.

fix wcstombs: convertion should always begin with
an initial shift state. and nuke s_wcstombs.

fix state backup.
XXX this should be move into wcrtomb_priv.

- change _ENCODING_MB_CUR_MAX to take a pointer to encoding info.
- don't use MB_CUR_MAX here since it's depend on current locale.
  use _ENCODING_MB_CUR_MAX instead.

fix wcsrtombs to restart correctly.

- handle the case that buffer size == 0 for wcsrtombs.
- add a XXX comment.

fix mbtowc to return if the encoding's state-dependent or not
correctly.

calculate the return value of wcsrtombs correctly
for stateful encodings.

Catch up to -current.

file citrus_ctype_template.h was added on branch nathanw_sa on 2002-03-22 20:42:01 +0000

don't set errno here. caller will set it.

make mbsrtowcs work.(fix a typo)

refine i18n stuffs.
  - add libc/citrus directory.
    this directory contains the common stuffs not only for locale
    but also other facilities relating to i18n (e.g. iconv).
  - To keep the binary compatibility for the ctype modules easily,
    the interface between libc and the modules is simplified.
  - For the future integrated extension, module pool is renamed
    from "/usr/lib/runemodule" to "/usr/lib/i18n".
    In the future, this directory will contain the modules for "iconv",
    "collation", etc.
  - some cosmetic changes.
  - Bug fix for runetype.h; __attribute__((__packed__)) is placed at
    the wrong position and it is invalid unintentionally.
    But, the all members of the structures seem well-aligned.  Thus,
    this bug causes no problem, hopefully.

HEADS UP:
  - /usr/lib/rumemodule is obsoleted.  If you use the multibyte locales,
    you need to install /usr/lib/i18n/* from sys/lib/i18n_module .
  - The binary compatibility of /usr/share/locale/*/LC_CTYPE is probably kept.
    Perhaps, the bug fix about __packed__ mentioned above breaks the
    compatibility...  Be careful especially on 64bit platforms.