CVS log for src/usr.bin/make/str.c

Up to [cvs.NetBSD.org] / src / usr.bin / make

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

make: only generate code for cleanup functions in CLEANUP mode

make: error out on a matching malformed matching pattern '[['

make: make string matching platform-independent

Previously, whether the character range '[a-ä]' matched, depended on the
signedness of the plain 'char' type.  Since make operates on byte
strings and does not support UTF-8 or other multi-byte character
encodings, this edge case is not expected to occur in practice.

No change in the unit tests as this edge case is not covered by tests.

make: miscellaneous cleanups

make: speed up pattern matching in the ':M' modifier

In the common patterns where '*' is followed by a regular character,
such as in the patterns '*.c' or '*.mk', search the next possible
matching position in a small loop, instead of repeatedly comparing the
first remaining pattern character to all special characters.

make: eliminate a local variable in Str_Match

No functional change.

make: reduce indentation in pattern matching code

No functional change.

make: warn about malformed patterns in ':M', ':N' and '.if make(...)'

These patterns shouldn't occur in practice, as their results are tricky
to predict.  Generate a warning for now, and maybe an error later.

Reviewed by sjg@.

make: unclutter string matching code

make: rename variables in string matching, remove redundant code

No functional change.

make: speed up pattern matching in the ':M' and ':N' modifiers

In the code coverage report, the highest count for Str_Match goes from
5,298,924 down to 79,646.

make: clean up comments

make: clean up comments for string splitting and string matching

No binary change.

make: condense Str_Match

The test for '\\' followed by '\0' was redundant since at that point,
*str is guaranteed to be not '\0', which takes the next 'return false'.

No functional change.

make: document platform dependency in string pattern matching

No unit test for this edge case since all other unit tests are platform-
independent.

To reproduce:
$ make clean
$ make -s PROG=s-make NOMAN=yes USER_CFLAGS=-fsigned-char
$ make clean
$ make -s PROG=u-make NOMAN=yes USER_CFLAGS=-funsigned-char
$ make clean
$ range=$(lua -e 'print(("[%c-%c]"):format(0xe4, 0x61))')
$ ./s-make -V "\${:UM:M$range}\${:UN:N$range}"
M
$ ./u-make -V "\${:UM:M$range}\${:UN:N$range}"
N

make: clean up low-level comments, eliminate common subexpression

No functional change.

make: make code for string matching syntactically more consistent

No functional change.

make: in CLEANUP mode, free interned strings at the very end

Noticed by sjg.

make: fix memory leak for filenames in .for loops (since 2013-06-18)

Previously, each time a .for directive pushed its buffer on the input
file stack, the current filename was duplicated.  This was a waste of
memory.

The name of a file is typically only used while it is read in.  There is
one situation when the filename is needed for longer, which is when a
target is defined.

Since .for loops are implemented as a special form of included files,
each .for loop duplicated the current filename as well.

$ cat << EOF > for.mk
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.endfor
.endfor
.endfor
.endfor
.endfor
.endfor
.endfor

all:
	@ps -o rsz -p ${.MAKE.PID}
EOF

$ make-2021.12.13.03.55.16 -r -f for.mk
  RSZ
10720

$ ./make -r -f for.mk
 RSZ
1716

The difference is 8 MB, which amounts to 1 million .for loops.

make: eliminate local variable in Substring_Words

No functional change.

sync with head

make: inline str_concat4

This function is only ever used for forming strings of the form
"archive(member)".

No functional change.

make: avoid unnecessary calls to strlen when evaluating modifiers

No functional change.

make: backport to C90

In the past few months I had accidentally used C99 features in the make
code.  According to tools/README, tools that are used in the build
system should restrict themselves to C90.

This allows make to build with GCC's options "-pedantic
-Wno-system-headers -Dinline= -Wno-error=cast-qual".

I didn't notice anyone actively complaining though, I just wanted to see
how much work this backporting would be.  The identifier __func__ is
still used, as in other tools.

No functional change.

make: use C99 bool type instead of defining its own

No functional change.

make: document necessary tests for Str_Match

make: replace pre-increment with post-increment or simple addition

The rest of the code already prefers post-increment if there is no
actual difference.

make(1): remove do-not-format markers from comments

These markers had been used inconsistently.  Furthermore the source code
had not been formatted automatically before 2020 at all, otherwise there
wouldn't have been any trailing whitespace left.

make(1): make a few more bool expressions more precise

The previous version of lint(1) from a few hours ago didn't catch all
occurrences.  And even the current one doesn't catch everything.
Function arguments and return types still need some work.  The "return
quietly" from shouldDieQuietly still implicitly converts from int to
_Bool.

No functional change.

make(1): consistently use boolean expressions in conditions

Most of the make code already followed the style of explicitly writing
(ptr != NULL) instead of the shorter (ptr) in conditions.

The remaining 50 instances have been found by an experimental,
unpublished check in lint(1) that treats bool expressions as
incompatible to any other scalar type, just as in Java, C#, Pascal and
several other languages.

The only unsafe operation on Boolean that is left over is (flags &
FLAG), for an enum implementing a bit set.  If Boolean is an ordinary
integer type (the default), some high bits may get lost.  But if Boolean
is the same as _Bool (by compiling with -DUSE_C99_BOOLEAN), C99 6.3.1.2
defines that a conversion from any scalar to the type _Bool acts as a
comparison to 0, which cannot lose any bits.

make(1): format multi-line comments

make(1): remove redundant call to strlen in Str_Words

make(1): use postfix increment where possible

make(1): clean up make.h, meta.c, parse.c, str.c

The main changes are in the comments, which have been shortened and
corrected.

Some local variables changed their names.

In ParseErrorInternal, the scope of va_start is now narrower.

In ParseDoDependency, the type of tOp has been fixed.

ParseGetLine doesn't take flags anymore but instead a parsing mode.
Previously, the flags had not been combined anyway.

At the beginning of Parse_File, fatals is already guaranteed to be 0, and
even if not, it would be wrong to just discard the fatal errors.

make(1): add pp_skip_hspace to skip horizontal whitespace during parsing

make(1): clean up code stylistically

* Replace character literal 0 with '\0'.
* Replace pointer literal 0 with NULL.
* Remove redundant parentheses.
* Parentheses in multi-line conditions are not redundant at the
  beginning of a line.
* Replace a few !ptr with ptr == NULL.
* Replace a few ptr with ptr != NULL.
* Replace (expr & mask) == 0 with !(expr & mask).
* Remove redundant braces for blocks in cases where the generated code
  stays the same.  (Assertions further down in the code would get
  different line numbers.)
* Rename parameters in CondParser_String to reflect the data flow.
* Replace #ifdef notdef with #if 0.

The generated code stays exactly the same, at least with GCC 5.5.0 on
NetBSD 8.0 amd64 using the default configuration.

make(1): remove macros MIN and MAX

These macros typically evaluate one of their arguments twice.  Until
2020-08-31, they had not parenthesized their arguments properly.  They
are only used in a few places, therefore it doesn't hurt much to have
them expanded.

make(1): remove redundant type casts

This mainly affects the void pointers in callback functions for lists.
These had been necessary once when the parameter type was still
ClientData instead of void pointer.

make(1): revert previous commit

It had accidentally reverted all the work from the past few days.

make(1): fix double-free bug in -DCLEANUP mode (since 2020-10-02)

The bug had been introduced with dir.c 1.155 on 2020-10-02 22:20:25.  In
that commit, openDirectories was replaced with a combination of a list
with a hash table, for more efficient lookup by name.

Upon cleanup, OpenDirs_Done is called, which in turn called
Dir_ClearPath.  Dir_ClearPath takes full ownership of the given list and
empties it.  This was no problem before since afterwards the list was
empty and calling Lst_Free just frees the remaining list pointer.

With OpenDirs, this list was combined with a hash table, and the hash
table contains the list nodes, assuming that the OpenDirs functions have
full ownership of both the list and the hash table.  This assumption was
generally correct, except for the one moment during cleanup where full
ownership of the list was passed to Dir_ClearPath, while the hash table
still contained pointers to the (now freed) list nodes.  This by itself
was not a problem since the hash table would be freed afterwards.  But
as part of Dir_ClearPath, OpenDirs_Remove was called, which looked up
the freed directory by name and now found the freed list node, trying to
free it again.  Boom.

Fixed by replacing the call to Dir_ClearPath with code that only frees
the directories, without giving up control over the list.

make(1): inline Str_FindSubstring in JobOutput

make(1): clean up RCSID blocks

These blocks mostly consisted of redundant structure, following the same
#ifndef pattern over and over, with only minimal variation.

It's easier to maintain if the common structure is only written once and
encapsulated in a macro.

To avoid "defined but unused" warnings from GCC in the case where
MAKE_NATIVE is not defined, I had to add volatile.  Adding
MAKE_ATTR_UNUSED alone would not preserve the rcsid variable in the
resulting binary.

make(1): replace brk_string with Str_Words

The API is much simpler, and there is less detail that is exposed by
default and fewer punctuation to type on the caller's side.  To see that
there is some memory to be freed, one would have to look into the
struct.  Having part of the return value as the actual return value and
the rest in output parameters was unnecessarily asymmetrical.

make(1): allow for strict type checking for Boolean

Having Boolean aliased to int creates ambiguities since int is widely
used.  Allow to occasionally compile make with -DUSE_DOUBLE_BOOLEAN to
check that the type definitions still agree.

make(1): make brk_string return size_t for the number of words

make(1): prepare str.c for WARNS=6

The next step is to make brk_string return size_t instead of int.

make(1): add str_concat4 to make the other code simpler

There's no need for arch.c to call strlen when there is a high-level API
for the same purpose.

make(1): replace str_concat with str_concat2 and str_concat3

The new functions have a simpler interface, and str_concat3 is even more
general-purpose, since the middle string is no longer required to be
exactly of length 1.

make(1): fix parameter name of str_concat

The previous documentation mentioned Str_Concat, but str_concat has been
written in lowercase for years.  The "flags" are not flags since they
cannot be combined, not even when they are written in hex.

make(1): fix integer type in str_concat

make(1): format code in str.c consistently

make(1): no declaration-after-statement anymore

NetBSD make is intended to be maximally portable, therefore it uses only
C89.  This was not declared in the Makefile before.

There are still a few places in parse.c and metachar.c that use
end-of-line comments.  These will be fixed in a follow-up commit.

make(1): reduce scope of local variables in brk_string

This also removes the unused assignment to words_cap.

make(1): document the circumstances in which brk_string returns NULL

make(1): move SysV string matching to var.c

This kind of string matching is only used in variable modifiers, and only
if this feature is enabled by SYSVVARSUB.

make(1): refactor Str_Match

- use shorter variable names to avoid line breaks
- eliminate c2 variable
- change return type to Boolean

make(1): clean up Str_Match

- removed unnecessary parentheses
- uniform test for end of string

make(1): clean up documentation of Str_Match

make(1): fix performance problem in specially crafted :M modifier

This fix was previously suspected to make the vax build fail.  The next
build succeeded though, and it started 2 hours before this fix was
reverted.

usr.bin/make: revert performance improvement

That change might be the cause of a build failure for vax.
http://releng.netbsd.org/builds/HEAD/202006131940Z/ builds fine.
http://releng.netbsd.org/builds/HEAD/202006141020Z/ doesn't.

The build fails with:
/home/source/ab/HEAD/src/external/gpl3/gcc/dist/gcc/machmode.h:524:28:
error: 'mode_size_inline' was not declared in this scope

usr.bin/make: make Str_Match faster for repeated asterisks

Conceptually related to https://en.wikipedia.org/wiki/ReDoS.

usr.bin/make: remove redundant parentheses around return

usr.bin/make: consistently use ++ for incrementing pointers

usr.bin/make: fix typo in comment

Behave like gmake: empty string does not match % pattern

usr.bin/make: refactor brk_string

The variables are renamed to reflect to which memory region each pointer
belongs.

The variable "curlen" was always zero.

The type of "ch" has changed to char, and its scope is now limited to
its actual use.

Comparisons of pointers are now consistently written as p != NULL
instead of !p, like character comparisons are written as ch != '\0'.

The "store_words_buf" is updated when the function returns, not before.

- Percent on the rhs is special only if the lhs has one too.
- If the rhs does not have percent, but the lhs has, the result is the rhs
This behavior matches gmake

Merge changes from current as of 20200406

Fix out-of-bounds read in Str_Match.

Sync with HEAD

Str_Match: fix closure tests for [^] and add unit-test.

Sync with HEAD

Str_Match: allow [^a-z] to behave as expected.

Fully avoid the nonstandard types u_char, u_int, and u_long, which
were only used inconsistently in a few places anyway.

Rebase to HEAD as of a few days ago.

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

Reduce the calls to realloc from brk_string by basing initial argmax guess
on length of input.

Reviewed by: christos

sync with head

Note in comment that Str_Match() does not detect or report invalid
patterns. This is not critical, since for the moment both possible
cases (unclosed [], and \ at the end of the string) are handled more
or less sanely.

Sync with HEAD.

Third (and last) commit. See http://mail-index.netbsd.org/source-changes/2009/05/13/msg221222.html

replace c++ style comment by a c-style one

Change 'Buffer' so that it is the actual struct, not a pointer to it.
Saves having to malloc/free a fixed size structure.
Buf_Init() now takes ptr to Buffer to initialiase.
Change Buf_Destroy() to return ptr to string when not freed.
Remove large number of casts to (Byte) and (Byte *) - 'Byte' is 'char' here.
Buf_AddByte[s] guarantees that the data is 0 termininated, so never add '\0'.
Keep 'count' not 'left' and 'inPtr', code is simplier with only one update.
Fix fallou, no functional change.

PR/32154: Roland Illig: brk_string fails to report errors in unquoted strings.

PR/34138: Roland Illig: In the non-expand case, fix brk_string not to eat
quotes.

Don't use emalloc and friends directly, but call them consistently
bmake_malloc and friends. Implement them via macros for the native case
and provide fallback implementations otherwise. Avoid polluting the
namespace by not defining enomem globally. Don't bother to provide
strdup and strndup, they were only used for the estrdup and estrndup
comapt code.

This addresses the presence of emalloc in system libraries on A/UX and
resulted strange issues as reported by Timothy E. Larson.

back all changes out until I fix it properly.

- use pid_t/size_t as appropriate instead of int.
- use %ld to print pids.
- fix a bit of lint.
- WARNS=4

Sync w/ NetBSD-4-RC_1

Apply patch (requested by tron in ticket #696):
	usr.bin/make/compat.c		patch
	usr.bin/make/cond.c		patch
	usr.bin/make/dir.c		patch
	usr.bin/make/for.c		patch
	usr.bin/make/main.c		patch
	usr.bin/make/make.1		patch
	usr.bin/make/make.c		patch
	usr.bin/make/make.h		patch
	usr.bin/make/nonints.h		patch
	usr.bin/make/parse.c		patch
	usr.bin/make/str.c		patch
	usr.bin/make/targ.c		patch
	usr.bin/make/util.c		patch
	usr.bin/make/var.c		patch

Synchronize make(1) with HEAD branch to increase perfomance and
improve stability.

Correctly handle '\' at the end of a variable in the non-expanding case.

If a word is "" eat the last quote since we ate the first.

From Max Okumoto:
- Remove casts to NULL.
- Remove space between cast and object.

PR/29203, PR/29204: Max Okumoto: KNF changes to make [no functional changes]

Pull up revision 1.22 (requested by sjg in ticket #282):
Simplify build, no functional changes.
Instead of adding MAKE_BOOTSTRAP for hosted environments, i.e., when
you want things simple, instead add MAKE_NATIVE to get those hugely
important features like __RCSID().
It's now possible to build make on some hosts with: cc *.c */*.c

Simplify build, no functional changes.

Instead of adding MAKE_BOOTSTRAP for hosted environments, i.e., when
you want things simple, instead add MAKE_NATIVE to get those hugely
important features like __RCSID().

It's now possible to build make on some hosts with: cc *.c */*.c

Implement :[] modifier to allow picking a range of words out of a variable.
Also :tW and a W flag to :C and :S to allow treating value as a single word.
Add unit tests for the above, and fix some corner cases.
Based on patches supplied by Alan Barrett <apb@cequrux.com>

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22365, verified by myself.

Pass WARNS=3

Remove !__STDC__ stuff, de-__P(), ANSIfy, and de-register.

- full prototypes
- more conservative printf
- char -> unsigned char

Sync with trunk, per request of christos.

PR/5210: Hauke Fath: make core dumps with .SHELL
Unfortunately this revealed a deeper problem with the brk_string code.
To fix it:
	- remove sharing of the buffer between brk_string invocations
	- change the semantics of brk_string so that the argument array
	  starts with 0, and return the buffer where the strings are
	  stored

wrap #include <sys/cdefs.h>, __RCSID(...) stuff in #ifndef MAKE_BOOTSTRAP

Add WARNS=1
RCSID police

Update make(1) from trunk, by request from Christos Zoulas.  Fixes many bugs.

Import 4.4BSD-Lite2 sources onto CSRG branch (already merged at head)

- Merge in FreeBSD and Lite2 changes.
- Fix bug where a non-archive target with a .a suffix would always
  be considered to be out of date, since it does not have a TOC.

Added erealloc() function that calls enomem() if the allocation fails
(this is like the existing emalloc() function).  Changed all realloc()
calls to erealloc().

Update from trunk: Always ensure space for trailing NULL in argv

Always ensure space for trailing NULL in argv

- $NetBSD$ rcsids
- Fixed so that .[A-Z]* targets that do not match keywords are ignored as
  Posix mandates
- Added .PHONY target keyword

str.c: Don't free NULL; does not work with non posix frees.
var.c: Minor memory leak plugged.
suff.c: Don't add extra sources on the null suffix if it has dependency
	lines or commands attached to it [POSIX says so]

Christos' fix for quoting variable extraction properly.

Fixes from Christos Zoulas, who used purify, objectcenter and testcenter
to find memory leaks and illegal memory accesses.

Properly parse null command line arguments in makefile rules.
Fix from Christos Zoulas.

Fixes from Christos Zoulas:
The following two patches fix a couple of problems with make(1)

1. Null Suffixes were not being copied, but they were being free'd
   This caused rules of the form:

   .c:
	${CC} ...

   to access invalid memory and potentially core dump..
   [That was always broken; I did not break that one :-)]

2. My recent fixes to parse ${VAR:%.foo=%.bar} handled the null string
   case incorrectly (${VAR:=.c} was broken).

fixes/improvements from Christos Zoulas <christos@deshaw.com>.

Include appropriate header files to bring prototypes into scope.

Add RCS identifiers.

initial import of 386bsd-0.1 sources

Initial revision