CVS log for src/usr.bin/make/var.c

Up to [cvs.NetBSD.org] / src / usr.bin / make

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

make: fix confusing error message when overriding a read-only variable

make: simplify printing of an evaluation stack element

make: reduce line length in error messages

The error messages for deeply nested parse errors were hard to decipher,
due to the large amount of text.  Split these messages into individual
lines, just as in the backtrace for .include files and .for loops.  This
unified backtrace makes the output more uniform.

make: treat recursive variables non-fatally

A recursive variable is no worse than an unknown modifier, so treat them
in the same way by continuing parsing until the end of the makefile.

make: add more context to error message about recursive variables

make: remove wrong error message about an undefined variable

make: error out on syntax errors in ':M' and ':N' modifiers

More than a year ago, the warning has been added.  Now it has been
promoted to an error.

make: move initialization of variable scopes to targ.c

The variable scopes are freed by Targ_End, so initialize them there as
well.  Separate printing statistics and freeing memory, which makes
Var_End unnecessary.

make: in error messages, distinguish parsing from evaluating

make: in error messages for anonymous variables, log the value

make: error out on unclosed expressions during parse time

In exchange, this adds location information.

For unnamed expressions, the value is no longer printed.  This will be
added back in a follow-up commit.

make: reduce lint-specific comments about ARGSUSED

make: error out on the "Bad modifier" error message

Previously, the "Bad modifier" error message did not affect make's exit
status.  As a side effect, this kind of error now gets more context
information.

make: add more context information to error messages

In case of a parse error or evaluation error, print the variable value
in addition to the variable name, to see the effects of previous
expression modifiers.

In nested make calls, print the current directory at the bottom of a
stack trace, as that information is otherwise hard to get in a parallel
build spanning multiple directories.

make: trim trailing whitespace

make: add :tc to capitalize first letter of each word

This is very hard to do without :tc

Reviewed by: rillig

Sync with HEAD.

make: error out on some more syntax errors

Previously, these errors only produced a message on stderr.  They only
affected make's exit status when they were evaluated at parse time, but
not when evaluating the commands for a specific target right before
executing them.

The affected syntax errors are:
* invalid regular expressions in the ':C' modifier
* out-of-range references to regex groups in the ':C' modifier
* unfinished modifiers

make: error out on syntax error in conditions in ':?then:else' modifier

The 'Error' function only reports errors but does not affect the exit
status, the 'Parse_Error' function does, while providing more details to
find the cause of the syntax error.

make: sync error handling between '!=' assignment and '::!=' modifier

make: add more context to "returned non-zero status" message

Previously, this message was an "error" but had no influence on the exit
status, so make it a warning instead.  In the seldom used -W mode that
treats warnings as errors, this change influences the exit status.

make: error out if an internal read-only variable is overwritten

The affected variables are:
* .newline
* .MAKE.OS
* .MAKE.JOBS.C
* .MAKE.LEVEL.ENV
* .MAKE.PID
* .MAKE.PPID
* .MAKE.UID
* .MAKE.GID

This change does not affect the .SHELL and .SUFFIXES variables, as well
as variables that are manually turned read-only by the special .READONLY
target.

make: clean up collection of context information for error messages

make: sync VarEvalMode constant names with their debug log names

make: fix out-of-bounds read when parsing indirect modifiers

make: add .export-all

An explicit syntax for exporting all global variables is much safer
than allowing .export with no argument to do the same.

Add .export-all and have .export with no argument throw a warning saying
to use .export-all

Reviewed by: rillig

make: remove unused VARE_EVAL_KEEP_DOLLAR

make: merge functions that parse modifier parts

This commit subtly changes the behavior for the :gmtime and :localtime
modifiers, but only in irrelevant cases since the time value must be an
integer, and neither '\' nor ':' nor '$' may occur in these values.

make: clean up API for iterating over hash tables

make: don't log anything when freeing memory

make: fix some more memory leaks

make: fix a few more memory leaks

make: in -DCLEANUP mode, free variables and their values

The variables in the 3 scopes must be freed before the scopes themselves
are freed by Targ_End.

The test opt-m-include-dir creates a directory of the form '*.tmp', thus
it must be removed before attempting to only remove regular files of
this name.

POSIX requires setenv to copy the passed name and value, so there is no
need to keep that memory allocated any longer.

make: all command line overrides go in .MAKEOVERRIDES

Not all variables that start with '.' are internals,
and unless they are explicitly flagged as internal should go
into .MAKEOVERRIDES

Update varname-dot-makeoverrides to check this.

Also avoid using SCOPE_CMDLINE when ReadOnly will do.

make: don't reallocate memory after evaluating an expression

When an expression is evaluated, the resulting text is short-lived in
almost all cases.  In particular, the compaction neither affects the
target names nor the global variable values, which are the prime
candidates for permanent memory usage.

make: fix comment about forcing a use-after-free

The previous expression didn't cause any bug, as the modifier
':@VAR@loop@' changed the value of the expression, thus making the
expression independent from the variable value.

Instead, the variable needs to be deleted from within an indirect
modifier, and that modifier needs to evaluate to an empty string, thus
doing nothing and preserving the original expression value.

make: clean up, test .NOPATH

Trim down the comments in the archive module, as they mainly repeated
the code.  Trim down the binary code size in the archive module, as it
is rarely used.

In Var_Parse, delay two variable assignments until they are actually
needed.

make: clean up comments, code and tests

make: trim down code for parsing the :gmtime and :localtime modifiers

The :gmtime and :localtime modifiers are not used often and thus are not
time-critical. Exchange the custom code that parses an integer from a
substring for an additional memory allocation.

Thanks sjg@ for suggesting to avoid the custom parsing code.

make: fix out-of-bounds read when evaluating :gmtime and :localtime

The function TryParseTime takes a pointer to a string, but the LazyBuf
returns a Substring, which is not guaranteed to be null-terminated or
delimited.  In TryParseTime, calling strtoul on the Substring read past
the end of the substring.

Noticed in the NetBSD build in libntp, where the :gmtime modifier is
used in two places with the same timestamp value, of which the first was
evaluated correctly and the second wasn't.

The bug was introduced in var.c 1.1050 from 2023-05-09, when the
argument of the :gmtime and :localtime modifiers was allowed to be an
expression instead of an integer constant.

make: provide more context information for parse/evaluate errors

make: fix type mismatch in lint's strict bool mode (since today)

make: export target scope values

Pass target scope to Var_ReexportVars so that a target process
will see the correct values in its env.
We must then mark any Global scope variable as unexported
so targets without local value get the Global one.

make: remove unneeded conditional-compilation toggles

The toggles INCLUDES, LIBRARIES, POSIX, SYSVINCLUDE, SYSVVARSUB,
GMAKEEXPORT and SUNSHCMD are no longer needed, they were unconditionally
set.

The toggle NO_REGEX was configurable from the command line, but
disabling it would result in various error messages about the unknown
':C' modifier.

OK sjg@.

make: do not evaluate indirect modifiers in parse-only mode

Discovered by sjg.

make: in parse-only mode, don't evaluate modifiers

Previously, the ':S', ':ts', ':tA' and ':from=to' modifiers were
evaluated in parse-only mode, unnecessarily.  This is only noticeable
when an indirect modifier is evaluated in parse-only mode, which is
another bug that will be fixed in a follow-up commit.

make: state of save_dollars affects what is a parse error

When save_dollars is false it is not a parse error to
encounter $$ rather than \$.

make: clean up redundant 'const' from automatic variables

No binary change.

make: clean up comments, constify shell name

make: miscellaneous cleanups

make: clean up string functions

No binary change.

make: inline Substring_Sub

Remove redundant assertions.

make: simplify debug message for the ':@var@...@' modifier

The previous variant was hard to understand.

make: clean up variable handling

All variables from the command line scope have the fromCmd flag set, so
there is no need to check for it.

Inline redundant local variables.

Variables from a scope cannot be short-lived, so there is no need to
call VarFreeShortLived.

No functional change.

make: clean up comments

No binary change, except for line numbers in assertions.

make: simplify memory allocation for string buffers

In edge cases and short-lived buffers, the initial buffer size is
irrelevant, so use the default.

No functional change.

make: use consistent debug messages style when ignoring variables

When a variable is not modified or not deleted, clearly say so and state
the reason.  Use the same style of debug messages everywhere, putting
the word 'ignoring' at the front.  Previously, that word sticked out to
the right, but only in some cases.

make: fix confusing debug logging when deleting a variable

Be clear when deleting a variable is ignored and state the reason,
instead of only providing a keyword as a hint for insiders.

make: clean up comments

No binary change, except for line numbers in assertions.

make: clean up names of local variables

No binary change.

make: explain in the debug log why a variable assignment is ignored

make: remove redundant text from diagnostic about bad '?:' modifier

make: clean up the check for command line variables

It looked suspicious that to check whether a variable was set via the
command line, the variable value would be needed.  Moving the debug
diagnostic to the calling function resolved this smell.

A variable from the command line scope is never short-lived, so there's
no need to clean up after accessing the variable.

No functional change.

make: clean up comments and local identifiers

No binary change, except for line numbers in assertions.

make: document how to trigger a use-after-free bug

Since 2017-02-01.

make: fix crash in ':C' modifier on an empty expression

Since var.c 1.982 from 2021-12-13.

make: Var_SetWithFlags set fromCmd for SCOPE_CMDLINE

Set fromCmd true for SCOPE_CMDLINE
regardless of the variable name.

Reviewed by: rillig

make: replace 'variable expression' with 'expression' in comments

No binary change.

make: replace 'variable expression' with 'expression' in diagnostics

tests/make: cover all code paths for the ':mtime' modifier

make: produce more accurate error message for invalid ':mtime' argument

make: extract detection of ':from=to' to a separate function

No functional change.

tests/make: test and explain exporting of variables

make: clean up the modifier ':[...]'

No functional change.

make: when comparing substrings, don't read beyond the substring's end

Right now, Substring_Words terminates each word with a '\0', but that's
an implementation detail that is not required by the interface, so don't
rely on it.

make: miscellaneous cleanups

No functional change.

make: clean up comments

No functional change.

make: sync comments with the extracted ParseModifier_Match

No functional change.

make: only work around wrong strftime if actually necessary

The workaround is only needed if the time format contains '%s', in all
other cases there is no need to preserve, set and restore the TZ
environment variable.  Suggested by sjg@.

Only check for 's' in the format string, not for '%s', to allow for
optional modifiers of the conversion specifier.

make: clean up comments regarding gmtime/strftime

These two functions are not supposed to be used together, thus it is not
a bug if combining them produces unexpected results.

make: work around bugs in gmtime on several platforms

make: be strict when parsing the argument of the ':mtime' modifier

make: clean up variable names and indentation for the ':mtime' modifier

No functional change.

make: clean up variable and function names

No functional change.

make: warn about malformed patterns in ':M', ':N' and '.if make(...)'

These patterns shouldn't occur in practice, as their results are tricky
to predict.  Generate a warning for now, and maybe an error later.

Reviewed by sjg@.

make: merge common code for handling the ':M' and ':N' modifiers

No functional change.

make: do not allow delete of readOnly variable.

Sometimes we mark a variable readOnly to guard against .undef

make: shorten function names, clean up comments

No functional change.

Make :mtime operate on each word in variable value.

Reviewed by: rillig

make: improve error message for failed stat in ':mtime'

make: :mtime=error throw error on stat(2) failure

Sometimes we want fatal error if stat fails on the
presumed pathname.

make: add :mtime to provide mtime of file

The value of the variable is passed to stat(2)
and st_mtime is new value.
An optional arg can be used if stat(2) fails, otherwise
the current time is used.

See varmod-mtime.mk for usage examples.

make: allow ':gmtime' and ':localtime' with dynamic argument

This allows ${%Y:L:gmtime=${mtime}} instead of the indirect
${%Y:L:${:Ugmtime=${mtime}}}.

The direct form also prevents any ':' from the nested expression to be
interpreted as a separator, which doesn't matter for the ':gmtime' and
':localtime' modifiers but will prove useful for other modifiers that
follow the same pattern.

make: declare all common symbols in headers, unexport others

No functional change.

make: fix memory leak when parsing ':from=to' modifiers

Since var.c 1.925 from 2021-04-12.

make: fix parsing of unevaluated subexpressions with unbalanced '{}'

Since var.c 1.323 from 2020-07-26, modifiers containing unbalanced
braces or parentheses were parsed differently, depending on whether they
were relevant or not.

For example, the expression '${VAR:...}' is enclosed with braces. When
this expression has a modifier ':S,},}},g' that would double each '}' in
that expression, the parser got confused:

If the expression was relevant, the modifier was parsed as usual, taking
into account that the 3 '}' in the modifier are ordinary characters.

If the expression was irrelevant, the parser only counted the '{' and
the '}', without taking into account that a '}' might be escaped by a
'\' or be an ordinary character.  Parsing therefore stopped at the first
'}', assuming it would finish the expression '${VAR:S,}'.

This parsing mode of only counting balanced '{' and '}' makes sense for
the modifier ':@var@...@', which expands each word of the expression
using the template from the '...'.  These templates tend to be simple
enough that counting the '{' and '}' suffices.

make: inline macros for variable names

The variable name '.MAKEOVERRIDES' was already used in the non-macro
form.

No binary change.

make: inline macro for variable name ".MAKE.EXPORTED"

The variable name is distinctive enough to be searched directly in the
code instead of having a named constant for it.

No binary change.

make: remove redundant type VarParseResult

No functional change.

make: clean up calls to Var_Subst

None of the calls to Var_Subst used the return value, and the return
value was always VPR_OK.

No functional change.

make: reduce complexity of evaluating expressions

No functional change.

make: extract the decision of using a specific log format

No functional change.

make: ApplyModifier_Remember always use SCOPE_GLOBAL

Conditionals evaluate vars with SCOPE_CMDLINE, if :_
is used within a conditional expression and is thus
set in SCOPE_CMDLINE, it breaks any use of :_ in SCOPE_GLOBAL.

The simplest solution is for :_ to always use SCOPE_GLOBAL

Reviewed by: rillig

make: some variables should be read-only

Make variables like .newline and .MAKE.{GID,PID,PPID,UID} read-only.

Reviewed by: rillig

make: log adjustments of var.readOnly

make: .[NO]READONLY for control of read-only variables

Reviewed by: rillig

make: inline LazyBuf_AddBytesBetween

No binary change.

make: constify, fix comment indentation

No functional change.

make: merge duplicate code

No functional change.

make: set WARNS to 6, from the default 5

No binary change on x86_64.

make: prevent future out-of-bounds errors when parsing expressions

A modifier in an expression ends not only at the next ':' or at the
closing '}' or ')', but also at the end of the string.

Previously, testing for the end of the string had been done separately,
which was error-prone since 2006-05-11, when indirect modifiers were
introduced.  Since then, it was possible that the string terminator '\0'
was accidentally skipped in cases where the loop condition only tested
for the ending character.  When parsing indirect modifiers, the ending
character is indeed '\0', but when parsing direct modifiers, it is '}'
or ')'.

A welcome side effect is that in the case of unclosed expressions such
as '${VAR:Modifier', the amount of error messages is reduced from 2 or 3
to only 1.  The removed error messages were wrong and thus confusing
anyway.

make: fix out-of-bounds read when parsing the ':M' modifier

Since at least 2009-01-17, probably already since 2006-02-18, when
modifiers were allowed to be nested expressions.

make: fix out-of-bounds read when parsing the ':D' modifier

Since 2000-04-29, when the ':D' and ':U' modifiers were added.

make: revert parsing of modifier parts (since 2022-08-08)

The modifier ':@var@body@' parses the body in parse-only mode and later
uses Var_Subst on it, in which each literal '$' must be written as '$$'.

Trying to parse the loop body using Var_Parse treated the text
'$${var:-0}' as a single '$' followed by the expression '${var:-0}',
wrongly complaining about the 'Unknown modifier "-0"'.

Found by sjg.

make: fix parsing of modifiers containing unbalanced subexpressions

make: when parsing ':D' or ':U', only copy text if necessary

When parsing the expression ${:D any ${uninterpreted} text}, parsing of
the ':D' modifier only needs to skip to the next delimiter, there is no
need to keep record of the text that has been skipped.  In this
particular example, there had been an unnecessary memory allocation
since the nested expression was not copied to the result buffer.  Due to
that, the resulting buffer contained " any  text", which was not a
substring of the text passed to the parser, thereby forcing the memory
allocation.

No functional change.

make: extract parsing of ':D' and ':U' modifiers into separate function

No functional change.

make: document parsing of short variable names, such as $i

No binary change.

make: reduce indentation in ParseVarnameShort

No functional change.

make: simplify return type of IsShortVarnameValid

No functional change.

make: reorganize Parse_Error

Determining the location where the error occurred is now done by
ParseVErrorInternal.  This frees the remaining code from keeping the
filename and the line number together.  It also makes Parse_Error short
enough that it might be worth providing a separate function for each of
the 3 log levels.

No functional change.

make: clean up comments in Var_Parse

make: remove obsolete comment

All instances of VarFind that could be replaced with VarFindSubstring
have been replaced.

make: fix spacing, and a typo in a test

make: avoid trailing whitespace in debug log for variables

Since trailing whitespace is invisible, describe the variable value in
words to make it visible.

make: add space after colon in debug logging for variables

These log messages are intended for human interpretation, so don't make
them unnecessarily hard to read.

make: prefer 'long long' over 'long' on 32-bit C99 platforms

When sorting the words of an expression numerically using the modifier
':On' (added on 2021-07-30), use 64-bit numbers even on 32-bit
platforms.  A typical use case is comparing file sizes.

When tracing the execution of jobs, fix an integer overflow after 2038.
32-bit platforms that use a pre-C99 compiler still have this problem.

No change to the test suite since most tests simply skip any potential
differences between 32-bit platforms and 64-bit platforms (see
varmod-order-numeric.mk) or already account for both variants (see
varmod-localtime.mk).

make: clean up comments

make: fix crash on .undef of an environment variable (since 2020-10-06)

make: simplify ParseModifier_Match

No functional change.

make: remove comment about environment variable with empty name

At least on NetBSD, make cannot be fooled with an environment variable
having an empty name.  When running '/usr/bin/env =undefined make', the
argument is parsed as a variable assignment, but putenv(3) refuses to
process an empty variable name.  Calling execve(2) directly got a step
further, the kernel didn't filter '=undefined' from the environment
variables.  But getenv(3) always returns NULL when querying the
environment variable with the empty name.

On other operating systems, things may be different.  Trying to set an
environment variable with an empty name may cause errors in env(1),
putenv(3), execve(2), getenv(3) or other places, so don't add an
automatic test for now.

make: simplify control flow in ModifyWord_SysVSubst

No functional change.

make: clean up variable names

No binary change.

make: use fixed type for comparing numbers using the modifier ':On'

When the modifier ':On' was added on 2021-07-30, there were concerns
that pre-C99 environments would not have the type 'long long', therefore
the type was made configurable, but parsing such numbers was hard-coded
to using strtoll.

To improve compatibility with C90 environments, use 'long' and 'strtol'
in these environments.  In C99 environments, use 'long long' and
'strtoll', to account for larger file sizes.

If the flexibility of choosing yet another type for these numbers should
ever arise, it can still be implemented.  Until then, reduce the number
of possible build configurations.

make: for recursive variables in commands, print location

Print the approximate location based on the last command that has been
defined for the target.  It would be possible to get more detailed
location information by counting the number of commands of the target,
but that would get messy due to .USEBEFORE, .USE and .DEFAULT, and
still, this is an edge case, so don't waste too much code for it now.
Having this hint about the location is more helpful than just a plain
"Variable X is recursive" without any further details.

make: inline functions that have a single call site

No functional change.

Allow local variable assignments in dependency lines

The variable is set in the context of the target.
This syntax has been supported by gmake for ~ever.
If necessary a makefile can set .MAKE.TARGET_LOCAL_VARIABLES=false
to disable this.

Expose GetBooleanExpr so parse.c can use it.

tests/make: demonstrate that the 'static' in Var_Parse has an effect

make: remove unsatisfiable condition in VarSubstExpr

At that point, emode can have only 2 values, none of which is
VARE_UNDEFERR.

No binary change since GCC had figured this out all the time, it just
didn't reveal this little secret.

make: when expanding nested variables, check simple things first

No functional change, just a little performance improvement for
expressions that are not evaluated anyway but only parsed.

make: remove unnecessary functions for expanding variable names

In meta mode, the affected variable patterns do not contain a '$'.

Outside of meta mode, Global_SetExpand was only called a single time, so
inline that call.

No functional change.

make: merge duplicate code for expanding variable expressions

No functional change.

make: in Cmd_Exec, return error message instead of format string

This change leaves only literal format strings in parse.c.  It allows
for more detailed error messages than the current "non-zero status" or
"exited on a signal".

No functional change.

make: shorten ApplyModifier_Assign

No functional change.

make: inline Buf_Clear

No functional change.

make: rename is_shell_metachar, fix character conversion warning

No binary change.

make: clean up comments, variable names, function names

The comment in ApplyDependencySourceOther repeated the code, its second
half didn't match any current code.

The comment above ParseDependencySourcesEmpty repeated the code.

No binary change, except for assertion line numbers.

make: merge calls to ApplyModifier_Time

This enables GCC 11 to inline ApplyModifier_Time, like all the other
modifiers.  Similar pattern as for ':M' and ':N', as well as for ':D'
and ':U'.

No functional change.

make: merge duplicate code for modifiers 'gmtime' and 'localtime'

No functional change.

make: clean up variable modifiers 'gmtime' and 'localtime'

Use consistent wording (zulu -> gmt), make VarStrftime parameter order
consistent with strftime, rename confusing 'time_t utc' to 't',
eliminate common subexpression in error message.

No functional change.

make: condense code in ApplyModifier_Assign

List the assignment operators in the same order as in the manual page.

No functional change.

make: merge duplicate types CondEvalResult and CondResult

No binary change.

make: remove redundant parameter for evaluating conditions

No functional change.

make: amend leftover cleanups from the previous commits

No functional change.

make: format comments according to /usr/share/misc/style

Assisted by indent(1), with manual corrections due to its many remaining
bugs.

No functional change.

make: remove space after ':' in bit-field declarations

As seen in /usr/share/misc/style.

make: fix memory leak when parsing a variable name

To trigger the memory leak, the expanded variable name must not be a
prefix of the textual variable name.

	textual name		expanded name
ok:	UNDEF.${undef}		UNDEF.
leak:	UNDEF.${undef}.		UNDEF..

make: reduce memory allocation in modifier ':?'

The memory allocation was in LazyBuf_DoneGet.  The substring for the
untaken branch had been allocated even though it was not necessary.

No functional change.

make: reduce memory allocations in modifier ':C'

The memory allocation was in the call to LazyBuf_DoneGet.

No functional change.

make: condense RegexReplace for the modifier ':C'

No functional change.

make: in the modifier ':C', don't call strlen unnecessarily

No functional change.

make: extract RegexReplaceBackref from RegexReplace

No functional change.

make: merge duplicate code for memory handling in Var_Parse

No functional change.

make: fix remaining memory leak for .SUFFIXES (since yesterday)

make: distinguish between short-lived and environment variables

No functional change.

make: rename VarFreeEnv to VarFreeShortLived

The newly added "variable" .SUFFIXES is short-lived as well, which makes
it necessary to distinguish between environment variables and
short-lived variables.

No binary change.

make: clean up code for freeing short-lived variables

Since yesterday's addition of the short-lived "variable" named
.SUFFIXES, not only environment variables are short-lived.  Clean up the
code to prepare for fixing the remaining memory leaks.

No functional change.

make: fix memory leak when evaluating ${.SUFFIXES} (since yesterday)

perl -e '
    printf(".SUFFIXES: %s\n", "x" x 224);
    printf("_:=\${.SUFFIXES}\n" x 100_000);
    printf("all:\n");
' \
| MALLOC_CONF=stats_print:true \
    ./make -r -f - 2>&1 \
| awk '/bins:/,/^ *256/ { print $1 "\t" $4}'

Roughly 100000 less allocations for bin size 16, for the variable name.

make: fix memory leak when evaluating ${.SUFFIX} (since yesterday)

Add .SUFFIXES as read-only variable.

References to ${.SUFFIXES} are handled dynamically in
ParseVarnameLong by calling Suff_NamesStr.

The variable cannot be set normally.

Reviewed by: rillig

ParseVarnameLong: minimize calls to LazyBuf_Get

Reviewed by: rillig

make: inline ParseEmptyArg into CondParser_FuncCallEmpty

No functional change.

make: in parse errors, mark whitespace more clearly

This prevents any trailing whitespace from going unnoticed.  It also
marks leading whitespace more clearly, as in the examples with the time
value " 1".

make: remove period from end of error messages and warnings

The majority of the existing error messages and warnings does not
include a period at the end.  Follow this style consistently.

make: merge branches for modifiers ':D' and ':U'

These branches have the exact same code, without involving any macros,
nevertheless GCC 10 doesn't merge them.  Further, GCC thinks that the
function ApplyModifier_Defined were called from two different places,
thus creating an actual function definition.  Merging the case labels
inlines the function, saving a few instructions and a few hundred bytes
in the binary.

No functional change.

make: extract common code into Expr_Words

No functional change.

make: move low-level implementation details out of Var_Parse

No functional change.

make: extract common code into Expr_Str

This change turns Expr into a more abstract data type.

No functional change.

make: fix types for modifier ':O' (since today)

In var.c 1.962 from 2021-12-05, the type of the array to be sorted
changed from 'const char *' to 'Substring'.  The code still worked since
the first member of Substring is 'const char *start', but that was pure
luck.

Fix the type assumptions in the comparison functions.

Add an assertion that the words must be null-terminated since the
comparison functions currently assume this.

This kind of inconsistency may be possible to detect using static
analysis, but NetBSD's lint is currently too limited for that.  To
detect this, it would have to analyze each possible comparison function
and remember the assumed type.  This type could then be compared to the
argument type of a function call to qsort or bsearch.

No functional change.

make: fix use-after-free in modifier ':@'

Without memory allocator debugging, the newly added test doesn't show
any obvious failure.

With memory allocator debugging enabled, all make versions since
2016.02.27.16.20.06 crash with a segmentation fault.

make: save a memory allocation in each modifier ':O' and ':u'

No functional change.

make: save a memory allocation in the modifier ':[...]'

No functional change.

make: save a memory allocation in the modifier ':[...]'

No functional change.

make: in UnexportVars, replace Str_Words with Substring_Words

This skips an allocation and a bit of memory copying.

No functional change.

make: condense ParseModifierPartSubst

No functional change.

make: extract ParseModifierPartExpr from ParseModifierPartSubst

No functional change.

make: extract ParseModifierPartDollar from ParseModifierPartSubst

No functional change.

make: fix type of variable in ApplyModifier_IfElse

No functional change.

make: use correct plural form in debug message

make: fix order of parameters in function comments

No functional change.

make: fix memory leak in error case of the ':?' modifier

make: reduce relocations, thereby reducing .text size

No functional change.

make: eliminate a few common subexpressions

This commit contains only changes that don't increase the code size on
x86_64.  Replacing the remaining instance of ModChain_ShouldEval would
either increase the code size or make the code harder to understand.

No functional change.

make: fix typos in comments

make: fix lint warnings on ilp32 platforms

The warnings in arch.c were about conversions from int to long.

The warnings in var.c were about conversions from long long to
ptrdiff_t.

No functional change.

make: remove workaround for initialization bug in lint

The bug has been fixed in init.c 1.208 from 2021-08-14.

make: remove redundant initialization in ApplyModifier_Order

No functional change.

make: fix lint warnings

The string functions from str.h are declared as 'static __unused' when
compiled with GCC, but lint explicitly undefines __GCC__ during
preprocessing.  Therefore, make those functions inline, to prevent
warnings that they are unused.

The macro UNCONST is used in a few places, and (again) since lint
undefines __GCC__, that macro expanded to a simple type cast, which lint
warned about.  To prevent this warning, implement UNCONST as a function
that works everywhere and hides the type cast.

In filemon_open, the code for closing F->in was obviously unreachable.

No functional change.

make: clean up ApplyModifier_Order

No functional change.

make: reword comment for ApplyModifier_Order

make: handle parse errors in ':O' uniformly

Previously, the error handling for the variable modifier ':O' differed
depending on the exact variant and in some cases led to misleading
or missing diagnostics.

make: merge duplicate code for sorting strings and numbers

No functional change.

make: fix typo in manual page, add more tests for the new ':On'

Add :On for numeric sort

Reviewed by: christos rillig

make: use more practical data type in RegexReplace

While size_t is most appropriate for array indexes, make needs to be
compatible with C90, which does not support the %zu printf conversion.
To avoid type casts, use a simple unsigned int here, which is more than
enough for storing a single decimal digit.

No functional change.

make: extract RegexReplace from ModifyWord_SubstRegex

No functional change.

make: move all labels in ModifyWord_SubstRegex to the same level

No functional change.

make: reduce indentation in ModifyWord_SubstRegex for ':C'

No functional change.

make: fix documentation for ParseModifierPart

Do not report unmatched regex subexpressions

It is not always an error for a subexpression to have not matched,
since the regex library can/does not convey how many matches are
expected, only report an error if opts.strict (-dL)

Reviewed by: christos

sync with head

make: uncompress code in ApplyModifier_Unique

The algorithm is easier to understand when each line of code only
focuses on a single topic.

No change to the resulting binary, except for line numbers in assertion
messages.

make: inline VarUniq into ApplyModifier_Unique

No functional change.

make: use straight quotes for error 'Bad conditional expression'

This diagnostic was supposed to be an error, see ApplyModifier_IfElse.

When such an error occurs while the makefiles are read, make stops with
an error, as can be expected.  But when such an error occurs later,
after all makefiles have been read, the message is printed but make does
not stop.

In lint mode (-dL), make stops in such a case.  I didn't dare to make
this the default behavior, out of fear of breaking existing build
infrastructure, not only in NetBSD or pkgsrc, but also FreeBSD and other
operating systems that use the bmake distribution, generated from the
same source code.

make: reduce memory allocations in the modifiers ':D' and ':U'

make: rename members of ModifyWord_LoopArgs

No functional change.

make: clean up pattern flags for the modifiers ':S' and ':C'

No special handling is necessary for C90 since none of the struct
members is a const_member.

The prefix 'Var' is not necessary since this type does not apply to a
variable but only to a modifier.

No functional change.

make: spell SysV in mixed case

No functional change.

make: reduce memory allocation and strlen calls in modifier ':from=to'

Previously, SysVMatch was quite verbose and felt like hand-optimized
assembler code, which made it difficult to discover the underlying idea
of the code.

All this code was replaced with two simple calls to Substring_HasPrefix
and Substring_HasSuffix.  Now that the operands of that modifier are no
longer passed as C strings, there is no need to collect all information
in a single scan through the word and the pattern.

It was not necessary to call Var_Subst unconditionally.  Calling it only
when the string contains a '$' saves another memory allocation and two
string copies (because of the Buf_DoneDataCompact).

No functional change.

make: in the ':Q' modifier, only allocate memory if necessary

make: improve performance for LazyBuf

The previous O(n^2) time complexity for parsing a long string with many
variable expressions was not meant to last for long.  I had hoped to fix
it within a few minutes, but that will take more time.

For now, make LazyBuf simpler by using a traditional C string for the
expected part instead of a Substring.  This avoids a strlen call per
Var_Parse.

No functional change, only performance.

make: remove redundant parameter from ParseVarnameLong

No functional change.

make: migrate ParseModifierPart to use Substring

This will reduce memory allocation for modifier parts without the escape
characters '$' or '\'.

No functional change.

make: avoid unnecessary calls to strlen when evaluating modifiers

No functional change.

make: migrate ModifyWord functions to use Substring

This benefits the modifiers ':T' and ':H' since these scan the word from
the end.  The SysV modifier '.c=.o' does not benefit yet, this will be
done in a follow-up commit.

Currently ModifyWords calls strlen for each single word, which degrades
performance.  This will be cleaned up in a follow-up commit as well.

No functional change.

make: migrate handling of the modifier ':S,from,to,' to Substring

Right now this does not gain any performance, it only makes the code in
ModifyWord_Subst a little simpler since it only uses
Buf_AddBytesBetween, not a mixture with Buf_AddBytes.

When the word passed to the ModifyWord functions is converted to
Substring as well, the call to strlen will become redundant.

No functional change.

make: clean up remaining references to VarEvalFlags

VarEvalFlags has been replaced with VarEvalMode.  There were some
comments and tests that still referred to the old names.

No functional change.

make: avoid allocating memory for simple variable names

The main change is in ParseVarname, where a Buffer is replaced with the
newly introduced LazyBuf.  LazyBuf is inspired by
https://golang.org/src/path/path.go.

In CanonicalVarname, the pre-comparison of the first letter of the
variable name is no longer necessary.  GCC 9 optimizes a fixed-length
memcmp so well that the code can finally be written to target human
readers, leaving the optimization to the compiler.

make: fix out-of-bounds read in Var_Parse_FastLane (since 30 minutes)

This bug made the test varmod-edge.mk fail sometimes with varying error
messages, as can be expected for an out-of-bounds read.

make: reword debug log message for empty variable name

The function names did not match anymore, after one of the many
refactorings in the last few months.

make: reduce debug logging and memory allocation for ${:U...}

Expressions of the form ${:U...} are often generated by .for loops.
Since these expressions are not generated knowingly by the make user, do
not fill the debug log with them since that would interrupt the normal
reading flow of the -dv log for nested expressions.

make: reduce verbosity of the -dv debug logging for standard cases

The verbosity was already removed from LogBeforeApply, now it is
consistent between LogBeforeApply and LogAfterApply.

make: in debug log, add space between scope and variable name

Without this space, the debug log looked more like line noise, even
though the only punctuation was a single innocent ':'.  From a make
user's perspective, the variable name is a word of its own and should
not be visually glued to its namespace.

make: omit unnecessary details from -dv debug log

When an expression is based on a defined variable, it does not matter
whether the evaluation mode is "eval" or "eval-defined", therefore omit
these details to reduce confusion.

make: be more verbose in -dv debug logging

The previous log output was too brief to be understandable.  Give more
hints by describing each part of the expression when evaluating a
modifier.  Distinguish between parse-only mode and eval mode since in
parse-only mode most of the details are irrelevant.

make: clean up debug logging for ':M' and ':N'

Using square brackets as quotes was confusing since patterns can contain
square brackets themselves.

The debug logging for VarMatch was a bit too detailed.  Having the
"before" and "after" states is enough for all practically relevant
cases.

make: disallow '$' in the variable name of the modifier ':@'

If this restriction should break any existing makefile, the author of
that makefile was probably heading for the IOMCC.

make: convert VarEvalFlags back into an enum, but not a bit-set

As was apparent in VarEvalFlags_ToString, a bit-set was not the best
data type since most of the flags were not freely combinable.  The two
flags that could be combined were keepDollar and keepUndef, but even
these have distinguished names in the debug log.

The downside of struct bit-fields is that they need extra helper
functions in C90 (see nonints.h).  Exchange these for a few helper
functions in var.c, to keep the code outside var.c simple.

No functional change.

make: extract decision for evaluating an expression

No functional change.

make: move VarEvalFlags_ToString down to the other functions

No functional change.

make: condense definition of VarPatternFlags

No functional change.

make: inline VarFlags into Var

No functional change.

make: replace enum bit-set with struct bit-fields

This makes the code easier to read, especially when setting one of the
flags to false.

No functional change.

make: remove unused Expr.varFlags

No functional change.

make: remove VarFlags from debug logging

Before the introduction of ExprDefined, VarFlags contained whether the
expression was defined or not, which was useful to know since the final
value of the expression depends on this information.  The other VarFlags
do not influence the evaluation, so there is no point logging them.

make: simplify access to the name of an expression during evaluation

During the evaluation, the flags of the variable don't change, except
for VFL_IN_USE, but in such a case make exits immediately anyway.

Logging the variable flags does not provide any useful information, so
it will be removed soon.  Logging the variable flags made sense when the
definedness of the expression was still encoded in the variable flags,
instead of in the separate ExprDefined.

No functional change.

make: backport to C90

In the past few months I had accidentally used C99 features in the make
code.  According to tools/README, tools that are used in the build
system should restrict themselves to C90.

This allows make to build with GCC's options "-pedantic
-Wno-system-headers -Dinline= -Wno-error=cast-qual".

I didn't notice anyone actively complaining though, I just wanted to see
how much work this backporting would be.  The identifier __func__ is
still used, as in other tools.

No functional change.

make: rename function parameters to match boolean constants

No functional change.

make: use C99 bool type instead of defining its own

No functional change.

make: remove workaround for bug in lint

The bug has been fixed in lint.

No functional change.

make: fix typos in documentation of ModChain

make: fix documentation for ModChain

The outer ModChain can be interrupted by an inner ModChain, but it
continues to exist.

No functional change.

make: rename ApplyModifiersState to ModChain

The new name accurately describes the structural element that holds such
properties as the separator character and whether the expression value
is considered a single word.  The old name ApplyModifiersState was too
long and was meant as a placeholder anyway, when I introduced it in
var.c 1.236 from 2020-07-03.

make: improve documentation of ApplyModifiersState

No functional change.

make: document an example for a 'chain of modifiers'

No functional change.

make: fix double varname expansion in the variable modifier '::='

This is an edge case that doesn't occur in practice since pretty much
nobody dares to use variable names that contain an actual '$' in their
name.  This is not about the fairly common VAR.${param} (as written in
the makefile), but instead about the variable whose name is literally
'VAR.${param}'.

The test demonstrates that after the fix, the variable name is taken
exactly as-is for the simple assignment modifier '::='.  There are no
such tests for the modifiers '::+=', '::!=' and '::?=', but that's ok.
The code in ApplyModifier_Assign would look assymetrical and suspicious
enough if one of these modifiers would expand its variable name and the
others wouldn't.

make: clean up documentation of ApplyModifiersState

No functional change.

make: change debug log for variable evaluation flags to lowercase

This makes them easier distinguishable from variable names since the
latter are usually uppercase.

No functional change outside debug mode.

make: replace enum bit-field with struct bit-field for VarEvalFlags

This makes the code easier to read, especially in var.c.  It also makes
debugging sessions easier since some debuggers don't show enum
bit-fields symbolically as soon as more than one bit is set.

The code outside var.c is basically unchanged, except that instead of
passing the individual flags, there are 4 predefined evaluation modes.
These suffice for all practical use cases.  Only in the implementation
deep inside var.c, the value of the flags keepDollar and keepUndef
differs.

There is no way of passing the struct to EnumFlags_ToString, which means
the ToString function has to be spelled out explicitly.  This allows for
fine-tuning the representation in the debug log, to reduce the amount of
uppercae letters.

No functional change.

make: rename VARE_NONE to VARE_PARSE_ONLY

The name 'NONE' described the bit pattern, which was not useful to
understand its meaning.  Omitting VARE_WANTRES only parses the
expression, without evaluating any part of it.

No functional change, not even in debug mode since Enum_FlagsToString
always returns "none" for all-bits-unset.

make: fix documentation of VarFreeEnv

No functional change.

make: skip variable lookup for '::=' modifiers in parse-only mode

This is just to keep the code consistent among the various variable
modifiers.  The performance gain is negligible.

The actual assignment to the variable had already been skipped
previously.

No functional change.

make: separate parsing and evaluating for modifier '::='

No functional change.

make: in parse-only mode, do not update the expression value in ':sh'

No functional change outside debug mode.  The other variable modifiers
behave in the same way.

make: do not evaluate the ':O' modifiers in parse-only mode

No functional change in practical usage.  Theoretically this change can
be observed by looking at the generated random numbers for the ':Ox'
modifier, but the quality or exact sequence of these random numbers is
not guaranteed anyway.

make: separate parsing and evaluating in the ':O' modifiers

No functional change.

make: do not evaluate modifier ':[...]' in parse-only mode

In parse-only mode, variable expressions in the argument to that
modifier are not resolved.  This led to the error message about the 'Bad
modifier' in var-eval-short.mk.

make: do not evaluate modifiers ':M' and ':N' in parse-only mode

No functional change outside debug mode (-dv for ModifyWord_Match).

make: do not return unevaluated 'else' part from the ':?' modifier

No functional change outside debug mode.

make: do not evaluate ':gmtime' and ':localtime' in parse-only mode

No functional change.

make: don't evaluate several simple modifiers in parse-only mode

This affects the modifiers ':E', ':H', ':P', ':Q', ':R', ':T', ':hash',
':q', ':range', ':tl', ':ts', ':tu', and ':u'.  All these modifiers are
side-effect free.

Skipping the evaluation for these modifiers is purely for code
consistency and performance.

No functional change.

make: skip strdup when parsing an irrelevant ':L' modifier

No functional change.

make: skip memcpy when parsing but not evaluating ':D' and ':U'

No functional change, just a tiny bit of performance improvement,
probably not even measurable.  Having the code nevertheless serves as a
copy-and-paste template for implementing other modifiers that might
perform more costly tasks.

make: only evaluate the ':@' modifier if the result is actually used

The test 'var-eval-short' had produced the output 'unexpected' before,
on stderr.  It had been generated by '${:Uword:@${FAIL}@expr@}' by
combining the following obscure "features" of make:

1.  the ':@' modifier loops over the words of the variable.  This
    modifier is not really obscure, it still takes some time to get used
    to it.

2.  the ':@' modifier allows a '$' sign in the variable name, which is
    useless in practice.

3.  the ':@' modifier creates a temporary loop variable in the global
    namespace.  Luckily there are only few collisions with other
    variable names since their naming conventions differ.

4.  after looping over the words of the expression, the temporary global
    loop variable is deleted, and at that point the '$' is expanded,
    being interpreted as the start of a variable expression.

5.  The ':@' modifier deleted the global variable even when it was
    called in parse-only mode (without VARE_WANTRES).

When the modifier ':@' was initially added to make in var.c 1.40 from
2000-04-29, Var_Delete didn't expand the variable name.  That feature
was added in var.c 1.174 from 2013-05-18, probably without thinking of
this very edge-casey combination of features.

This commit fixes item 5 from the above list.  The other obscurities
remain for now.

make: eliminate common subexpression in ApplyModifier_Remember

No functional change.

make: merge duplicate code in ApplyModifier_Remember

This way, parsing and evaluating the modifier is only written once in
the code.  The downside is that the variable name is allocated even if
VARE_WANTRES is not set, but since this modifier is so obscure and
seldom used this doesn't matter in practice.

make: do not expand the variable name in the ':_' modifier

This edge case had been so obscure that even discovering this takes
quite some time and requires reading the source code of make.

The manual page doesn't document whether the variable name is expanded
or not, it doesn't even give an example.  When this obscure modifier was
initially added in var.c 1.210 from 2017-01-30, Var_Set always expanded
the variable name once, and there was no way around it.  Therefore this
expansion has probably been unintentional.

make: only evaluate the ':_' modifier if the expression is needed

See var-eval-short.mk:46 for the test demonstrating this change.
Previously, the expression ${:Uword:_=VAR} was evaluated including all
its side effects even though it was in an irrelevant branch of the
condition.

make: return failure in TryParseIntBase0 for empty string

No functional change since the only caller of TryParseIntBase0 already
handles all possible parse errors.  Without this check, the code just
looked wrong though.

make: separate parsing from evaluating for several modifiers

This aligns the implementation of these modifiers with the requirements
in the long comment starting with 'The ApplyModifier functions'.

No functional change.

make: reduce indentation in ApplyModifier_SunShell

No functional change.

make: separate parsing and evaluating for modifier ':u'

No functional change.

make: separate parsing and evaluating for modifiers ':Q' and ':q'

No functional change.

make: extract duplicate code for parsing ':S' and ':C'

No functional change.

make: extract duplicate code for detecting a modifier's end

No functional change.

make: document how error handling should be done correctly

Right now, when a variable expression cannot be parsed, the result of
calling Var_Subst is a string containing garbage, and no error is
reported.  In addition, there are some silent errors that are not
reported at all.  This combination makes it difficult to change the
error handling without introducing subtle breakage in some edge cases.

An example for garbage output is in varmod-subst-regex.mk, in target
mod-regex-compile-error.

No functional change.

make: sort modifiers in ApplyModifier alphabetically

No functional change.

make: fix wrong expression evaluation in -dL mode

The modifier ':C' now only compiles the regular expression if the result
of the expression is actually needed.

Several other modifiers have the same bug of evaluating the expression
in cases where this is not needed.  It just doesn't show up because they
don't have any noticeable side effects, other than wasting CPU time.
This affects irrelevant conditions as well.

make: reduce indentation in Var_Parse

The extra condition had been necessary before FStr made memory
management simpler.

The Coverity annotation got out-of-date when the parameter was converted
to FStr since that type is not allocated on the heap, only its inner
members are.

No functional change.

make: restructure code in ParseVarname to target human readers

Breaking the loop once for depth == 0 and once for depth == 1 was
unnecessarily confusing, as was the nested 'if'.  Start counting with 0
since there is no reason to start at 1.

Evaluating the common subexpression '*p == endc' is left as an exercise
to the compiler.

No functional change.

make: improve error message for bad modifier in variable expression

The improvement is especially noteable for variable expressions based on
the empty variable, see moderrs.exp:103.

make: improve error message for unclosed modifier

Replace "variable specification" with the more modern "variable
expression", reduce the number of parentheses, output more than a single
character for modifiers, make it obvious that in expressions such as
${:Serror}, the "" means a variable name.

make: improve error message for unknown modifier

Back in 1995, the modifiers were all single-character, and it made sense
to print only the first character.  Nowadays, with ':S', ':@var@...@',
'::=' and several others, a little more context is useful to see where
the exact error is.  The actual modifier is still guessed, and the guess
may be wrong as soon as backslashes get involved, but it is still better
than before.

make: remove redundant parameter of ApplySingleModifier

make: explain non-obvious code around indirect variable modifiers

No functional change.

make: comment on possible inconsistency in handling modifier ':sh'

make: note inconsistent parsing behavior in modifier ':_='

make: quote ':S' in error message about missing delimiter

make: extract ParseModifier_Match into separate function

No functional change.

make: add context information to error message about ':range' modifier

make: fix local variable name for parsing arguments

The variable name 'arg' was misleading since after a successful
TryParseTime, it would no longer point to the argument of the variable
modifier, but to the _end_ of the argument.  To reduce confusion, use p
instead, like everywhere else.  This name is less specific, which is
still better than a wrong name.

make: add quotes around variable name in an error message

make: reorder code in ModifyWords

No functional change.

make: use more common parameter order for VarSelectWords

No functional change.

make: make ModifyWord_Subst a little easier to understand

Addition is easier than subtraction, and the expression 'word + wordLen'
obviously means 'the end of the word', which was not as easy to spot
before.

No functional change.

make: remove freestanding freeIt variables

These variables all belong to a string variable.  Connect them using
FStr, which reduces the number of variables to keep track of.

No functional change.

make: remove boolean parameter and return type from VarFreeEnv

There was only a single case where this parameter was false.  Inline
that case.  That was the only case that needed the return value, so
remove that as well.

make: add a few open questions to Var_SetWithFlags

make: do not expand variable name from the command line twice in a row

When

1.  there is a global variable containing a dollar in its expanded name
    (very unlikely since there are lots of undocumented edge cases that
    make variable names containing dollar signs fragile), and

2.  after that (unlikely since that requires .MAKEFLAGS instead of a
    normal command line)

3.  there is a command line variable of the same name (again very
    unlikely since that variable name would contain a dollar sign as
    well in the expanded form),

the global variable would not be undefined as promised by the comments
since its name was expanded once more than intended.

Because of the two 'very unlikely' above, this edge case hopefully does
not affect any practical use cases.

Note that this is not about VAR.${param} (which has a dollar sign in its
unexpanded form), but about the case where param itself would expand to
a dollar sign, such as after param=$$.

make: extract ExistsInCmdline from Var_SetWithFlags

No functional change.

make: save a hash map lookup when defining a cmdline variable

This is a preparation to extract the code for exporting a cmdline
variable.  That code differs in several details from the other code in
ExportVar.

No functional change.

make: document interaction between cmdline and global variables

Make prevents global variables from being or becoming visible when a
command line variable of the same name is already defined.

There is a double safety net here.  Even if the call to Var_DeleteExpand
were removed, there would be no noticeable effect, other than one less
line in the debug log.

No functional change.

make: clean up VarAdd, Var_Delete, Var_ReexportVars

No functional change.

make: clean up and update comments in var.c

During the refactorings of the last months, several comments have become
outdated, some are now redundant since the code is as clear as the
comment, and some code benefits from a bit of explanation.

make: use bit-shift expressions for VarFlags constants

These are easier to read than hex constants.

There was no need to skip bits 2 and 3 (there were no constants for 0x04
and 0x08).  Close this gap, to avoid confusing future readers.  Keep the
relative order of the flags since that affects the debug output of -dv.

No functional change.

make: rename constants for VarFlags

The old prefix was "VAR_" and this prefix is used for several other
constants as well, which made it ambiguous.

make: sync comment about duplicated code with reality

There are only very few places in var.c that contain really duplicate
code anymore.

There is still lots of _almost_ duplicate, for example the code for
parsing variable modifiers.  It differs subtly in behavior:

*   The modifiers ':M' and ':N' use '$$' to escape a '$' sign, while
    almost all other modifiers use '\$' for this purpose.

*   The modifiers ':M', ':N', ':S', ':@' and several others parse
    balanced parentheses and braces, allowing '(' to '}' to match.
    The modifiers ':D' and ':U' only treat the end character special but
    not the other 3 of '(){}'.

*   When parsing the modifier ':S' but not evaluating it, the code for
    nested variable expressions is parsed differently from when it is in
    evaluation mode (VARE_WANTRES).  This applies to an outer ':S'
    modifier and an inner ':D' or ':M' modifier.

Since these inconsistencies affect the behavior in edge cases and some
users of make might depend on it, they cannot be fixed by
behavior-preserving refactorings.

make: rename ExprDefined constants for debug logging

make: rename ExprStatus to ExprDefined

The type describes the definedness of an expression, not a general
status, therefore the new name is more precise.

The constants are renamed as well since their prefix 'VES' does not
match the type name anymore, it was correct 3 days ago when the type was
still named VarExprStatus.  The name VES_NONE was misleading since
'none' does not describe its actual effect.  That name came from the
time when the status was a bit set, and 'none' simply meant 'none of the
bits are set'.

The names used in debug logging will be renamed in a follow-up commit,
to demonstrate that the changes in this commit indeed have no functional
change, especially not the change from '!=' to '==' in line 4304.

No functional change.

make: update comments for Expr, amending the previous commit

make: split parameters for evaluating variable expressions

The details of how variable expressions are evaluated is controlled by
several parameters: startc and endc differ for $(VAR) and ${VAR}, the
value of the expression can be interpreted as a single big word, and
when joining several words (such as with ':M' or ':S'), there may be a
custom word separator (defined with ':ts*').

The scope of half of these parameters is the whole variable expression,
the other half of the parameters are reset after each chain of indirect
modifiers.  To make this distinction obvious in the code, extract Expr
from ApplyModifiersState.  Previously, these details were hidden in how
parameters are passed and restored among ApplyModifiersIndirect and
ApplyModifiers.

The changes in the individual ApplyModifier functions are numerous but
straight-forward.  They mostly replace 'st' with 'expr'.

The changes in ApplyModifiers and ApplyModifiersIndirect are more
subtle.  The value of the expression is no longer passed around but is
stored in a fixed location, in Expr, which makes it easier to reason
about memory management.

The code in ApplyModifiers after 'cleanup' looks quite different but
preserves the existing behavior.  Expr_SetValueRefer is nothing else
than the combination of FStr_Done followed by FStr_InitRefer.  Storing
exprStatus back at the end was responsible for passing the definedness
of the expression after applying the indirect modifiers back to the
outer ApplyModifiersState.  The same effect is now achieved by having
Expr.status with a wider scope.

No functional change.

make: clean up code and comments around ModifyWord

In ModifyWords, there is no "passed string" anymore since that function
now directly operates on the expression value.

While here, improve the documentation of ModifyWordsCallback and rename
it to ModifyWordProc, focusing on its purpose instead of where it is
used.

make: clean up memory management in evaluation of expressions

The condition "st->newValue.str != val" in ApplySingleModifier made the
memory management look more complicated than it really was.  Freeing an
object based on another object's value is harder to understand than
necessary.

To fix this, the "current value" of the expression is now stored in
ApplyModifiersState, and it gets updated in-place by the ApplyModifier
functions.  This reduces the number of parameters for the ApplyModifier
functions.

Accessing the current value of the expression is now more verbose than
before (st->value.str instead of the simple val).  To compensate for
this verbosity, ApplyModifiersIndirect is now much easier to understand
since there is no extra "current value" floating around.

There is still room for improvement.  In ApplyModifiers, passing an FStr
in and returning another (or possibly the same) makes it difficult to
understand memory management.  Adding a separate Expr type that outlives
the ApplyModifiersState will make this easier, in a follow-up commit.

make: reduce redundant code around ModifyWords

The result of ModifyWords had been passed to Expr_SetValueOwn in all
cases.  The last argument to ModifyWords had always been st->sep.

make: print error about failed shell command before overwriting variable

Memory management of the value of variable expressions is currently more
complicated than necessary.  It is the responsibility of ApplyModifiers,
even though conceptually the value belongs to an expression, so it
should rather be in Expr.  Right now, this is an alias for
ApplyModifiersState, but that will change soon.

When that is done, there will no longer be a "current value" and a "new
value", only a single "value" of an expression.  At that point, before
Expr_SetValueOwn will overwrite the old value with the output of the
shell command, the error message needs to refer to the latter.

make: clean up ValidShortVarname

The switch statement was hard to read, especially the "break" that
needed a comment since it was effectively a "continue".

make: clean up ParseVarnameShort

Single-character short variable expressions such as $V neither have a
starting character nor an ending character.  The only interesting
character forms the complete variable name.

No functional change.

make: rename VarExprStatus to ExprStatus

It is only used in var.c so there is no need to use a prefix.

make: revert part of previous commit

That code was not ready yet.

make: update line numbers in expected test output

The documentation from the previous commit added a few lines.

make: add functions for assigning the value of an expression

The plan is to have only the "current value" of the expression as a
member, not the "new value".  To do this consistently and get the memory
management right, there must be a single place (or two) where the value
of the expression is updated.

No functional change.

make: rename ApplyModifiersState_Define to Expr_Define

The type name ApplyModifiersState was only intended as a working draft,
its name is too long and its scope a little too narrow.

Applying the modifiers is the main part of evaluating a variable
expression, and the scope of that type will be extended to parsing the
name of the expression as well.  This will hopefully reduce the number
of parameters, which is currently at 14.

No functional change.

make: condense the code for parsing :S and :C modifiers

No functional change.

make: rearrange some comments to make them easier to spot

make: document purpose of stepping back in the parser

In ApplyModifier_Assign there was no need to compute the delimiter from
st->startc since that has already be done at that point.

make: sync comment for ApplyModifier_Assign with reality

In compat mode, empty shell commands are handled correctly since at
least 1993.

In jobs mode, empty shell commands are handled correctly since at least
job.c 1.93 from 2005-06-16.

The only place where empty shell commands led to problems was in
Cmd_Exec, but that has nothing to do with the example from the comment.
See var-op-shell.mk for more details.

make: improve documentation about variable scopes

In an experiment, I tried to separate the concepts of a GNode and a
variable scope.  The global variables SCOPE_GLOBAL, SCOPE_INTERNAL and
SCOPE_CMDLINE are implemented as GNode even though they only need the
members 'name' and 'vars'.  All their other members are unused.
Therefore it seemed natural to extract this part of the GNode into a
separate type called Scope.

The resulting code was harder to read though since it had split the
namespace of the functions into several parts that were not obviously
related: The Var_ functions, the Scope_ functions, and the short-cut
Global_ functions.  Because of this, I threw away the experiment.

All that is left are a few updated comments.

make: rename remaining ctx to scope

One less small inconsistency.  Everywhere else, that variable had been
named ctxt, not ctx.

make: in the Var_ functions, move the scope to the front

This change provides for a more natural reading order in the code.
Placing the scope first makes it immediately clear in which context the
remaining parameters are interpreted.

No functional change.

make: add shortcut Global_Delete for deleting a global variable

make: rename Var_ValueDirect to GNode_ValueDirect

make: rename context and ctxt to scope

This continues the previous commit, in which VAR_GLOBAL, VAR_INTERNAL
and VAR_CMDLINE were renamed.

Renaming the variable 'ctxt' was trivial since that word is used nowhere
else.  In the comments though, each occurrence of the word 'context' had
to be checked individually since the word 'context' was not only used
for referring to a variable scope.  It is also used to distinguish
different situations where characters are escaped in a certain way
('parsing context') and in a few other expressions.

make: rename some VAR constants to SCOPE

The word "context" does not fit perfectly to the variables that are
associate with a GNode, as the context is usually something from the
outside and the variables are more like properties inherent to the
GNode.

The term "global context" fits even less.  Since the thing where
variables are looked up is commonly named a scope, use that term
instead.

This commit only renames the global variables VAR_GLOBAL, VAR_INTERNAL
and VAR_CMDLINE, plus a few very closely related comments.  These are:

	GNode.vars (because of line breaks)
	GNode_Free (dito)
	varname-make_print_var_on_error.mk
	varname-make_print_var_on_error-jobs.mk

The debug message in Var_Stats is left as-is since there is no unit test
for it yet.

The other renamings (variable names "context", "ctxt", as well as
further comments) will be done in a follow-up commit.

make: merge SetVar into Var_SetWithFlags

No functional change.

make: rename Var_SetWithFlags to Var_SetExpandWithFlags

Add back Var_SetWithFlags for the one call that doesn't need to expand
the name.

Now one of the flags is encoded in the function name while the others
are encoded in VarSetFlags.  This is inconsistent.  Maybe there is a
better way to model the different variants of setting a variable.

make: rename Var_Set to Var_SetExpand

After doing the textual renaming across all files, I added a new
function Var_Set that does not expand the variable name.  I then undid
the renaming for all calls where the variable name cannot ever contain a
dollar sign.  I omitted the word "Expand" from the textual references in
the unit tests and in the debug logging messages since the focus is
usually on the "Set" part, not on the "Expand".

No functional change.

make: rename Var_Delete to Var_DeleteExpand, Var_DeleteVar to Var_Delete

The function names now follow the naming scheme from the other functions
that handle variables.

There are several calls that remain syntactically unchanged but that
omit the call to strchr('$') now.  Since all these calls use constant
variable names, there is no functional change.

make: split Var_Exists into plain Var_Exists and Var_ExistsExpand

Most previous calls to Var_Exists use constant variable names.  Only the
two calls in parse.c need to expand the variable name.

It may be a good idea to expand the variable name once in VarAssign_Eval
instead of repeating the expansion in each of its special cases.

No functional change.

make: replace Global_AppendExpand with Global_Append

All callers with a variable name that is guaranteed to not contain a
dollar sign have been converted to call Global_Append instead of the
previous Global_AppendExpand.  After that, Global_AppendExpand was
unused, therefore it was effectively just renamed.

make: split Var_Append into Var_Append and Var_AppendExpand

The plain Var_Append now does not expand the variable name anymore.  It
is used in situations where the variable name is known to not contain a
dollar sign.

This is a preparation for adding Global_Append, corresponding to
Global_AppendExpand.

make: fix double expansion when appending to a new variable

make: replace Global_SetExpand with Global_Set for constant names

make: use shortcut functions Global_SetExpand and Global_AppendExpand

There are many places where global variables are set or appended to.  To
reduce clutter and code size, encode the VAR_GLOBAL in the function
name.

The word Expand in the function names says that the variable name is
expanded.  In most of the cases, this is not necessary, but there are no
corresponding functions Global_Set or Global_Append yet.

Encoding the information whether the name is expanded or not in the
function name will make inconsistencies obvious in future manual code
reviews. Letting the compiler check this by using different types for
unexpanded and expanded variable names is probably not worth the effort.
There are still a few bugs to be fixed, such as in SetVar, which expands
the variable name twice in a row.

make: remove unused INTERNAL flag

It had been used for cached_realpaths, until this variable had its type
changed from GNode to HashTable in main.c 1.469 from 2020-11-14.

make: replace VarExprFlags with VarExprStatus

The combination of !VEF_UNDEF && VEF_DEF was not possible, which made it
rather strange to model this state as a bit set.

The only functional change is the renamed constants in the debug output.

Using ENUM_VALUE_RTTI_2 felt like overengineering since it's harder to
understand than a simple array of names.

make: replace enum bit-set with bit-field

No functional change.

The generated code from GCC 5.5 is very similar.  On x86_64, memory
access is no longer in 32-bit units but in 8-bit units since only the
first few bits are actually used.  The bit patterns are the same as
before, so if there is any difference in performance, GCC should have
chosen the more efficient variant all along.

In a previous experiment, the code size increased a lot, surprisingly.

make: replace pre-increment with post-increment or simple addition

The rest of the code already prefers post-increment if there is no
actual difference.

make(1): inline Buf_Len

make(1): inline Buf_GetAll

make(1): split Buf_Destroy into Buf_Done and Buf_DoneData

In all cases except one, the boolean argument to Buf_Destroy was
constant.  Removing that argument by splitting the function into two
separate functions makes the intention clearer on the call site.  It
also removes the possibility for using the return value of Buf_Done,
which would have made no sense.

The function Buf_Done now pairs with Buf_Init, just as in HashTable and
Lst.

Even though Buf_Done is essentially a no-op, it is kept as a function,
both for symmetry with Buf_Init and for clearing the Buffer members
after use (this will be done only in CLEANUP mode, in a follow-up
commit).

make(1): reduce boilerplate for printing bit sets in debug mode

No functional change.

make(1): fix a few inconsistencies for lint's strict bool mode

make(1): make a few more bool expressions more precise

The previous version of lint(1) from a few hours ago didn't catch all
occurrences.  And even the current one doesn't catch everything.
Function arguments and return types still need some work.  The "return
quietly" from shouldDieQuietly still implicitly converts from int to
_Bool.

No functional change.

make(1): consistently use boolean expressions in conditions

Most of the make code already followed the style of explicitly writing
(ptr != NULL) instead of the shorter (ptr) in conditions.

The remaining 50 instances have been found by an experimental,
unpublished check in lint(1) that treats bool expressions as
incompatible to any other scalar type, just as in Java, C#, Pascal and
several other languages.

The only unsafe operation on Boolean that is left over is (flags &
FLAG), for an enum implementing a bit set.  If Boolean is an ordinary
integer type (the default), some high bits may get lost.  But if Boolean
is the same as _Bool (by compiling with -DUSE_C99_BOOLEAN), C99 6.3.1.2
defines that a conversion from any scalar to the type _Bool acts as a
comparison to 0, which cannot lose any bits.

make(1): fix lint warnings

make(1): format multi-line comments

make(1): clean up code in extracted ExportVar functions

make(1): split ExportVar into separate functions

make(1): rename local variables in Var_ReexportVars

make(1): replace global preserveUndefined with VARE_KEEP_UNDEF

Controlling the expansion of variable expressions using a global
variable and a VARE flag was inconsistent.

Converting the global variable into a flag had to prerequisites:

1.  The unintended duplicate variable assignment had to be fixed, as
done in parse.c 1.520 from 2020-12-27.  Without this fix, it would have
been necessary to add more flags to Var_Exists and Var_SetWithFlags, and
this would have become too complex.

2.  There had to be a unit test demonstrating that VARE_KEEP_DOLLAR only
applies to the top-level expression and is not passed to the
subexpressions, while VARE_KEEP_UNDEF applies to all subexpressions as
well.  This test is in var-op-expand.mk 1.10 from 2020-12-28, at least
for the ':@word@' modifier.  In ParseModifierPartSubst, VARE_KEEP_UNDEF
is not passed down either, in the same way.

make(1): remove dead code from ApplyModifiersIndirect

At that point, the expression can never be varUndefined.  At the
beginning of ParseVarnameLong, the expression is initialized to a simple
empty string, and that string is only ever converted to varUndefined at
the very end of Var_Parse.

make(1): remove outdated comment about string comparisons

Back in 1993, the variables in a context were stored in a linked list.
Searching such a list indeed required literally thousands of calls to
strcmp.  In make.h 1.22 from 1999-09-15, the linked list was replaced
with a hash table, requiring much fewer string comparisons.  Since then,
the rationale doesn't apply anymore.

make(1): remove unnecessary VPR_ERR_SILENT

make(1): do not inspect output variables in ParseVarnameShort

make(1): move error handling code out of UndefinedShortVarValue

make(1): split Var_Subst into easily understandable functions

Extracting the character-level details makes the essence of Var_Subst
visible in the code, which is to iterate over the given text, handling a
few types of tokens.

make(1): clean up VarParseResult constants

The many constants were invented because at that time I didn't quite
understand the actual outcomes of Var_Parse that need to be
distinguished.  There are only a few:

(1) Errors, whether they are parse errors, or evaluation errors or
    undefined variables.  The old constants VPR_PARSE_MSG and
    VPR_UNDEF_MSG are merged into VPR_ERR.

(2) Undefined expressions in a situation in which they are allowed.
    Previously the documentation for VPR_UNDEF_SILENT talked about
    undefined expressions in situations where they were not allowed.
    That case is fully covered by VPR_ERR instead.

(3) Errors that are silently ignored.  These are probably bugs.

(4) Everything went fine, the expression has a defined value.

make(1): remove unnecessary VPR_UNKNOWN for error handling

There is no sensible way for a caller of Var_Parse to deal with an error
state of "maybe successful, maybe not", therefore remove the constant
for it.

make(1): align names of VarExportMode with the directives

make(1): rename CmdOpts.lint to strict

When running lint(1) on the code, it defines the preprocessor macro
"lint" to 1, which generated a syntax error in the declaration "Boolean
lint", as that became "Boolean 1".

make(1): fix memory leak in Var_Undef (since 2020-12-22)

make(1): allow .undef to undefine multiple variables at once

Since make doesn't support variable names containing spaces, this edge
case is not enough reason to stop this feature.  Having multiple
variable names as arguments nicely aligns with other directives such as
.for and .export.

make(1): remove excess newline from parse errors (since 2020-11-01)

For the modifiers :gmtime and :localtime, the excess newline had been
added in var.c 1.631 from 2020-10-31 21:40:20.

For the modifiers :range and :ts, the excess newline had been added in
var.c 1.635 from 2020-11-01 14:36:25.

make(1): remove redundant parameters from ParseModifierPart

make(1): save a few memory allocations in variable expressions

make(1): switch memory handling from MFStr to FStr

This makes all intermediate strings constant.  For this simple
search-and-replace refactoring, all intermediate locations where the
"current value of the expression" was stored had to be of the type
MFStr.

Using FStr instead of MFStr allows to save a few memory allocations,
which will be done in the follow-up commits.

make(1): clean up memory management for expanding variable expressions

Previously, memory management had been split among several variables.
The general idea was very simple though.  The current value of the
expression needs to be kept in memory, and each modifier either keeps
that value or replaces it with its own newly allocated result, or
var_Error or varUndefined.

Using MFStr, it does not matter anymore that var_Error and varUndefined
are statically allocated since these are assigned using MFStr_InitRefer.

The complexity of the implementation is now closer to the actual
complexity.  Most probably the code can be simplified even more.

make(1): use FStr for ApplyModifiersState.newVal

Memory management is still complicated in this area.  To clean this up,
the previous value of the expression needs to be converted to an MFStr
first, and later to an FStr.

make(1): remove redundant assignment in ApplyModifier_SysV

make(1): error out on unknown variable modifiers at parse time

Before, make printed an "error message" that did not include the word
error and thus was not easily identified as such.  This "error message"
also did not influence the exit status in the default mode but only in
-dL mode.  The error message also didn't include any line number
information and was thus rude.

make(1): remove wrong error message for indirect modifier in lint mode

make(1): document that indirect modifiers fall back to SysV modifiers

make(1): remove redundant const declarations for parameters

make(1): extract ApplySingleModifier from ApplyModifiers

make(1): turn ApplyModifiersState.val into a local variable

This reduces the scope and makes it more obvious at which places this
variable can be changed and how the memory management is done.

make(1): use FStr for memory management in Var_SetWithFlags

make(1): extract SetVar from Var_SetWithFlags

SetVar contains the part that is not concerned about memory management
and expansion of the variable name.

make(1): fix memory leak in Var_Delete (since yesterday)

The memory leak had been introduced in var.c 1.736 from 2020-12-19
20:47:24.

make(1): change return type of Var_Value to FStr

make(1): use FStr in VarNew

make(1): return FStr from Var_Parse

This reduces the number of variable declarations at the call sites.

make(1): export FStr and MFStr

These types are a replacement for the pattern "var + var_freeIt" that
enforces these two variables to be updated together.

make(1): document memory handling in ApplyModifiers

make(1): replace freeIt with FStr in EvalUndefined

Previously, the parameter out_freeIt was not guaranteed to be
initialized in every case, at least when looking only at EvalUndefined.
This contradicted the variable name.

Replace the two parameters with a single FStr to ensure that these
variables are always initialized together.

make(1): replace FStr_Assign with separate initialization functions

In GetVarnamesToUnexport, there is no need to free the local FStr since
the only place where it is assigned an allocated string is at the very
end.

Having separate functions for the two main use cases of a possibly
allocated string makes the calling code simpler.  This is a preparatory
commit for making the memory allocation in ApplyModifiers easier to
understand.

make(1): extract string functions from ApplyModifier_To

make(1): error out if .undef has not exactly 1 argument

make(1): extract Var_DeleteVar from Var_Delete

make(1): extract Var_Undef from ParseDirective

make(1): replace %zu with %u in printf calls

This is needed to compile bmake with GCC 2.8.1 on SunOS 5.9.

To support ancient systems like this, the whole code of usr.bin/make is
supposed to use only ISO C90 features, except for filemon, which is not
used on these systems.

make(1): add str_basename to reduce duplicate code

The function basename from POSIX has a few unfortunate properties, it is
allowed to return a pointer to static memory.  This is too unreliable,
therefore this trivial own implementation.

make(1): remove dead code from GetVarnamesToUnexport

Now that the parsing of the directives is unified and strict, there is
no need anymore for the dispatched functions to check for unknown
directives.  These functions don't even get the information to decide
that since this decision is already done.

make(1): clean up Var_Export