CVS log for src/usr.bin/make/unit-tests/varmod-loop.mk

Up to [cvs.NetBSD.org] / src / usr.bin / make / unit-tests

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

make: sync VarEvalMode constant names with their debug log names

make: remove unused VARE_EVAL_KEEP_DOLLAR

tests/make: replace 'variable expression' with 'expression'

Each expression is based on a variable, there's no need for the
verbosity.  The wording in make's diagnostics will be changed in a
follow-up commit.

tests/make: document the modifier ':@var@body@'

make: fix parsing of unevaluated subexpressions with unbalanced '{}'

Since var.c 1.323 from 2020-07-26, modifiers containing unbalanced
braces or parentheses were parsed differently, depending on whether they
were relevant or not.

For example, the expression '${VAR:...}' is enclosed with braces. When
this expression has a modifier ':S,},}},g' that would double each '}' in
that expression, the parser got confused:

If the expression was relevant, the modifier was parsed as usual, taking
into account that the 3 '}' in the modifier are ordinary characters.

If the expression was irrelevant, the parser only counted the '{' and
the '}', without taking into account that a '}' might be escaped by a
'\' or be an ordinary character.  Parsing therefore stopped at the first
'}', assuming it would finish the expression '${VAR:S,}'.

This parsing mode of only counting balanced '{' and '}' makes sense for
the modifier ':@var@...@', which expands each word of the expression
using the template from the '...'.  These templates tend to be simple
enough that counting the '{' and '}' suffices.

tests/make: document how the modifier ':@' works

make: revert parsing of modifier parts (since 2022-08-08)

The modifier ':@var@body@' parses the body in parse-only mode and later
uses Var_Subst on it, in which each literal '$' must be written as '$$'.

Trying to parse the loop body using Var_Parse treated the text
'$${var:-0}' as a single '$' followed by the expression '${var:-0}',
wrongly complaining about the 'Unknown modifier "-0"'.

Found by sjg.

tests/make: demonstrate parse error in ':@' modifier (since 2022-08-08)

Reported by sjg via private mail.

make: fix use-after-free in modifier ':@'

Without memory allocator debugging, the newly added test doesn't show
any obvious failure.

With memory allocator debugging enabled, all make versions since
2016.02.27.16.20.06 crash with a segmentation fault.

tests/make: demonstrate use-after-free in modifier ':@'

tests/make: test and document .MAKE.SAVE_DOLLARS and ':='

make: clean up remaining references to VarEvalFlags

VarEvalFlags has been replaced with VarEvalMode.  There were some
comments and tests that still referred to the old names.

No functional change.

tests/make: split test for modifier ':@' into separate files

The file varmod-loop.mk has grown too large to be single-purpose, plus
it combined parse-time and run-time tests.  This has the downside that
as soon as a parse-time test results in an error, the run-time tests are
not run anymore.

tests/make: convert varmod-loop to parse-time

In case of unexpected failures, this provides the line number of the
'.error' directive.

tests/make: add position marker in test 'varmod-loop'

make: replace enum bit-field with struct bit-field for VarEvalFlags

This makes the code easier to read, especially in var.c.  It also makes
debugging sessions easier since some debuggers don't show enum
bit-fields symbolically as soon as more than one bit is set.

The code outside var.c is basically unchanged, except that instead of
passing the individual flags, there are 4 predefined evaluation modes.
These suffice for all practical use cases.  Only in the implementation
deep inside var.c, the value of the flags keepDollar and keepUndef
differs.

There is no way of passing the struct to EnumFlags_ToString, which means
the ToString function has to be spelled out explicitly.  This allows for
fine-tuning the representation in the debug log, to reduce the amount of
uppercae letters.

No functional change.

make: demonstrate how to undefine variables during evaluation

For a very long time now, I had thought that it would be impossible to
undefine global variables during the evaluation of variable expressions.
This is something that the memory management in Var_Parse relies upon,
see the comment 'the value of the variable must not change'.

After several unsuccessful attempts at referring to an already freed
previous value of a variable, today I discovered how to unset a global
variable while evaluating an expression, which has the same effect.  To
demonstrate that this use-after-free can reliably crash make, it would
need a memory allocator with a debug mode that never re-allocates the
same memory block after it has been used once.  This is something that
jemalloc cannot do at the moment.  Valgrind would be another idea, but
that has not been ported to NetBSD.

Undefining a global variable while evaluating an expression is made
possible by an implementation detail of the modifier ':@'.  That
modifier undefines the loop variable, without restoring its previous
value, see ApplyModifier_Loop.

By the very old conventions of ODE Make, these loop variables are named
'.V.' and thus do not conflict with variables from other naming
conventions.  In NetBSD and pkgsrc, these loop variables are typically
called 'var', sometimes '_var' with a leading underscore, which also
doesn't conflict with the typical form 'VAR' of variables in the global
namespace.  Therefore, in practice these loop variables don't interfere
with other variables.

One case that can practically arise is when an outer variable has a
modifier ':@word@${VAR.${word}}@' and one of the referenced variables
uses the same variable name in the modifier, see varmod-loop.mk 1.10
line 91 for a detailed explanation.

By using the ${:@VAR@@} modifier in a place that is evaluated with
cmdline scope, it is not only possible to undefine global variables, it
is possible to undefine cmdline variables as well.  When evaluated in a
specific make target, the expression ${:@\@@@} can even be used to
undefine the variable '.TARGET', which will probably crash make with an
assertion failure.

make: rename context and ctxt to scope

This continues the previous commit, in which VAR_GLOBAL, VAR_INTERNAL
and VAR_CMDLINE were renamed.

Renaming the variable 'ctxt' was trivial since that word is used nowhere
else.  In the comments though, each occurrence of the word 'context' had
to be checked individually since the word 'context' was not only used
for referring to a variable scope.  It is also used to distinguish
different situations where characters are escaped in a certain way
('parsing context') and in a few other expressions.

make(1): fix tests varmod-defined and varmod-loop regarding dollars

Some derived versions of NetBSD's make set .MAKE.SAVE_DOLLARS to no.  In
these versions, running the tests would fail.  Therefore better set
.MAKE.SAVE_DOLLARS to yes explicitly as far as necessary.

Suggested by sjg.

make(1): add test demonstrating how ':=' and ':@var@' interact

make(1): in tests, replace "dollar character" with "dollar sign"

make(1): add test for combining the :@ and :? variable modifiers

make(1): add test for nested :@ modifiers using the same variable name

make(1): add test for empty variable name in :@var@...@ modifier

make(1): move tests for :S, :C, :@ from modmisc to their separate tests

make(1): add dummies for fine-grained tests, one per single feature

The test names have been derived from the current manual page.

All these tests are dummies right now, and the code from the existing
tests will be moved into the new tests step by step.

This is done to prevent modmisc, escape, varmod-edge and varmisc from
growing without any bounds, and to reduce the side-effects of one test
to the others.