CVS log for src/usr.bin/make/main.c

Up to [cvs.NetBSD.org] / src / usr.bin / make

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

make: fix confusing error message when overriding a read-only variable

make: allow -f .../Makefile

If the arg to -f or an entry in .MAKE.MAKEFILE_PREFERENCE
starts with ".../" look for the rest of the path in .CURDIR
and above.

Reviewed by: rillig

make: treat recursive variables non-fatally

A recursive variable is no worse than an unknown modifier, so treat them
in the same way by continuing parsing until the end of the makefile.

make: add more context to error message about recursive variables

Compat_RunCommand use tempfile if cmd too big

Extract the logic recently added to Cmd_Exec to handle
long commands via temp file to Cmd_Argv,
so it can also be leveraged by Compat_RunCommand

Reviewed by: christos

make: error out on parse/evaluation errors in shell commands

The expression ${VAR:X} has an unknown modifier ':X'.  Previously, this
expression errored out when the expression was evaluated at parse time,
but not when the expression was evaluated when generating the commands
to bring a target up to date.  The errors were previously reported, they
didn't affect the exit status, though.

Now, errors in expressions are handled in the same way, regardless of
the time at which they are evaluated.

make: move initialization of variable scopes to targ.c

The variable scopes are freed by Targ_End, so initialize them there as
well.  Separate printing statistics and freeing memory, which makes
Var_End unnecessary.

make: only generate code for cleanup functions in CLEANUP mode

make: clean up condition when printing an error

make: reduce lint-specific comments about ARGSUSED

make: on error, print the targets to be made

This helps to understand situations with several nested sub-makes in
varying directories.

Sync with HEAD.

make: add detailed exit status to message for failed sub-commands

Several commands communicate via the exit status and not only
distinguish between zero and non-zero, so make this information
available to ease investigations.

The command "false" is not guaranteed to exit with a consistent status,
so use "(exit 13)" instead in the tests, to keep these tests portable
across different operating systems.  The exit status 127 is required for
a shell that cannot find a command, so keep that one.

make: sync VarEvalMode constant names with their debug log names

make: fix memory leak in realpath cache

make: replace strncpy with snprintf

There's no point copying a MAXPATHLEN buffer in its entirety when only
the initial string part is ever used.

make: fix variable lifetime when initializing MAKE and .MAKE

The pathbuf buffer is aliased by abspath, which later escapes the block
where the buffer is declared.

make: ensure loop termination in purge_relative_cached_realpaths

You cannot ignore the return from HashIter_Next

make: clean up API for iterating over hash tables

make: allow debugging the warning about .OBJDIR

When make complains about an unwritable .OBJDIR
it is not always obvious how we derrived that value.

If MAKE_DEBUG_OBJDIR_CHECK_WRITABLE is enabled call PrintOnError
so we can examine variables that are likely relevant.

Fix description of MAKE_OBJDIR_CHECK_WRITABLE in make.1

make: in -DCLEANUP mode, free variables and their values

The variables in the 3 scopes must be freed before the scopes themselves
are freed by Targ_End.

The test opt-m-include-dir creates a directory of the form '*.tmp', thus
it must be removed before attempting to only remove regular files of
this name.

POSIX requires setenv to copy the passed name and value, so there is no
need to keep that memory allocated any longer.

Dir_FindFile treat sysIncPath and defSysIncPath as special

Rather than play games with adding .DOTLAST and then having to
exclude that from .SYSPATH, just have Dir_FindFile check if
path is sysIncPath or defSysIncPath and behave as if .DOTLAST seen
and also skip the final search of .CURDIR.

Fix Dir_SetSYSPATH to use defSysIncPath if sysIncPath is empty.

make: all command line overrides go in .MAKEOVERRIDES

Not all variables that start with '.' are internals,
and unless they are explicitly flagged as internal should go
into .MAKEOVERRIDES

Update varname-dot-makeoverrides to check this.

Also avoid using SCOPE_CMDLINE when ReadOnly will do.

make: ensure '.include <makefile>' respects MAKESYSPATH

Since Dir_FindFile is used by '.include' and its variants,
and will first search .CURDIR unless the give path starts with
".DOTLAST".

Update unit-tests/opt-m-include-dir to test this.

make: simplify freeing of lists

make: record exit status in GNode

SetErrorVars can now set .ERROR_EXIT which allows
a .ERROR target to ignore the case of .ERROR_EXIT == 6
which means failure happened elsewhere.

Reviewed by:

make: export target scope values

Pass target scope to Var_ReexportVars so that a target process
will see the correct values in its env.
We must then mark any Global scope variable as unexported
so targets without local value get the Global one.

make: remove unneeded conditional-compilation toggles

The toggles INCLUDES, LIBRARIES, POSIX, SYSVINCLUDE, SYSVVARSUB,
GMAKEEXPORT and SUNSHCMD are no longer needed, they were unconditionally
set.

The toggle NO_REGEX was configurable from the command line, but
disabling it would result in various error messages about the unknown
':C' modifier.

OK sjg@.

make: more consistent error messages

Move %s: progname from Job_CheckCommands to Fatal
to avoid is being repeated when Job_CheckCommands is passed Error.

This means some errors from var also report progname (and level)
which is useful.

Reviewed by: rillig

make: miscellaneous cleanups

Cmd_Exec use tempfile if cmd is too big

To avoid blowing commandline/env limits, if "cmd"
is more than 1000 bytes, write it to a file and pass that
to shell.

Reviewed by: rillig

make: ensure shellPath is set before using it.

make: clean up unused return value of str2Lst_Append

No functional change.

make: clean up names of local variables

No binary change.

make: miscellaneous cleanups

No functional change.

make: clean up comments

No functional change.

make: inline a single-line cross-file function

No functional change.

make: clean up code for parsing the '-j' command line option

No binary change.

make: fix typo

make: add more details to usage message of -j option

make: fix lint warning about strchr

main.c(416): warning:
    call to 'strchr' effectively discards 'const' from argument [346]

Even though C23 turns strchr into a const-generic function, it doesn't
do the same for strtol, so use separate pointers for the current parsing
position and the end of a number, as their constness differs.

Add .MAKE.JOBS.C to indicate wether -jC is supported

add explicit cast for long -> int truncation warning-as-error.

as this is number of CPUs, i don't think we have to care about
it for a long, long, *long* time...

make: allow -j to compute a multiple of ncpu

If _SC_NPROCESSORS_ONLN is supported; and -j arg is a floating point
number or ends in 'C' compute .MAKE.JOBS as a multiple of _SC_NPROCESSORS_ONLN

Based on a suggestion from des at freebsd.org
Discussed with: rillig, christos

make: declare all common symbols in headers, unexport others

No functional change.

make: rename function for parsing command line options

No binary change.

make: inline macros for variable names

The variable name '.MAKEOVERRIDES' was already used in the non-macro
form.

No binary change.

make: clean up calls to Var_Subst

None of the calls to Var_Subst used the return value, and the return
value was always VPR_OK.

No functional change.

make: some variables should be read-only

Make variables like .newline and .MAKE.{GID,PID,PPID,UID} read-only.

Reviewed by: rillig

make: .SYSPATH: to add dirs to sysIncPath

.SYSPATH: with no sources will clear sysIncPath
otherwise sources are added

Reviewed by: rillig

make: inline macro for variable name

This fixes the inconsistency of using the macro name in one place and
its value in another place (since 2010).

No binary change.

make: don't assemble identifiers from smaller tokens

No binary change.

make: change return type of unlink_file back to int

As unlink_file is a wrapper around unlink, use the same encoding for the
possible return values as in the wrapped function.  This consistency is
more important than expressing all possible return values in the return
type 'bool'.

https://mail-index.netbsd.org/tech-toolchain/2022/10/06/msg004155.html

No functional change.

make: document the guard for directories in unlink_file

Document only the POSIX requirement for now, as I didn't find
information about _which_ ancient UNIX systems would corrupt the
filesystem on unlinking a directory.

http://man.cat-v.org/unix-1st/2/sys-unlink (1971) says:
> It is also illegal to unlink a directory (except for the super-user).

https://mail-index.netbsd.org/tech-toolchain/2022/10/06/msg004147.html

Don't ignore return from snprintf or getcwd

make: allow to randomize build order of targets

In complex dependency structures, when a build fails, a probable cause
is a missing dependency declaration between some files.  In compat mode,
the build order is deterministic, in jobs mode, it is somewhat
deterministic.  To explore more edge cases, add the line ".MAKE.MODE +=
randomize-targets" somewhere in the makefile.

Fixes PR bin/45226 by riastradh.  Reviewed by christos.

make: rename Compat_Run to Compat_MakeAll

No functional change.

make: only switch to POSIX mode if '.POSIX:' is the first line

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/make.html
says that in order to make a makefile POSIX-conforming, its first
non-comment line must be the special dependency line '.POSIX:' without
any source dependencies.

Previously, make switched to POSIX mode even if such a line occurred
anywhere else, which was allowed by POSIX but was deep in the
"unspecified behavior" area.  For NetBSD make, there is no big
difference since it doesn't ship any <posix.mk> file, this change mainly
affects the bmake distribution.

Previously, makefiles that contain '.POSIX:' somewhere in the middle
could fail due to <posix.mk> resetting .SUFFIXES, among other things.

Suggested by Simon J. Gerraty, who also reviewed an earlier version of
this change.

make: clean up comments and code for setting debug flags

No binary change.

make: prefix the warning about read-only .OBJDIR with a colon

For consistency with the other warnings.

make: print stack trace on fatal errors

The only fatal error that occurs while the makefiles are read in is the
one about recursive variables, which didn't give any hint about the
location before.

If a recursive variable is detected while evaluating the commands of a
target to be made, there is no location information, as before.

Allow local variable assignments in dependency lines

The variable is set in the context of the target.
This syntax has been supported by gmake for ~ever.
If necessary a makefile can set .MAKE.TARGET_LOCAL_VARIABLES=false
to disable this.

Expose GetBooleanExpr so parse.c can use it.

make: add missing newline after "cannot continue" message

It was wrong of Parse_File to output an unfinished line and hope for
some other code to finish it.  As demonstrated in the test, PrintOnError
did not do that in the case of additional debug output.

To keep the overall behavior as close as possible to before, the other
callers of PrintOnError now have to pass the newline themselves. Passing
strings that start with newlines but don't end with them looked
suspicious anyway.

make: clean up comments

make: remove extra semicolon after statement

Lint will complain about this very soon.

No binary change.

make: replace Var_Value with Var_Exists where applicable

The latter function already existed in 1993, no idea why it was not
used.

No functional change.

make: remove unnecessary functions for expanding variable names

In meta mode, the affected variable patterns do not contain a '$'.

Outside of meta mode, Global_SetExpand was only called a single time, so
inline that call.

No functional change.

make: merge duplicate code for expanding variable expressions

No functional change.

make: add debug logging for capturing the output of external commands

This applies to all 4 situations in which the output of an external
command is used for modifying a variable or an expression:

* the assignment operator '!='
* the assignment modifier '::!='
* the SUN shell modifier ':sh'
* the other shell modifier ':!cmd!'

Previously, only the shell modifier ':!cmd!' had debug logging.

Suggested by Christoph Badura.

make: in Cmd_Exec, return error message instead of format string

This change leaves only literal format strings in parse.c.  It allows
for more detailed error messages than the current "non-zero status" or
"exited on a signal".

No functional change.

make: have as few statements as possible between va_start and va_end

No functional change.

make: clean up comments, variable names, function names

The comment in ApplyDependencySourceOther repeated the code, its second
half didn't match any current code.

The comment above ParseDependencySourcesEmpty repeated the code.

No binary change, except for assertion line numbers.

make: fix error message when reading more than 1 GB from stdin

Previously, the error message was:
	make: (null): file too large
Now it is:
	make: (stdin): file too large

make: reduce indentation in Main_SetObjdir

No functional change.

make: unexport types VarAssignOp and VarAssign

These types are only needed in the parsing module.

No functional change.

make: fix double-free in CLEANUP mode (since 2021.12.27.23.11.55)

When make is run without the '-f' option, it searches for the files
'makefile' and 'Makefile' in the current directory.  The function
ReadFirstDefaultMakefile allocated memory for these filenames, added the
filenames to opts.makefiles and then freed the memory.  From that
moment, opts.makefiles contained dangling pointers.

The function main_CleanUp cleans the list, but only if make is compiled
with -DCLEANUP.  Since main.c 1.557 from 2021.12.27.23.11.55, the
strings in opts.makefiles are freed as well, before that, only the list
nodes were freed.  Freeing the strings led to the double-free.

Fix this bug by using a separate list for these short-lived strings.  At
the point where ReadFirstDefaultMakefile is called, opts.makefiles is
not used anymore, therefore there are no side effects.

To reproduce, run 'make test-coverage', which compiles with -DCLEANUP.
The test opt-chdir failed with a segmentation fault in main_Cleanup.
This test may be the only one that doesn't use the option '-f'.

make: clean up stylistically

No binary change.

make: constify cached_realpath

No binary change.

make: clean up Cmd_Exec

Renaming savederr to saved_errno makes the comment redundant.

Group the conditions for setting errfmt, retaining their relative order.

No functional change.

make: reduce indentation of Cmd_Exec

No functional change.

make: free the names of the makefiles in cleanup mode

Since parse.c 1.576 from 2021-12-13 the filenames from opts.makefiles no
longer end up in the GNodes, they are copied by Str_Intern.

make: merge local variables holding the path to the object directory

This is a similar pattern as in the other situations where a string is
fed through Var_Subst.  In this case though, the unexpanded string may
need to be freed, therefore the FStr_Done that is not needed in the
other places.

No functional change.

make: prevent out-of-bounds read for debug log file name

Even though the name of the debug log file currently only occurs in
strings of the form '-dFname' or '-dF+name', the code for replacing '%d'
with the PID accesses the passed string out of bounds.  That's not a
problem in practice but looks suspicious anyway.

make: remove usage message if the debug file cannot be opened

Since a non-writable file is not a syntax error, there is no point in
showing the usage in this situation.  Showing the usage may have been a
copy-and-paste mistake from a few lines below, when this option was
added back in main.c 1.133 from 2006-10-15.

make: narrow down the return type of ReadMakefile

No functional change.

make: rename local variables to be simpler

No binary change.

make: clean up 'explode'

Breaking out of the first 'for' loop was unnecessarily complicated.  The
call to strlen was not necessary since f already pointed at the end of
the string.

No functional change.

make: clean up comments

make: remove unnecessary words from command line options

Several years ago, the command line options were individual global
variables.  The global variable could therefore not be named 'silent'
since that would have conflicted with local variables of the same name.
After moving the global variable to the namespace 'struct CmdOpts',
there is no conflict anymore.

There doesn't seem to be any risk of naming collisions for the names
'touch' and 'query'.

No functional change.

make: rename eunlink to unlink_file

The name eunlink suggested a relation with the similarly named functions
emalloc or esnprintf, but that was misleading.  Instead, unlink_file
works like unlink, except that it refuses to remove an empty directory.

No functional change.

make: format comments according to /usr/share/misc/style

Assisted by indent(1), with manual corrections due to its many remaining
bugs.

No functional change.

make: use consistent indentation for statements and continuations

No binary change, except for line numbers in assertions in suff.c.

make: remove bmake_free

It was only used in 2 places, and in both these places, the pointer was
never null.

make: in CLEANUP mode, free interned strings at the very end

Noticed by sjg.

make: convert debugging flags from enum to bit-field

This gets rid of the magic numbers, making it possible to add another
debug flag without renumbering the others.

No functional change.

make: fix memory leak for filenames in .for loops (since 2013-06-18)

Previously, each time a .for directive pushed its buffer on the input
file stack, the current filename was duplicated.  This was a waste of
memory.

The name of a file is typically only used while it is read in.  There is
one situation when the filename is needed for longer, which is when a
target is defined.

Since .for loops are implemented as a special form of included files,
each .for loop duplicated the current filename as well.

$ cat << EOF > for.mk
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.endfor
.endfor
.endfor
.endfor
.endfor
.endfor
.endfor

all:
	@ps -o rsz -p ${.MAKE.PID}
EOF

$ make-2021.12.13.03.55.16 -r -f for.mk
  RSZ
10720

$ ./make -r -f for.mk
 RSZ
1716

The difference is 8 MB, which amounts to 1 million .for loops.

make: rename variable and function for handling parse errors

The word 'fatals' was an unnecessary abbreviation.

No functional change.

make: clean up access to character iterator

Having both p[0] and *p intermixed was inconsistent.

No functional change.

Pull up following revision(s) (requested by mrg in ticket #1678):

	bin/pax/options.c: revision 1.119
	usr.bin/config/scan.l: revision 1.32
	usr.bin/make/main.c: revision 1.274
	bin/pax/extern.h: revision 1.60
	usr.bin/config/defs.h: revision 1.106
	usr.bin/make/make.h: revision 1.105
	sbin/newfs_udf/udf_create.c: revision 1.27 (patch)
	sbin/newfs_udf/newfs_udf.h: revision 1.7 (patch)
	usr.bin/config/main.c: revision 1.100
	usr.sbin/installboot/Makefile:	revisions 1.53,1.54 (patch)
	usr.sbin/installboot/installboot.h: revision 1.41 (patch)
	usr.sbin/installboot/machines.c: revisisons 1.41,1.42 (patch)

Avoid depending on common symbols.

Pull up following revision(s) (requested by mrg in ticket #1263):

	bin/pax/options.c: revision 1.119
	usr.bin/config/scan.l: revision 1.32
	usr.bin/make/main.c: revision 1.274
	bin/pax/extern.h: revision 1.60
	usr.bin/config/defs.h: revision 1.106
	usr.bin/make/make.h: revision 1.105
	sbin/newfs_udf/udf_create.c: revision 1.27 (patch)
	sbin/newfs_udf/newfs_udf.h: revision 1.7 (patch)
	usr.bin/config/main.c: revision 1.100

Avoid depending on common symbols.

make: avoid double slash in name of temporary directory

If the environment variable TMPDIR is not set, make uses a default path
that includes a trailing '/'.

For extra correctness it always appended an extra '/', leading to paths
of the form '/tmp//makeXXXXXX'.  This looked suspicious, as if there had
been a forgotten empty part between the two '/'.  Avoid this ambiguity
by replacing '//' with '/'.

make: turn run-time string concatenation into compile-time

No functional change.

make: remove unnecessary modifier ':U' for certain fixed expressions

No functional change, since the expression is evaluated using
VARE_WANTRES, not using VARE_UNDEFERR.

make: remove filler word 'Do' from function names for parsing

No functional change, except for debug logging.

make: rename a few functions to be more descriptive

No functional change.

make: use C99 bool type instead of defining its own

No functional change.

Avoid strdup in mkTempFile

Require caller to pass a buffer and size if they
want the tempfile not unlinked.

Add Job_TempFile to handle blocking signals around
call to mkTempFile, so that meta_open_filemon can use it
in jobs mode.

make: in the Var_ functions, move the scope to the front

This change provides for a more natural reading order in the code.
Placing the scope first makes it immediately clear in which context the
remaining parameters are interpreted.

No functional change.

make: add shortcut Global_Delete for deleting a global variable

make: rename some VAR constants to SCOPE

The word "context" does not fit perfectly to the variables that are
associate with a GNode, as the context is usually something from the
outside and the variables are more like properties inherent to the
GNode.

The term "global context" fits even less.  Since the thing where
variables are looked up is commonly named a scope, use that term
instead.

This commit only renames the global variables VAR_GLOBAL, VAR_INTERNAL
and VAR_CMDLINE, plus a few very closely related comments.  These are:

	GNode.vars (because of line breaks)
	GNode_Free (dito)
	varname-make_print_var_on_error.mk
	varname-make_print_var_on_error-jobs.mk

The debug message in Var_Stats is left as-is since there is no unit test
for it yet.

The other renamings (variable names "context", "ctxt", as well as
further comments) will be done in a follow-up commit.

make: replace Global_AppendExpand with Global_Append

All callers with a variable name that is guaranteed to not contain a
dollar sign have been converted to call Global_Append instead of the
previous Global_AppendExpand.  After that, Global_AppendExpand was
unused, therefore it was effectively just renamed.

make: replace Global_SetExpand with Global_Set for constant names

make: use shortcut functions Global_SetExpand and Global_AppendExpand

There are many places where global variables are set or appended to.  To
reduce clutter and code size, encode the VAR_GLOBAL in the function
name.

The word Expand in the function names says that the variable name is
expanded.  In most of the cases, this is not necessary, but there are no
corresponding functions Global_Set or Global_Append yet.

Encoding the information whether the name is expanded or not in the
function name will make inconsistencies obvious in future manual code
reviews. Letting the compiler check this by using different types for
unexpanded and expanded variable names is probably not worth the effort.
There are still a few bugs to be fixed, such as in SetVar, which expands
the variable name twice in a row.

make: always use vfork, never fork

Before compat.c 1.217, job.c 1.390 and main.c 1.504 from 2020-12-27, the
exported make variables were exported from each freshly forked child
process.  There was no practical difference though between exporting the
variables from the parent process or the child process since these two
processes share the same address space, except that the forked process
is very limited in what it may actually do.  This limitation was
violated on a regular basis.

When an exported variable referred to a variable that used the :sh
variable modifier, this led to a fork from within vfork, which is not
allowed.  Since 2020-12-27, exporting the variables is done from the
main process, which prevents this situation from ever occurring.

Since that day, there is no need anymore to distinguish between vfork
and fork, which removes any need for the macro.

make: clean up main.c

make: simplify MakeMode

make: replace pre-increment with post-increment or simple addition

The rest of the code already prefers post-increment if there is no
actual difference.

make: rename functions for parsing command line arguments

The function names now match the names around them.

make: clean up comments

make(1): inline Buf_Len

make(1): inline Buf_GetAll

make(1): split Buf_Destroy into Buf_Done and Buf_DoneData

In all cases except one, the boolean argument to Buf_Destroy was
constant.  Removing that argument by splitting the function into two
separate functions makes the intention clearer on the call site.  It
also removes the possibility for using the return value of Buf_Done,
which would have made no sense.

The function Buf_Done now pairs with Buf_Init, just as in HashTable and
Lst.

Even though Buf_Done is essentially a no-op, it is kept as a function,
both for symmetry with Buf_Init and for clearing the Buffer members
after use (this will be done only in CLEANUP mode, in a follow-up
commit).

make(1): convert SearchPath to struct

This prepares for making dotLast a simple struct member instead of a
fake CachedDir, which is easier to understand.

make(1): rename Dir_AddDir, reorder parameters of SearchPath_ToFlags

make(1): rename Dir_Expand to SearchPath_Expand

The main subject of this function is the search path.  In this search
path the pattern is expanded.

make(1): remove do-not-format markers from comments

These markers had been used inconsistently.  Furthermore the source code
had not been formatted automatically before 2020 at all, otherwise there
wouldn't have been any trailing whitespace left.

make(1): fix a few inconsistencies for lint's strict bool mode

make(1): make a few more bool expressions more precise

The previous version of lint(1) from a few hours ago didn't catch all
occurrences.  And even the current one doesn't catch everything.
Function arguments and return types still need some work.  The "return
quietly" from shouldDieQuietly still implicitly converts from int to
_Bool.

No functional change.

make(1): consistently use boolean expressions in conditions

Most of the make code already followed the style of explicitly writing
(ptr != NULL) instead of the shorter (ptr) in conditions.

The remaining 50 instances have been found by an experimental,
unpublished check in lint(1) that treats bool expressions as
incompatible to any other scalar type, just as in Java, C#, Pascal and
several other languages.

The only unsafe operation on Boolean that is left over is (flags &
FLAG), for an enum implementing a bit set.  If Boolean is an ordinary
integer type (the default), some high bits may get lost.  But if Boolean
is the same as _Bool (by compiling with -DUSE_C99_BOOLEAN), C99 6.3.1.2
defines that a conversion from any scalar to the type _Bool acts as a
comparison to 0, which cannot lose any bits.

make(1): fix lint warnings

Ensure PrintOnError always reports 'stopped in' on first call.

We may still suppress the rest of the noise if shouldDieQuietly
says to, but the 'stopped in' output is too important to lose.
Avoid repeating it though in the same process.

For the case of aborting due to failure detected elsewhere,
exit 6 so we have a clue.

PR: 55578
Reviewed by:

make(1): replace pointers in controlling conditions with booleans

make(1): format multi-line comments

make(1): replace global preserveUndefined with VARE_KEEP_UNDEF

Controlling the expansion of variable expressions using a global
variable and a VARE flag was inconsistent.

Converting the global variable into a flag had to prerequisites:

1.  The unintended duplicate variable assignment had to be fixed, as
done in parse.c 1.520 from 2020-12-27.  Without this fix, it would have
been necessary to add more flags to Var_Exists and Var_SetWithFlags, and
this would have become too complex.

2.  There had to be a unit test demonstrating that VARE_KEEP_DOLLAR only
applies to the top-level expression and is not passed to the
subexpressions, while VARE_KEEP_UNDEF applies to all subexpressions as
well.  This test is in var-op-expand.mk 1.10 from 2020-12-28, at least
for the ':@word@' modifier.  In ParseModifierPartSubst, VARE_KEEP_UNDEF
is not passed down either, in the same way.

make(1): exit 2 on technical errors

This allows the -q option to distinguish errors from out-of-date
targets.  Granted, it's an edge case but it should be solved
consistently anyway.

The majority of cases in which make exits with exit status 1, even in -q
mode, is when there are parse errors.  These have been kept as-is for
now as they affect many of the unit tests.

The technical errors, on the other hand, occur so rarely that it's hard
to write reliable tests for them that fail consistently on all platforms
supported by make.

make(1): re-export variables from the actual make process

Since make uses vfork if available, re-exporting the variables happens
in the address space of the main process anyway, so there is no point in
mentioning anything about "our client process" anywhere.

Use .MAKE.DEPENDFILE as makefiles set it

make(1): fix a few lint warnings

make(1): rename CmdOpts.lint to strict

When running lint(1) on the code, it defines the preprocessor macro
"lint" to 1, which generated a syntax error in the declaration "Boolean
lint", as that became "Boolean 1".

make(1): remove constant parameter from MakeMode

make(1): change return type of Var_Value to FStr

make(1): add str_basename to reduce duplicate code

The function basename from POSIX has a few unfortunate properties, it is
allowed to return a pointer to static memory.  This is too unreliable,
therefore this trivial own implementation.

make(1): constify progname

make(1): replace *line with line[0]

Since a line is not an iterator and since the expression *line typically
means "the current element", not "the first character", replacing *line
with line[0] more directly expresses the idea of accessing the first
character of a string.

make(1): rename Var_ExportVars to Var_ReexportVars

make(1): clean up comments for command line options

make(1): remove DEFMAXLOCAL and DEFMAXJOBS

These have been hard-wired all the time.  The word LOCAL doesn't make
sense anymore since the code for running jobs remotely has gone.

make(1): define constants for enum zero-values

make(1): remove Dir_InitDir

The function name had been too ambiguous since it didn't mention the
particular directory that was initialized.  Instead of that function,
Dir_InitCur is called directly from main_Init.

The pseudo CachedDir entry ".DOTLAST" is initialized at the very
beginning.  The observable behavior is unchanged since this a
memory-only object with no connection to the file system.

make(1): reduce memory allocation for dirSearchPath

make(1): fix main_CleanUp in -DCLEANUP mode (broken since 1 hour)

Broken since main.c 1.486 from 2020-11-28, about 1 hour ago.

make(1): reduce memory allocation for targets

This change moves the initialization and finalization of the list of
targets to the same function.  They had been split before.

make(1): reduce memory allocation in ReadBuiltinRules

make(1): reduce memory allocation for CmdOpts.create

make(1): reduce memory allocation in CmdOpts.variables

make(1): reduce memory allocation in CmdOpts.makefiles

make(1): remove pointer indirection from GNode.commands

Just to save a few memory allocations.  No noticeable effect on the
performance though.

make(1): rename local variable in Cmd_Exec

It conflicts with the global fds in job.c.

make(1): rename conflicting global variables called 'error'

When compiling make in all-in-one mode, these variable names conflict.
They could have been merged into a single variable, but that would have
required to make it a global variable for the other modules as well.
The parse module has a similar variable called 'fatals'.  All these can
possibly be merged into a single variable, but not now.

Add .MAKE.UID and .MAKE.GID

make(1): fix indentation for short expressions in main.c

make(1): use comparisons in boolean expressions

The generated code stays exactly the same.

make(1): use properly typed comparisons in boolean contexts

make(1): fix initialization order of modules (broken since today)

In CLEANUP mode, Var_Init depends on Targ_Init since the variable scopes
are implemented as GNodes.

By the way, since 1999-09-15 variables are no longer stored in lists but
in hash tables.

make(1): use postfix increment where possible

make(1): clean up ParseBoolean

make(1): clean up getTmpdir

Reduce indentation, structure code into paragraphs.

make(1): clean up PrintOnError

No functional change.

make(1): fix previous commit for parse_debug_options

make(1): reduce code size of parse_debug_options

Each access to a global variable requires a relocation.  Skip these.

make(1): switch cache for realpath from GNode to HashTable

An unintended side effect from the GNode implementation was that the
variable modifier :tA and the other places where cached_realpath are
used could be affected by setting a variable in the global scope,
thereby "redirecting" absolute paths to completely unrelated but
existing paths.

Another unintended side effect was that filenames containing a dollar
sign would not be resolved correctly since the dollar sign would be
expanded as a variable expression by Var_Set.

While here, the debugging output for the realpath cache has been
adjusted to the standard behavior.  Previously, when a new entry was
added to the cache, this was logged for the module VAR, as a side effect
of calling Var_Set, but only if the preprocessor macro
DEBUG_REALPATH_CACHE was defined at compilation time.  When relative
paths were purged from the cache because the current directory changed
and logging for the DIR module was active, the log output went directly
to stderr instead of the usual opts.debug_file.  This deviation from the
standard behavior was probably not intended as well.

All logging concerning the realpath cache now goes into the standard
debug log file and is controlled by the -dd option, not -dv.

make(1): document the -S option

This option has been available at least since 1993-03-21, and for the
same time it has been undocumented.

make(1): fix trailing whitespace in usage (since 2009-08-27)

make(1): don't modify progname in usage

make(1): clean up Fatal

Use the shortest possible scope between va_start and the corresponding
va_end.

make(1): clean up InitDefSysIncPath

Flipping the arguments to strncmp puts the subject first in the
sentence. Inverting the condition makes the code flow easier to follow.

make(1): use progname instead of hard-coded 'make' in warning

This only affects the warning that chdir to objdir failed.  In sub-makes
the progname includes the [n], allowing to narrow down the actual cause
of the problem.

make(1): remove redundant parameter from str2Lst_Append

make(1): rename Main_SetVarObjdir to SetVarObjdir

Only exported functions should be prefixed with the module name.

make(1): use different style of accessing characters in MainParseArgs

The * is preferred for iterators.  Since argv[i] is not an iterator but
a fixed string, argv[i][0] expresses the idea "read the first character"
more directly.

make(1): rename DEFAULT to defaultNode

In C, uppercase names are typically used for constants, and this is not
a constant.

make(1): clean up preprocessor macros in InitVarMachine

make(1): flatten #ifdef in InitVarMachineArch

Without proper indentation, that chain of nested #ifndef directives was
really hard to read.

Pass a writable flag to Main_SetObjdir to control writable check

For curdir and an explicit .OBJDIR target, we allow for
the directory to be read-only.
During InitObjdir we otherwise default to requiring objdir to be
writable - this can be controlled by env variable
MAKE_OBJDIR_CHECK_WRITABLE

Add unit-tests/objdir-writable

Reviewed by: christos rillig

make(1): move lint option away from the debug flags

The lint option affects much more than a typical debug flag.  Not only
does it produce additional diagnostics, it also changes the control flow
and fixes the traditional error handling in a few places.

No functional change.

make(1): use strict typing in conditions of the form !var

make(1): reformat main.c more closely to share/misc/style

make(1): remove redundant comment from main_ReadFiles

The code has become so simple that it made the comment redundant.

make(1): rename init_machine to InitVarMachine

For consistency with the other Init functions.

make(1): move usage further to the top

This avoids a forward declaration.

make(1): move documentation from main to the phase functions

make(1): split main into manageable pieces

The purpose of a main function is to give a high-level overview about the
whole program.  270 lines of code with lots of tricky details did not serve this
purpose.  Split the code into functions corresponding to the phases.

make(1): indent main consistently

make(1): inline ReadMakefileSucceeded into ReadFirstDefaultMakefile

This gets rid of a few void pointers and an unused function parameter.

make(1): inline Lst_ForEachUntil in ReadBuiltinRules

The list of expanded filenames for sys.mk is freed in the end, but not
the filenames themselves, as these are still needed.

make(1): use common indentation style for else

make(1): split ReadMakefiles into separate functions

This keeps the indentation of the code small.

It also reduces the possible confusion about the two similar branches in
that function that differ in a small but important detail:
ReadAllMakefiles reads all the makefiles while ReadFirstDefaultMakefile
stops after the first existing makefile.

make(1): fix unrealistic memory leak in Main_ParseArgLine

It's unlikely that anyone ever defines an environment variable named
".MAKE" and then runs make with the -e option, which would make the
environment variable stronger than the built-in global variable.

make(1): add remarks for missing command line options in MAKEFLAGS

make(1): parse the internal option -J strictly

make(1): merge duplicate code in is_relpath

make(1): clean up overview comment in main.c

make(1): fix type mismatch between int and PrintVarsMode

make(1): remove trailing whitespace

make(1): make API of Buf_Init simpler

In most cases, the caller doesn't want to specify the exact number of
preallocated bytes.

make(1): document InitDefSysIncPath

make(1): clean up code stylistically

* Replace character literal 0 with '\0'.
* Replace pointer literal 0 with NULL.
* Remove redundant parentheses.
* Parentheses in multi-line conditions are not redundant at the
  beginning of a line.
* Replace a few !ptr with ptr == NULL.
* Replace a few ptr with ptr != NULL.
* Replace (expr & mask) == 0 with !(expr & mask).
* Remove redundant braces for blocks in cases where the generated code
  stays the same.  (Assertions further down in the code would get
  different line numbers.)
* Rename parameters in CondParser_String to reflect the data flow.
* Replace #ifdef notdef with #if 0.

The generated code stays exactly the same, at least with GCC 5.5.0 on
NetBSD 8.0 amd64 using the default configuration.

make(1): rename getBoolean and s2Boolean

make(1): fix comment of Error

make(1): clean up Cmd_Exec stylistically

make(1): rename dieQuietly to shouldDieQuietly

It was too confusing to have a function named die that doesn't actually
die.  Plus, the return type int didn't give any clue about what the
function actually returns.

make(1): fix typo from previous commit

make(1): fix wrong condition in mkTempFile (since 2020-10-31)

The wrong negation had been added in main.c 1.414 from 2020-10-31.
Found by GCC 10, which complained about a potential null pointer
dereference in line 2188.

make(1): remove redundant parentheses from sizeof operator

The parentheses are only needed if the argument is a type, not an
expression.

make(1): move declaration of parseIncPath from main.c to make.h

make(1): rename MAKEFILE_PREFERENCE for consistency

The names of the other special variables are all prefixed with MAKE_.

make(1): negate discardUndefined to preserveUndefined

make(1): rename oldVars to discardUndefined

While here, moved all the documentation about this variable into a
single place.

make(1): in lint mode, exit with error status on errors

Calling Parse_Error during parsing has always led to a nonzero exit
status.  Calling Parse_Error later, when expanding the shell commands,
has had no effect on the exit status.  Neither had calling Error.

To make make a reliable tool, it has to report errors as they occur.
Enable this strict behavior in lint mode for now.  Lint mode has to be
enabled explicitly, preserving the default behavior.

Main_SetObjdir is first called for curdir which may be readonly

Skip access check if path is curdir.
This ensures that all proper initialization is done at least once.

If path is not curdir it should be writable to be useful.

Reviewed by: rillig

make(1): revert requiring a writable objdir

The change in main.c 1.413 broke the NetBSD build.sh if it uses a
read-only source tree, as in the daily builds.

Original commit:
https://mail-index.netbsd.org/source-changes/2020/10/31/msg123560.html

Build log:
make warning: /home/source/ab/HEAD/src: Permission denied.
[1]   Segmentation fault      "${make}" -m ${T...

make(1): reduce the scope where recursive expressions are detected

Only the call to Var_Subst needs to be protected since the other
functions have nothing to do with expanding variables.

make(1): remove redundant :tl from getBoolean

Uppercase and lowercase letters are already handled the same by
s2Boolean, which makes the :tl modifier redundant.

make(1): remove redundant null check from s2Boolean

make(1): write s2Boolean in a more compact form

make(1): clean up mkTempFile

Do not use an objdir that is not writable.

make(1): change char * to void * in Var_Value

The only purpose of the parameter freeIt is to free the memory
associated with the return value.  To do this, no pointer arithmetic is
needed.  Therefore, change to a void pointer, to catch accidental use of
that pointer.

make(1): fix indentation in source code

make(1): rename VAR_CMD to VAR_CMDLINE

Since make has to do with both the command line and child commands, the
former name was confusing.

make(1): rename defIncPath to defSysIncPath

There are two variables, parseIncPath and sysIncPath, which made the
name defIncPath ambiguous.

make(1): extract InitMaxJobs from main

make(1): extract InitObjdir from main

make(1): make InitVarMake simpler and document its intention

make(1): extract InitVarMake from main

make(1): extract InitRandom from main

make(1): extract ReadMakefiles from main

make(1): extract CleanUp from main

This makes it a bit easier to spot the main task of the main function,
which is the runTargets call.

make(1): free the variable names given on the command line

These are not aliased anywhere.  The opts.makefiles on the other hand
may be referenced in GNode.fname or pretty much anywhere else, so don't
free them.  There's still a Targ_PrintGraph call below it that might use
them.

make(1): extract InitVpath from main

make(1): use constant string for Var_Subst

The times where Var_Subst needed a modifiable string are long gone.

make(1): extract ReadBuiltinRules from main

make(1): extract InitDefIncPath from main

make(1): initialize all CmdOpts fiels

make(1): extract CmdOpts_Init from main

make(1): move debugVflag to CmdOpts

make(1): extract UnlimitFiles from main

make(1): remove "Results: none" from the documentation of void functions

make(1): group the command line options and arguments

By having a single struct that holds all command line options and
arguments, it is easy to see in the code when such a command line
argument is modified.  It also cleans up the namespace since the command
line options don't follow a common naming style.  Having them in a
struct also means that there is a single place for putting the
documentation, not two as before.

The struct also suggests to extract the initialization code out of main,
which is still too large, having more than 400 lines of code and
covering far too many topics.

make(1): rename hash functions to identify the type name

This makes it easier to spot mismatches between the function name and
its first parameter, although the compiler should already catch most of
them.  Except for void pointers.

Skip reading .MAKE.DEPENDFILE if set to
"/dev/null" or anything starting with "no".

Ref: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=223564

make(1): convert macros for debug flags into enum

make(1): remove unused Lst_Find and Lst_FindFrom

make(1): inline ReadMakefileFailed, eliminating some void pointers

make(1): remove void pointers from addErrorCMD

make(1): clean up PrintVar

Conceptually, the last parameters of Var_Subst and Var_Value differ a
lot.  The former is the actual variable value while the latter is just a
pointer to be freed.  It will be changed to a void pointer in a few
commits.

To keep the number of needed variables small and the code simple, the
printf statement is mentioned in each branch.  An additional benefit is
that in 2 of the 3 branches, no null pointer can occur.

make(1): extract PrintVar from doPrintVars

make(1): clean up HandlePWD

make(1): extract HandlePWD from main

make(1): remove redundant type casts

This mainly affects the void pointers in callback functions for lists.
These had been necessary once when the parameter type was still
ClientData instead of void pointer.

make(1): inline simple Lst getters

The function call variant takes more screen space than the direct field
access.  Having an abstract API is usually a good idea, in this case of
simple read-only member access it makes the code more difficult to read.

LstNode_Set has been kept as a function since it is not a read-only
accessor function.

make(1): rename Lst_Init to Lst_New

For the other types such as HashTable and Buffer, the Init function does
not allocate the memory for the structure itself, it only fills it.

make(1): rename HashEntry.name to key

make(1): remove underscore from Hash_Table and Hash_Entry

For consistency with the other type names, such as GNodeListNode.

make(1): prepare for WARNS=6

The FD_* macros from sys/sys/fd_set.h use signed integers on NetBSD 8
and thus produce conversion errors.  On NetBSD 9, these macros are fixed
to use 1U instead of 1.

make(1): make API for iterating over hash tables simpler

make(1): remove USE_IOVEC

When a system call in a child process failed, there was no guarantee
that the error message would be written completely.  Using writev
correctly is harder than building the string in a buffer and then
writing it in the traditional way.

According to POSIX 2004, using memory allocation or even calling
write(2) from the child process invokes undefined behavior.  The
remaining code from make has done all this for several years now,
successfully, therefore adding one more of that won't hurt.

Make still tries to write the error message atomically by passing the
whole buffer at once to write(2), just as in the previous writev(2)
implementation.

make(1): replace execError with execDie

All calls to this function were followed by _exit(1).

make(1): remove pathname limit for Dir_FindHereOrAbove

While trying to compile the code with GCC's -Wformat-truncation, the
snprintf calls felt quite complicated.  The function Dir_FindHereOrAbove
is not in a bottleneck execution path, therefore it doesn't hurt to
dynamically allocate the memory instead of using size-limited stack
memory.

make(1): prepare job.c, main.c, parse.c, suff.c for WARNS=6

In job.c, GCC 5 complains about the macro FILENO that it has type
unsigned int, which is then passed as the argument of dup2, which
expects a simple int.  Maybe this workaround from 1995 is not necessary
anymore, or maybe it is but can be restricted to the few affected
platforms.

This leaves meta.c and the filemon files to be converted.  I didn't do
them since they are not well covered by the unit tests.

make(1): revert previous commit

It had accidentally reverted all the work from the past few days.

make(1): fix double-free bug in -DCLEANUP mode (since 2020-10-02)

The bug had been introduced with dir.c 1.155 on 2020-10-02 22:20:25.  In
that commit, openDirectories was replaced with a combination of a list
with a hash table, for more efficient lookup by name.

Upon cleanup, OpenDirs_Done is called, which in turn called
Dir_ClearPath.  Dir_ClearPath takes full ownership of the given list and
empties it.  This was no problem before since afterwards the list was
empty and calling Lst_Free just frees the remaining list pointer.

With OpenDirs, this list was combined with a hash table, and the hash
table contains the list nodes, assuming that the OpenDirs functions have
full ownership of both the list and the hash table.  This assumption was
generally correct, except for the one moment during cleanup where full
ownership of the list was passed to Dir_ClearPath, while the hash table
still contained pointers to the (now freed) list nodes.  This by itself
was not a problem since the hash table would be freed afterwards.  But
as part of Dir_ClearPath, OpenDirs_Remove was called, which looked up
the freed directory by name and now found the freed list node, trying to
free it again.  Boom.

Fixed by replacing the call to Dir_ClearPath with code that only frees
the directories, without giving up control over the list.

make(1): extract init_machine and init_machine_arch from main

make(1): remove duplicate code for parsing a variable name

make(1): make documentation of Error more precise

make(1): clean up #include sections

make(1): replace backslash-newline with ordinary string literals

The character sequence backslash-newline is not mentioned in ISO C99.

make(1): allow to disable debug logging options

This is only intended for the unit tests, to selectively enable and
disable debug logging as needed.  Previously the tests for debug logging
required complicated postprocessing that was not obvious when looking at
the test .mk file alone, but was specified externally in
unit-tests/Makefile.

This is the simplest possible implementation.  Another variant would
have been to selectively disable individual debug logging options, but
that would have made the code more complicated.

The -dL option is not affected by -d0 since that is not really a debug
option, even if it is implemented as one.

make(1): clean up documentation of Main_ParseArgLine

The previous documentation contained lots of irrelevant implementation
details.

make(1): add missing const for Parse_AddIncludeDir

make(1): clean up option handling

The test for argvalue == NULL for the individual options was unnecessary
since that is already done in the common part.

make(1): format code in MainParseArgs

make(1): extract handling of the -m option into separate function

make(1): extract option handling for -j/-J into separate functions

make(1): extract MainParseArgChdir into separate function

This removes the struct stat from MainParseArg.

make(1): extract MainParseArg into separate function

That part of the code is long enough and independent enough to get its
own function.  Having a struct stat in MainParseArgs just didn't feel
like a correct abstraction level.

make(1): remove redundant function prototypes

make(1): make debugging code shorter

make(1): normalize whitespace in source code

There is no more space tab.  Either only tabs or only spaces or tabs
followed by spaces, but not spaces followed by tabs.

make(1): rename Buf_Size to Buf_Len

The new name better matches the field name Buffer.len as well as the
variables around the calls to this function.

make(1): revert migration from Lst_ForEachUntil to Lst_ForEach

There is a crucial difference between these functions, in that
Lst_ForEachUntil can cope with a few concurrent modifications while
iterating over the list.  This is something that Lst_ForEach doesn't do.

This difference led to a crash very early in NetBSD's build.sh.

make(1): inline and remove LstNode_Prev and LstNode_Next

These functions made the code larger than necessary.  The prev and next
fields are published intentionally since navigating in a doubly-linked
list is simple to do and there is no need to wrap this in a layer of
function calls, not even syntactically.  (On the execution level, the
function calls had been inlined anyway.)

make(1): replace a few Lst_ForEachUntil with simpler Lst_ForEach

make(1): clean up API for finding and creating GNodes

The previous API had complicated rules for the cases in which the single
function returned NULL or what it did.  The flags for that function were
confusing since passing TARG_NOHASH would create a new node even though
TARG_CREATE was not included in that bit mask.

Splitting the function into 3 separate functions avoids this confusion.
It also reveals several places where the complicated API led to
unreachable code.  Such code has been removed.

make(1): fix undefined behavior for .MAKEFLAGS: -f file

Since at least 1993-03-21, adding other makefiles in a .MAKEFILES
dependency has invoked undefined behavior because the command line
arguments were copied directly into the global makefiles variable,
without a proper strdup.  Shortly after that, the word list created by
Str_Words (formerly brk_string) was freed.

This applies to both the -f and the -v and -V options.  Luckily it is an
edge case to use these options in .MAKEFLAGS at all.

The -T option had already been fixed at 2000-12-30, but not the other
options.

make(1): extract parsing of the -dF option into separate function

make(1): extract InitVarTarget from main

make(1): rename Lst_ForEach to Lst_ForEachUntil

Since the callback function returns a terminating condition, this is not
really a foreach loop.

Many of the calls to Lst_ForEachUntil don't make use of the terminating
condition, and several don't modify the list structurally, which means
they don't need this complicated implementation.

In a follow-up commit, Lst_ForEach will be added back with a much
simpler implementation that iterates over the list naively, without a
terminating condition and without taking the iteration state from
Lst_Open/Lst_Next/Lst_Close into account.  The migration to this simpler
implementation will be done step by step since each callback function
needs to be examined closely.

make(1): prepare Var_Subst for proper error handling

Returning a VarParseResult instead of a string makes it possible to let
the error bubble up, until it reaches the main expression.

make(1): fix unnecessary length limitation for -v option

make(1): fix type of global variable 'create'

make(1): use fine-grained type names for lists and their nodes

This is only intended to help the human reader.  There is no additional
type safety yet.

make(1): clean up RCSID blocks

These blocks mostly consisted of redundant structure, following the same
#ifndef pattern over and over, with only minimal variation.

It's easier to maintain if the common structure is only written once and
encapsulated in a macro.

To avoid "defined but unused" warnings from GCC in the case where
MAKE_NATIVE is not defined, I had to add volatile.  Adding
MAKE_ATTR_UNUSED alone would not preserve the rcsid variable in the
resulting binary.

make(1): remove #endif comments for very short blocks

make(1): remove redundant VARARGS comments

These had been necessary at a time when the functions were declared with
"unknown parameter types".

On 1994-03-05, conditional support for function prototypes was added,
for those compilers that already supported this feature.

On 2002-06-15, the prototypes were made mandatory, and since then the
VARARGS comments had been completely redundant, or worse, simply wrong
(for ParseVErrorInternal).

make(1): consense documentation for Fatal

make(1): move PrintAddr to where it belongs

make(1): add wrappers around ctype.h functions

This avoids casting the argument to unsigned char, and to cast the
result of toupper/tolower back to char.

make(1): replace brk_string with Str_Words

The API is much simpler, and there is less detail that is exposed by
default and fewer punctuation to type on the caller's side.  To see that
there is some memory to be freed, one would have to look into the
struct.  Having part of the return value as the actual return value and
the rest in output parameters was unnecessarily asymmetrical.

make(1): rename Lst_Datum to LstNode_Datum

make(1): trust that Var_Subst never returns NULL

It really never does, and it doesn't even report errors.  It just
returns the content of the buffer, up to the first parse error.

make(1): make getBoolean simpler

This function is so seldom used that there is no point having
complicated code for it.

make(1): rename LstNode functions to match their type

make(1): rename Lst_FindB back to Lst_Find

The migration from "comparison function" to "match function" is done,
the "B" in the names is no longer needed.

make(1): start replacing Lst_Find with Lst_FindB

Lst_Find is called with a "comparison" function that returns the integer
0 if the desired node is found.  This leads to confusion since there are
so many different return value conventions for int, such as 0/1 for
mimicking false/true, -1/0 as in close(2), and the sign as in strcmp(3).
This API is much easier to understand if the "comparison" function is
not called a comparison function (since that is too close to strcmp),
but a "match" function that just returns a boolean.

In Lst_FindFromB, the node argument may be null.  This deviates from the
other Lst functions, which require Lst and LstNode to generally be
non-null.  In this case it is useful though to make the calling code
simpler.

In arch.c, this makes a lot of the previous documentation redundant.

In cond.c, the documentation is reduced a little bit since it had
already been cleaned up before.  It also removes the strange negation
from CondFindStrMatch.

In dir.c, the documentation collapses as well.

In main.c, separating the ReadMakefile function from the callbacks for
Lst_FindB allows the former to get back its natural function signature,
with proper types and no unused parameters.

To catch any accidental mistakes during the migration from Lst_Find to
Lst_FindB, the code can be compiled with -DUSE_DOUBLE_BOOLEAN, which
will complain about incompatible function pointer types.

make(1): fix null pointer dereference when sys.mk is not found

This is quite hard to trigger in a real-life scenario since it requires
precise timing.

Instead, I created /tmp/sys.mk and ran "./make -m /tmp -f /dev/null" in
the debugger, after setting a breakpoint in this line:

ln = Lst_Find(sysMkPath, ReadMakefile, NULL);

Once this line was reached, I removed /tmp/sys.mk to make ReadMakefile
fail.  Just adding a few parse errors won't help since ReadMakefile only
fails if the file cannot be found.

make(1): allow for strict type checking for Boolean

Having Boolean aliased to int creates ambiguities since int is widely
used.  Allow to occasionally compile make with -DUSE_DOUBLE_BOOLEAN to
check that the type definitions still agree.

make(1): clean up code in make.c

Var_Subst never returns NULL.

In Main_ExportMAKEFLAGS, don't compare ints with booleans.

In MainParseArgs, use char for the current character.  First, that's
more precise and correct, and second, it makes debugging easier for
those who don't know the ASCII table by heart.

make(1): rename confusing function ReadAllMakefiles

The old name implied that the function would read multiple files, which
was not the case.

The comment above that function was highly confusing.  It's not that the
function returns a boolean, but rather 0 or non-zero, and 0 means that
Lst_Find should stop searching.

One of the next refactorings will be to make Lst_Find return the first
list node for which the function returns TRUE.  This will reduce the
confusion about the several functions called SomethingP in suff.c.  The
P suffix means to return TRUE or FALSE, not 0 or non-zero.

make(1): remove trailing 'S' from names of Lst functions

The migration from null-passing Lst functions to argument-checking Lst
functions is completed.

There were 2 surprises: The targets list may be NULL, and in Dir_AddDir,
the path may be NULL.  The latter case is especially surprising since
that function turns into an almost-nop in that case.  This is another
case where probably 2 independent functions have been squeezed into a
single function.  This may be improved in a follow-up commit.

All other lists were fine.  They were always defined and thus didn't
need much work.

make(1): migrate Lst_Find to Lst_FindS

make(1): migrate Lst_First to Lst_FirstS

make(1): migrate Lst_IsEmpty to Lst_IsEmptyS

make(1): migrate Lst_Succ to Lst_SuccS

make(1): migrate Lst_ForEach to Lst_ForEachS

Most lists are always valid.  Only the "targets" variable may be null in
some cases, probably.

make(1): add stricter variants for remaining Lst functions

In most cases the Lst functions are only called when the arguments are
indeed valid.  It's not guaranteed though, therefore each function call
needs to be analyzed and converted individually.

While here, remove a few statements that were only useful when the Lst
functions handled circular lists.

make(1): fix obvious bugs in -DCLEANUP mode

The outdated type name FreeProc had been renamed to LstFreeProc.
Casting the function free to it is not necessary since the type of this
function is already exactly the correct type.  Anything else would be
undefined behavior anyway.

The uninitialized sufflist in Suff_ClearSuffixes was ok until now
because the Lst functions had silently skipped any calls with invalid
arguments.  This silent skipping is a good argument to have strict
argument validation since it detects these unintended control flows.

make(1): make brk_string return size_t for the number of words

make(1): reverse order of the Lst_Find parameters

The other callbacks all have (function, param), only the Lst_Find had
(param, function), which was inconsistent.

make(1): remove unused variable jobServer

make(1): fix indentation

make(1): replace Lst_Datum with non-null guaranteeing Lst_DatumS

make(1): replace "(void)Lst_AtEnd" with stricter "Lst_AppendS"

This change ensures that there is actually something added to the list.
Lst_AtEnd had silently skipped the addition if the list was invalid
(null pointer), which was not intended in these cases.  The "(void)" is
assumed to mean "I know that this cannot fail", while it could also mean
"I don't care whether something actually happened".

Running "./build.sh -j6 tools" still succeeds after this change,
therefore chances are very low that this change breaks anything.  If
there is any change, it's an obvious assertion failure.  There is no
silent change in behavior though.

make(1): split Dir_Init into two functions

There's just no point in having a function consisting of a big
if-then-else.

make(1): remove unused code for circular lists

The list library had probably been imported from a general-purpose
library that also supported circular lists.  These are not used by make
though.

After replacing Lst_Init(FALSE) with Lst_Init(), only a single call to
Lst_Init remained with a non-constant argument, and that was in
Lst_Concat, which was to be expected.

make(1): replace snprintf/malloc in ReadMakefile with str_concat3

make(1): replace str_concat with str_concat2 and str_concat3

The new functions have a simpler interface, and str_concat3 is even more
general-purpose, since the middle string is no longer required to be
exactly of length 1.

make(1): clean up indentation, includes, add documentation

make(1): remove outdated comment

make(1): avoid undefined behavior in Cmd_Exec

Iterating the command output backwards was dangerous since at the end,
the pointer cp pointed outside of the array.  Even without dereferencing
this pointer, this already invokes undefined behavior (C11, 6.5.6p8).
Don't risk anything.  Iterating forwards is probably faster anyway, since
it is more common.

make(1): remove redundant assignment from Cmd_Exec

A Buffer is always null-terminated.

make(1): split local variable in Cmd_Exec into two

This avoids a mismatch between signed and unsigned types.

make(1): remove trailing Z from buffer functions

This Z had been useful during the migration from int to size_t.  This
migration is finished, at least for the Buffer type, so the Z is no
longer necessary.

make(1): no declaration-after-statement anymore

NetBSD make is intended to be maximally portable, therefore it uses only
C89.  This was not declared in the Makefile before.

There are still a few places in parse.c and metachar.c that use
end-of-line comments.  These will be fixed in a follow-up commit.

make(1): replace one instance of strncpy with snprintf

GCC 9 incorrectly claims that the string might not be null-terminated.
Since objdir is a global variable, it is initialized to zero, and the +1
in the size guarantees that this byte is always 0.

Still, using strncpy to initialize a string is a waste of memory access,
since it is enough if only the actual data is copied, without zeroing
out all the remaining bytes.

make(1): switch Buffer size from int to size_t

This change helps to make the various integer types compatible and is a
preparational step for setting WARNS=6 in the Makefile.

The documentation of buf.c has been cleaned up and condensed since it
was mostly redundant, and some statements were even slightly wrong.

All code changes are covered by the existing unit tests, except for the
few lines in for.c around for_var_len.  These changes have been reviewed
thoroughly and manually, like all the others in this commit.

Those buffer functions that deal with sizes have been renamed by
appending a Z, to make sure that no function call was accidentally
forgotten.  They will be renamed back in a follow-up commit.

As usual, the scope of a few affected variables has been reduced, and
some variables had to be split since they had been incorrectly merged
before.

The order of the arguments to Buf_AddBytes has changed from (mem_len,
mem) to (mem, mem_len), in order to make it consistent with the
functions from the C standard library, such as snprintf.

make(1): fix parameter name in Cmd_Exec

A format string is not a number.

make(1): use ordinary string concatenation in usage text

make(1): use consistent indentation in source code

Tabs for multiples of 8, then spaces.

The usage string has been kept as-is since the spaces there are
indentional and do influence the output.

make(1): avoid calls to free(3) in the common case of a NULL pointer

make(1): let Var_Value return a const char *

The return value must not be modified anyway, so let the compiler check
this for free.

make(1): make Main_SetVarObjdir const-correct and use separate variables

The return value of Var_Value should be const char *, but changing that
now would be too large a change.

make(1): merge duplicate code for concatenating strings

make(1): eliminate unsatisfiable condition in is_relpath

Add -dL for LINT

When parsing variable assignments other than := and if
value contains '$' attempt Var_Subst the same as for :=,
if the value does not parse correctly, we get a fatal error
including file an line number.

This can greatly help with finding the cause of problems.

Reviewed by: christos

make(1): remove dead code from Var_Subst

The first parameter from Var_Subst had been a literal NULL in all cases.
These have been fixed using this command:

sed -i 's|Var_Subst(NULL, |Var_Subst(|' *.c

The one remaining case was not found because the "NULL," was followed by
a line break instead of a space.

The removed code probably wouldn't have worked as expected anyway.
Expanding a single variable to a literal string would have led to
unexpected behavior for cases like ${VAR:M${pattern}}, in case pattern
would contain an unescaped ':' itself.

Make DEBUG_HASH less of a fire-hose.

Reporting keys on every lookup is overkill unless
playing with a new HASH, so wrap in #ifdef DEBUG_HASH_LOOKUP
Also add some stats at the end so we can see
final size and max chain length - maxchain is a better
variable name than maxlen.

make(1): clean up unnecessary snprintf and multi-line function calls

make(1): rename Varf_Flags to VarEvalFlags

In var.c there are lots of different flag types.  To make any accidental
mixture obvious, each flag group gets its own prefix.

The only flag group that is visible outside of var.c is concerned with
evaluating variables, therefore the "e", which replaces the former "f"
that probably just meant "flag".

Add -dh for DEBUG_HASH

Allow tracking of max chain length, to see how well the hash
tables are working.
Pull the actual hash operation into a marco so it can be
easily changed - for experimenting.

The current hash, is pretty good.

Reviewed by: christos

make(1): remove trailing whitespace

make(1): remove redundant parentheses around return values

make(1): remove useless parameter from Var_Set

The enum corresponding to this int parameter is only defined in var.c,
which makes it impractical for the outside to set this parameter to
anything but 0.

On x86_64, this reduces the size of the resulting executable by 5 kB.

dieQuietly: ignore OP_SUBMAKE

Stick to OP_MAKE for deciding when to supress noise
as OP_SUBMAKE can be too aggressive.

Reviewed by: gson

Avoid unnecessary noise when sub-make or sibling dies

When analyzing a build log, the first 'stopped' output
from make, is the end of interesting output.

Normally when a build fails deep down in a parallel build
the log ends with many blockes of error output from make,
with all but the fist being unhelpful.

We add a function dieQuietly() which will return true
if we should supress the error output from make.
If the failing node was a sub-make, we want to die quietly.

Also when we read an abort token we call dieQuietly telling we
want to die quietly.

This behavior is suppressed by -dj or
setting .MAKE.DIE_QUIETLY=no

Reviewed by: christos

Merge changes from current as of 20200406

make: fix -fno-common build

debug was declared extern, but debug_file was not; correct this and define
debug_file in main.c (as debug is) to fix the -fno-common build.

-fno-common will become the default with GCC10/LLVM11.

Patch from kevans at freebsd

Ignore empty MAKEOBJDIR

Otherwise we end up with .OBJDIR = ${.CURDIR}/
which is quivalent, but fails the typial
.if ${.OBJDIR} == ${.CURDIR}

Pull up following revision(s) (requested by jmcneill in ticket #113):
	usr.bin/make/main.c: 1.266-1.272
	usr.bin/make/make.1: revision 1.267
move some code out of the gigantic main function; no functional change.
--
-V: try to expand the variable again if the value contains a variable.
--
simplify
--
a variable that starts with \\ is not expanded.
--
Remove previous variable expansion code; sjg had already added the code to
do it. Note that the manual page already documents this behavior and does
not need to change:
        -dV -V VAR:     prints the raw variable
        -V VAR:         prints the expanded variable
--
make the code look like to 1.266
--
Add -v variable that always expands variables; restore -V the way it was.

Add -v variable that always expands variables; restore -V the way it was.

make the code look like to 1.266

Remove previous variable expansion code; sjg had already added the code to
do it. Note that the manual page already documents this behavior and does
not need to change:

	-dV -V VAR:	prints the raw variable
	-V VAR:		prints the expanded variable

a variable that starts with \\ is not expanded.

simplify

-V: try to expand the variable again if the value contains a variable.

move some code out of the gigantic main function; no functional change.

Sync with HEAD

Main_SetObjdir: ensure buf2 is in scope

Sync with HEAD

Sync with HEAD

We cannot tollerate things like trailing /.. etc in .CURDIR
so only accept -C arg "as is" if it contains no relative components.

We cannot use buf as both src and destination for snprintf.

Factor out parts of cached_realpath.

This wacky API confused Coverity (CID 1405167) and is no good for
human readers either.

Use bmake_malloc where we don't test the return value of malloc currently.

Appeases coverity.

PR/52159: Jamie Landeg-Jones: Don't try to expand $'s in paths or system
variables. Only expand them ${MAKE} variable values.

When we change .OBJDIR we should delete any cached realpath results
for relative paths.

Reviewed by: christos

Sync with HEAD

Do not consider using -C arg "as is" for .CURDIR unless it starts with '/'.

Reviewed by: christos

Use strncpy here because we need to be portable, and we have the +1 byte
to save us :-)

use the logical path for -C if correct.

Partially initialize Dir before MainParseArgs can be called.

The rest can be done once curdir is finalized.

Sync with HEAD.  (Note that most of these changes are simply $NetBSD$
tag issues.)

Back in 1993 ${MACHINE} was a unique way to define objdirs for
every platform. Thus setting OBJMACHINE, allowed one to keep
in-tree objdirs with multiple builds. With the advent of evb*, this
has become impossible until now. Introduce OBJMACHINE_ARCH that
adds ${MACHINE_ARCH} to the objdir so that we have unique objdirs
per build again. Until we restructure things to that this is not
necessary, it is the simplest fix.

Don't apply the suffix (curdir) to MAKEOBJDIR

Refactor and simplify objdir setting code.