CVS log for src/usr.bin/make/meta.c

Up to [cvs.NetBSD.org] / src / usr.bin / make

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

make: sync VarEvalMode constant names with their debug log names

make: replace magic numbers with POSIX FILENO constants

No binary change.

make: simplify freeing of lists

make: clean up unused return value of str2Lst_Append

No functional change.

meta_ignore: check metaIgnorePaths on raw path

Some path prefixes can trigger automount, if we want to ignore them
it might be best to check metaIgnorePaths before we call realpath.
If the raw path does not match, check again after realpath.

make: declare all common symbols in headers, unexport others

No functional change.

Use inline function rather that macro BM

meta.c: use macro to access job->bm

and if job is NULL use Mybm.

No functional change.

make: clean up calls to Var_Subst

None of the calls to Var_Subst used the return value, and the return
value was always VPR_OK.

No functional change.

Don't ignore return from snprintf or getcwd

make: do not use __CONCAT and __STRING in meta_needed

Using __CONCAT for identifiers makes it impossible to find their usages
easily, as could be seen in targ.c 1.22 from 2001-07-03.  It also
requires that the parts of the identifier are not macros themselves.

No binary change.

Fix some unused warnings

make: remove redundant conditions

A line starting with '======' is parsed as a variable assignment, with
an empty variable name and a value starting with '=====', making that
condition unreachable.

No functional change.

Use a variable to store command line to be filtered.

This avoids any nasty side effects from content of command line.

Reviewed by: rillig

make: use unsigned int for line numbers everywhere

Previously, some line numbers were stored as signed int while others
were stored as size_t.  Since line numbers are never negative, use an
unsigned type.  Since the maximum file size for makefiles is 1 GB (see
loadfile), unsigned int is large enough even on 64-bit platforms.

Using a single data types reduces the number of type conversions.  Using
unsigned int improves compatibility with C90 (printf %u instead of %zu),
which is needed by bmake, which is derived from usr.bin/make.

No functional change.

Allow local variable assignments in dependency lines

The variable is set in the context of the target.
This syntax has been supported by gmake for ~ever.
If necessary a makefile can set .MAKE.TARGET_LOCAL_VARIABLES=false
to disable this.

Expose GetBooleanExpr so parse.c can use it.

make: clean up eat_dots for meta mode, part 2

At least on OpenBSD, "If the src and dst strings overlap, the behavior
is undefined" (since 2013-09-25), so rather use memmove instead.

Keep all other bugs for now, for example "/a/b//../" is reduced to
"/a/b/" instead of the correct "/a/", and for repeated "/././././" in
long paths, the performance is quadratic instead of linear.

Inline the two calls to eat_dots, turning the first into a two-liner.

make: clean up eat_dots for meta mode

No binary change.

make: replace Var_Value with Var_Exists where applicable

The latter function already existed in 1993, no idea why it was not
used.

No functional change.

make: remove unnecessary functions for expanding variable names

In meta mode, the affected variable patterns do not contain a '$'.

Outside of meta mode, Global_SetExpand was only called a single time, so
inline that call.

No functional change.

make: merge duplicate code for expanding variable expressions

No functional change.

make: remove trailing whitespace in meta.c

meta_cmd_cmp: use bool for once

Just use Buf_Init() some mallocs will be inevitable.

meta.c: add .MAKE.META.CMP_FILTER

On rare occasions it is useful to be able to filter command lines
before comparison.

make: fix indentation in meta.c

make: remove CONSTCOND comments, lint no longer needs them

Do not ignore write failures.

We should not ignore failure to write to cmdFILE,
meta files and filemon.

Reviewed by: rillig

make: fix lint warning about strrchr discarding 'const'

No functional change.

make: fix lint warning about strchr removing 'const'

No functional change.

make: rename a few functions to be more descriptive

No functional change.

make: use C99 bool type instead of defining its own

No functional change.

make: replace enum bit-field with struct bit-field for VarEvalFlags

This makes the code easier to read, especially in var.c.  It also makes
debugging sessions easier since some debuggers don't show enum
bit-fields symbolically as soon as more than one bit is set.

The code outside var.c is basically unchanged, except that instead of
passing the individual flags, there are 4 predefined evaluation modes.
These suffice for all practical use cases.  Only in the implementation
deep inside var.c, the value of the flags keepDollar and keepUndef
differs.

There is no way of passing the struct to EnumFlags_ToString, which means
the ToString function has to be spelled out explicitly.  This allows for
fine-tuning the representation in the debug log, to reduce the amount of
uppercae letters.

No functional change.

make: remove freestanding freeIt variables

These variables all belong to a string variable.  Connect them using
FStr, which reduces the number of variables to keep track of.

No functional change.

Avoid strdup in mkTempFile

Require caller to pass a buffer and size if they
want the tempfile not unlinked.

Add Job_TempFile to handle blocking signals around
call to mkTempFile, so that meta_open_filemon can use it
in jobs mode.

make: in the Var_ functions, move the scope to the front

This change provides for a more natural reading order in the code.
Placing the scope first makes it immediately clear in which context the
remaining parameters are interpreted.

No functional change.

make: rename some VAR constants to SCOPE

The word "context" does not fit perfectly to the variables that are
associate with a GNode, as the context is usually something from the
outside and the variables are more like properties inherent to the
GNode.

The term "global context" fits even less.  Since the thing where
variables are looked up is commonly named a scope, use that term
instead.

This commit only renames the global variables VAR_GLOBAL, VAR_INTERNAL
and VAR_CMDLINE, plus a few very closely related comments.  These are:

	GNode.vars (because of line breaks)
	GNode_Free (dito)
	varname-make_print_var_on_error.mk
	varname-make_print_var_on_error-jobs.mk

The debug message in Var_Stats is left as-is since there is no unit test
for it yet.

The other renamings (variable names "context", "ctxt", as well as
further comments) will be done in a follow-up commit.

make: rename Var_Delete to Var_DeleteExpand, Var_DeleteVar to Var_Delete

The function names now follow the naming scheme from the other functions
that handle variables.

There are several calls that remain syntactically unchanged but that
omit the call to strchr('$') now.  Since all these calls use constant
variable names, there is no functional change.

make: replace Global_AppendExpand with Global_Append

All callers with a variable name that is guaranteed to not contain a
dollar sign have been converted to call Global_Append instead of the
previous Global_AppendExpand.  After that, Global_AppendExpand was
unused, therefore it was effectively just renamed.

make: replace Global_SetExpand with Global_Set for constant names

make: use shortcut functions Global_SetExpand and Global_AppendExpand

There are many places where global variables are set or appended to.  To
reduce clutter and code size, encode the VAR_GLOBAL in the function
name.

The word Expand in the function names says that the variable name is
expanded.  In most of the cases, this is not necessary, but there are no
corresponding functions Global_Set or Global_Append yet.

Encoding the information whether the name is expanded or not in the
function name will make inconsistencies obvious in future manual code
reviews. Letting the compiler check this by using different types for
unexpanded and expanded variable names is probably not worth the effort.
There are still a few bugs to be fixed, such as in SetVar, which expands
the variable name twice in a row.

make: clean up comments

make(1): fix a few inconsistencies for lint's strict bool mode

make(1): consistently use boolean expressions in conditions

Most of the make code already followed the style of explicitly writing
(ptr != NULL) instead of the shorter (ptr) in conditions.

The remaining 50 instances have been found by an experimental,
unpublished check in lint(1) that treats bool expressions as
incompatible to any other scalar type, just as in Java, C#, Pascal and
several other languages.

The only unsafe operation on Boolean that is left over is (flags &
FLAG), for an enum implementing a bit set.  If Boolean is an ordinary
integer type (the default), some high bits may get lost.  But if Boolean
is the same as _Bool (by compiling with -DUSE_C99_BOOLEAN), C99 6.3.1.2
defines that a conversion from any scalar to the type _Bool acts as a
comparison to 0, which cannot lose any bits.

make(1): fix lint warnings

make(1): replace pointers in controlling conditions with booleans

make(1): fix return type of macro DEBUG

This macro was supposed to return a boolean expression all the time, it
just hadn't been implemented this way.  This resulted in wrong output
for the test sh-flags, in compilation modes -DUSE_UCHAR_BOOLEAN and
-DUSE_CHAR_BOOLEAN, since in ParseCommandFlags, the expression
DEBUG(LOUD) didn't fit into a boolean.

make(1): fix undefined behavior in meta_oodate

Do not increment a null pointer.

Do not assign to a variable twice in the same statement.  To be fair,
this may be safe because of the sequence point when the function is
called, but anyway, it looks too close to undefined behavior.

make(1): reduce scope of cp in meta_oodate

That function is way too long (550 lines) and has too creative variable
names such as fname, fname1, fname2, fname3.

make(1): fix declared variable type in printCMDs

No functional changes since StringListNode and GNodeListNode are both
typedefs of a simple generic ListNode.

make(1): change return type of Var_Value to FStr

make(1): replace %zu with %u in printf calls

This is needed to compile bmake with GCC 2.8.1 on SunOS 5.9.

To support ancient systems like this, the whole code of usr.bin/make is
supposed to use only ISO C90 features, except for filemon, which is not
used on these systems.

make(1): add str_basename to reduce duplicate code

The function basename from POSIX has a few unfortunate properties, it is
allowed to return a pointer to static memory.  This is too unreliable,
therefore this trivial own implementation.

make(1): split JobFlags into separate fields

Having all these flags in a single bitmask makes it harder to see where
exactly they can possibly be used since their state could also be
modified using the unsuspicious job->flags = 0.  Using individual names
just leaves the single memset, and that is only used during
initialization.

make(1): remove unused parameter from meta_needed

make(1): reduce memory allocation in meta_oodate

make(1): reduce memory allocation in meta mode

make(1): use space instead of tab for preprocessor directives

make(1): remove pointer indirection from GNode.commands

Just to save a few memory allocations.  No noticeable effect on the
performance though.

make(1): rename parameter in meta_needed and meta_create

It conflicts with the global variable in dir.c when make is built in
all-in-one mode.

make(1): fix local variable name in meta_create

make(1): inline meta_file_t for printCMD

This struct had only been necessary as long as printCMD was a callback
to Lst_ForEach or Lst_ForEachUntil.

make(1): inline Lst_ForEachUntil in meta mode

This means no more unnecessary void pointers in function signatures and
no more abstraction level at checking a single element of a list.  In
most cases it is more appropriate to define a function that operates on
the list as a whole, thereby hiding implementation details like the
ListNode from the caller.

make(1): remove redundant null check from meta_needed

The result of bmake_realloc can never be NULL.

make(1): use comparisons in boolean expressions

The generated code stays exactly the same.

make(1): use properly typed comparisons in boolean contexts

make(1): align end-of-line comments with tabs

make(1): clean up make.h, meta.c, parse.c, str.c

The main changes are in the comments, which have been shortened and
corrected.

Some local variables changed their names.

In ParseErrorInternal, the scope of va_start is now narrower.

In ParseDoDependency, the type of tOp has been fixed.

ParseGetLine doesn't take flags anymore but instead a parsing mode.
Previously, the flags had not been combined anyway.

At the beginning of Parse_File, fatals is already guaranteed to be 0, and
even if not, it would be wrong to just discard the fatal errors.

make(1): remove redundant struct make_stat

In the cache for stat(2) and lstat(2), only one of the two timestamps
was ever used.  To prevent a result from stat(2) leaking into the cache
for lstat(2), there have been two completely separate caches all the
time.  Using different fields in the struct was therefore unnecessary.

By removing the redundant field, the internal struct in the cache is the
same as the external struct.  This makes one of them redundant, thus
struct make_stat has been renamed to cached_stat, which better describes
its purpose, and the internal struct cache_st has been removed.

Just as before, the cache prevents any direct access to its internal
data.  When passing it to the caller, it is copied.

Just as before, the field names of struct cached_stat cannot correspond
to those from struct stat, since the latter are often defined as macros.
Therefore they are prefixed with cst instead of st.

The redundancy had been added on 2020-06-05.

make(1): remove redundant parameter from str2Lst_Append

make(1): use strict typing in conditions of the form !var

make(1): fix inconsistent indentation after #ifdef

Combining #ifdef with regular if-then-else calls for trouble.

make(1): clean up code stylistically

* Replace character literal 0 with '\0'.
* Replace pointer literal 0 with NULL.
* Remove redundant parentheses.
* Parentheses in multi-line conditions are not redundant at the
  beginning of a line.
* Replace a few !ptr with ptr == NULL.
* Replace a few ptr with ptr != NULL.
* Replace (expr & mask) == 0 with !(expr & mask).
* Remove redundant braces for blocks in cases where the generated code
  stays the same.  (Assertions further down in the code would get
  different line numbers.)
* Rename parameters in CondParser_String to reflect the data flow.
* Replace #ifdef notdef with #if 0.

The generated code stays exactly the same, at least with GCC 5.5.0 on
NetBSD 8.0 amd64 using the default configuration.

make(1): remove redundant parentheses from sizeof operator

The parentheses are only needed if the argument is a type, not an
expression.

make(1): remove mistyped extern variable

Robust programs don't have extern variable declarations in .c files, as
that risks incomatible definitions that are not detected by the compiler
and invoke undefined behavior.  Make make a little more robust.

make(1): make memory allocation simpler in meta_create and meta_oodate

Since there is only a single variable left that needs to be freed at the
end (and probably never actually needs to be freed since nobody defines
an environment variable named .OBJDIR), there is no need to loop over
these variables.

make(1): do not look up local variables like .TARGET anywhere else

Nobody defines a global variable named .TARGET since that would have
many unpredictable effects, applying to all targets at once.

Nobody defines an environment variable named .TARGET since that's
against the naming conventions for environment variables and would have
the same effect.

Because of this, there is no point looking up the variables that are
local to a GNode anywhere else.  This means they cannot come from the
environment and thus their value doesn't need to be freed after use,
which makes the code simpler.

The newly added accessor functions in make.h refer to external
functions, but since that header is not used anywhere outside of
usr.bin/make, it doesn't matter.  Between 2020-08-25 and 2020-10-30,
that header had been referenced by usr.bin/xinstall.

make(1): fix indentation in source code

make(1): change char * to void * in Var_Value

The only purpose of the parameter freeIt is to free the memory
associated with the return value.  To do this, no pointer arithmetic is
needed.  Therefore, change to a void pointer, to catch accidental use of
that pointer.

make(1): fix indentation in source code

make(1): add GNode_Path to access the path of a GNode

make(1): inline Lst_Find in meta_oodate

make(1): make Lst_RemoveIf simpler in meta_oodate

Using Lst_Find and Lst_FindFrom to implement Lst_RemoveIf was a bad
idea.  It made the code much more complicated than necessary.  There is
no predefined Lst_RemoveIf, but that can be implemented easily.  By
inlining the list handling, path_match does not need void pointers
anymore.

Freeing the path from the missingFiles list had been implemented in a
surprisingly complicated way, intermangling it unnecessarily with the
list operations, even though these are completely independent.

make(1): remove redundant parameter from meta_name

make(1): inline simple Lst getters

The function call variant takes more screen space than the direct field
access.  Having an abstract API is usually a good idea, in this case of
simple read-only member access it makes the code more difficult to read.

LstNode_Set has been kept as a function since it is not a read-only
accessor function.

make(1): remove void pointers from printCMD

make(1): rename Lst_Init to Lst_New

For the other types such as HashTable and Buffer, the Init function does
not allocate the memory for the structure itself, it only fills it.

make(1): prepare for WARNS=6

The FD_* macros from sys/sys/fd_set.h use signed integers on NetBSD 8
and thus produce conversion errors.  On NetBSD 9, these macros are fixed
to use 1U instead of 1.

make(1): replace execError with execDie

All calls to this function were followed by _exit(1).

make(1): make debug logging simpler

This avoids referring to the debug_file variable in many places where
this implementation detail is not necessary.

make(1): make debugging code shorter

make(1): inline and remove LstNode_Prev and LstNode_Next

These functions made the code larger than necessary.  The prev and next
fields are published intentionally since navigating in a doubly-linked
list is simple to do and there is no need to wrap this in a layer of
function calls, not even syntactically.  (On the execution level, the
function calls had been inlined anyway.)

make(1): migrate printCMD in meta mode to Lst_ForEach

make(1): rename Lst_ForEach to Lst_ForEachUntil

Since the callback function returns a terminating condition, this is not
really a foreach loop.

Many of the calls to Lst_ForEachUntil don't make use of the terminating
condition, and several don't modify the list structurally, which means
they don't need this complicated implementation.

In a follow-up commit, Lst_ForEach will be added back with a much
simpler implementation that iterates over the list naively, without a
terminating condition and without taking the iteration state from
Lst_Open/Lst_Next/Lst_Close into account.  The migration to this simpler
implementation will be done step by step since each callback function
needs to be examined closely.

make(1): prepare Var_Subst for proper error handling

Returning a VarParseResult instead of a string makes it possible to let
the error bubble up, until it reaches the main expression.

make(1): use fine-grained type names for lists and their nodes

This is only intended to help the human reader.  There is no additional
type safety yet.

make(1): add specific typedefs for lists

These typedefs are only intended to help human readers, they do not
provide any type-safety.  They also make the pointers explicit, which
had been hidden before by the typedef for Lst and LstNode.  Typing a few
'*' is less work than finding out which of the many types are pointers
and which aren't.

In meta.c, the variable "ln" served two completely different purposes,
which have been split again.  Register allocation is the job of the
compiler, not of the human source code reader.

make(1): fix inconsistent code indentation

make(1): reduce number of stat fields returned by cached_stat

Only st_mtime and st_mode are actually filled, the remaining fields had
been set to zero.  To prevent these from ever being accessed, a custom
struct make_stat replaces the previously used struct stat.

The fields in struct make_stat are intentionally named different from
the fields in struct stat because NetBSD and some other operating
systems define st_mtime as a macro, and that would not work in a field
declaration.

make(1): rename Lst_Datum to LstNode_Datum

make(1): trust that Var_Subst never returns NULL

It really never does, and it doesn't even report errors.  It just
returns the content of the buffer, up to the first parse error.

make(1): rename LstNode functions to match their type

make(1): rename Lst_FindB back to Lst_Find

The migration from "comparison function" to "match function" is done,
the "B" in the names is no longer needed.

make(1): migrate remaining Lst_Find to Lst_FindB

While here, rename SuffSuffIsSuffix to SuffSuffGetSuffix since a
function named "is" should return a boolean, not a string pointer.

make(1): fix the other assertion from Lst_FindFrom

When I migrated the Lst_FindFrom to the strict API variant, I forgot
that Lst_FindFrom requires both arguments (list and node) to be
non-null.  I had only checked that the list is non-null.

There are only very few calls to Lst_FindFrom, and they are all ok now.

make(1): remove trailing 'S' from names of Lst functions

The migration from null-passing Lst functions to argument-checking Lst
functions is completed.

There were 2 surprises: The targets list may be NULL, and in Dir_AddDir,
the path may be NULL.  The latter case is especially surprising since
that function turns into an almost-nop in that case.  This is another
case where probably 2 independent functions have been squeezed into a
single function.  This may be improved in a follow-up commit.

All other lists were fine.  They were always defined and thus didn't
need much work.

make(1): migrate Lst_Find to Lst_FindS

make(1): migrate Lst_First to Lst_FirstS

make(1): migrate Lst_IsEmpty to Lst_IsEmptyS

make(1): migrate Lst_Succ to Lst_SuccS

make(1): migrate Lst_ForEach to Lst_ForEachS

Most lists are always valid.  Only the "targets" variable may be null in
some cases, probably.

make(1): add stricter variants for remaining Lst functions

In most cases the Lst functions are only called when the arguments are
indeed valid.  It's not guaranteed though, therefore each function call
needs to be analyzed and converted individually.

While here, remove a few statements that were only useful when the Lst
functions handled circular lists.

make(1): reverse order of the Lst_Find parameters

The other callbacks all have (function, param), only the Lst_Find had
(param, function), which was inconsistent.

make(1): define aliases for function types in list processing

This makes the prototypes of the functions clearer.

make(1): fix indentation

make(1): replace Lst_Datum with non-null guaranteeing Lst_DatumS

make(1): convert remaining Lst_AtEnd to the stricter Lst_Append

The general-purpose list library that is included in make allows to call
Lst_AtEnd for invalid lists, silently ignoring this programming error.
This is a flexibility that make doesn't need.

Another unneeded "feature" is that list items can theoretically be null
pointers.  This doesn't make sense as well and is therefore not needed
by make.

These programming errors are now caught early by assertions.

make(1): make list library code stricter

Up to now, the list library didn't distinguish between programming
mistakes (violations of invariants, illegal parameter values) and
actually interesting situations like "element not found in list".

The current code contains many branches for conditions that are neither
exercised by the unit tests nor by real-world usage.  There is no point
in keeping this unnecessary code.

The list functions will be migrated from their lenient variants to the
stricter variants in small parts, each function getting the S suffix
when it is made strict, to avoid any confusion about how strict a
particular function is.  When all functions have been migrated, they
will be renamed back to their original names.

While here, the comments of the functions are cleaned up since they
mention irrelevant implementation details in the API comments, as well
as "side effects" that are really main effects.

make(1): remove unused code for circular lists

The list library had probably been imported from a general-purpose
library that also supported circular lists.  These are not used by make
though.

After replacing Lst_Init(FALSE) with Lst_Init(), only a single call to
Lst_Init remained with a non-constant argument, and that was in
Lst_Concat, which was to be expected.

make(1): no declaration-after-statement anymore

NetBSD make is intended to be maximally portable, therefore it uses only
C89.  This was not declared in the Makefile before.

There are still a few places in parse.c and metachar.c that use
end-of-line comments.  These will be fixed in a follow-up commit.

make(1): avoid calls to free(3) in the common case of a NULL pointer

make(1): let Var_Value return a const char *

The return value must not be modified anyway, so let the compiler check
this for free.

make(1): remove dead code from Var_Subst

The first parameter from Var_Subst had been a literal NULL in all cases.
These have been fixed using this command:

sed -i 's|Var_Subst(NULL, |Var_Subst(|' *.c

The one remaining case was not found because the "NULL," was followed by
a line break instead of a space.

The removed code probably wouldn't have worked as expected anyway.
Expanding a single variable to a literal string would have led to
unexpected behavior for cases like ${VAR:M${pattern}}, in case pattern
would contain an unescaped ':' itself.

make(1): clean up unnecessary snprintf and multi-line function calls

make(1): rename Varf_Flags to VarEvalFlags

In var.c there are lots of different flag types.  To make any accidental
mixture obvious, each flag group gets its own prefix.

The only flag group that is visible outside of var.c is concerned with
evaluating variables, therefore the "e", which replaces the former "f"
that probably just meant "flag".

meta_oodate: if target is explicitly .META missing is oodate

If a .meta file is missing for a target marked .META
it is out-of-date

make(1): remove trailing whitespace

make(1): remove redundant parentheses around return values

make(1): remove useless parameter from Var_Set

The enum corresponding to this int parameter is only defined in var.c,
which makes it impractical for the outside to set this parameter to
anything but 0.

On x86_64, this reduces the size of the resulting executable by 5 kB.

filemon_read: report error if lseek fails

Reviewed by: riastradh

Merge changes from current as of 20200406

CHECK_VALID_META is too agressive for CMD check.

An empty CMD is perfectly valid.
So using CHECK_VALID_META results in too much rebuilding.

Check for meta file corruption earlier and more often.

meta_compat_parent check for USE_FILEMON

patch from Soeren Tempel

Fix meta_compat mode.

We need to call meta_job_output() from meta_compat_parent()
Also, the need for extra fds for meta mode only applies when
using filemon_ktrace.c.
Since we hope to see more interfaces like that, test for
USE_FILEMON_DEV being not defined.

Reviewed by: riastradh

Check for filemon valid.

If meta_needed() returns FALSE, mfp and filemon will be NULL
in which case we should not call filemon_*().

meta.c: report OODATE value if not empty

When a target is out-of-date per normal make rules
.OODATE will be non-empty, report this in .meta file
to help clarify why target was updated.

Per sjg's suggestion, split filemon API into separate back ends.

By default we use the ktrace back end, but the /dev/filemon back end
is available as a compile-time option, by setting USE_FILEMON=dev in
make.  sjg raised concerns about ktrace performance and would like to
continue using /dev/filemon on FreeBSD (which has seen more
maintenance kernel-side) without forking make.

Reimplement make(1) meta mode without filemon(4).

Revert the filemon removal in bmake, as pointed out by maya we do care
about not introducing divergence with FreeBSD, and the cost of unused
is acceptable here.

Fix the build

Retire filemon, discussed on tech-kern@.

Avoid unused variable warning

Avoid full path meta file names for subdir of .OBJDIR

Ensure that command output is always followed by newline,
even when filemon is not being used.

Tweak MAKE_META_IGNORE_PATTERNS matching to avoid using path name
with :L as it does not handle ':' in path names.

fgetLine: an extra check to avoid shrinking the buffer.

Refactor the handling of .MAKE.META.IGNORE_* to meta_ignore()
so that we can call it before adding paths to missingFiles.
Ie. do not track files we have been told to ignore.

Add .MAKE.META.IGNORE_FILTER to allow more complicated filtering of filemon data.

The filter is applied to paths Read or Excuted, and if it expands to
nothing, the entry is skipped.

For example; dirdeps.mk can set this to:

.MAKE.META.IGNORE_FILTER = M*/${.MAKE.DEPENDFILE_PREFIX}*

when checking if DIRDEPS_CACHE is up to date, where only Makefile.depend*
are of interest.

Use Lst_FindFrom and Lst_Succ(ln) to find next entry in missingFiles.

Linux filemon only records D some/dir for rm -rf
rather than D entries for each file removed.
Thus we need a loop to remove all matching missingFiles entries

meta_compat_parent: fflush each line

meta_oodate: set needOODATE if oodate due to missing .meta file.

Extend the mtimes cache used by dir.c so it can be used by others.

We store both st_mtime and st_mode, since some callers care about the
later.

Reviewed by: christos

meta_oodate even when missing-meta is true, we do not make a target in
.CURDIR out of date just because of a missing meta file.

Also we can only be missing filemon data if we had a meta file.

Add cached_realpath()

realpath(3) is expensive, and meta mode at least uses it extensively.
We use cached_realpath() to save the result of successful calls to
realpath(3) in a private variable context.

This improves the worst case performance (eg examining libc with
nothing to do) by a factor of 4.

Reviewed by: christos

Refactor the logic for deciding when a .meta file is needed.

Both meta_create() and meta_oodate() now share meta_needed()
to indicate if a .meta file is needed.

This allows for deciding that missing .meta file is cause of out-of-date.
The new knob 'missing-meta' controlls this, and is off by default
since the side effect files of generators like yacc will typically need
.NOMETA to avoid problems.

This however makes it much easier for folk to switch back and forth
with meta mode (eg. FreeBSD's normal build can leverage meta mode for
improved reliablility without need to clean).

Also if .MAKE.MODE does not contain 'nofilemon', the knob
'missing-filemon' allows control as to whether missing filemon data
should cause the target to be out-of-date.
Again off by default.

Propagate errors from filemon.

If we encounter errors producing a .meta file,
we should not consider the target completed successfully.

Allow for ignoring paths that match a set of patterns.
This can be expensive, so use with caution.

Apply realpath() to p before matching against metaIgnorePaths.

Handle missing file for Read the same as for Write, since if it
is Deleted we don't care.

Patch from Bryan Drewery.

str2Lst_Append tokenizes the string and uses it in the list so we can't
free the string afterwards. Keep a copy of it and cleanup at the end.

CID 1025009: Fix wrong code (NULL deref)

CID 1025008: Free memory returned by Var_Subst

CID 1025007: Don't leak missingFiles

CID 1025006: pbm can't be NULL at this point.

Let makefiles decide if depend file gets a meta file

Adapt to changed Var_Subst()

Actually FALSE as last arg to Var_Subst makes more sense

Adapt to new Var_Subst prototype

remove free NULL checks (Tilman Sauerbeck)

provide missing FD_CLOEXEC for the havenots.

Use FD_CLOEXEC (Tilman Sauerbeck)

Use lstat(2) if we have a link, incase it is a symlink.

Patch from bdrewery at FreeBSD.org

Add Boolean wantit to Var_Parse and Var_Subst

wantit will be FALSE when we are just consuming to discard
in which case we skip "expensive" things like Cmd_Exec.

Reviewed by: christos

Do not hard-code ingnoring of makeDependfile

Use separate vars to track cwd and last dir a process read.

Close child's reference to meta file.

Ensure -- filemon is at start of line

Use read and fwrite rather than fgets and printf when reading filemon data.

src is too big these days to tolerate superfluous apostrophes.  It's
"its", people!

Rebase to HEAD as of a few days ago.

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

meta_oodate: 'L'ink and 'M'ove entries need extra processing.
The src of a Link should be treated as a file Read
while the target a file Written.
The src of a Move is like a file Deleted
and the target like a file Written.

Add .MAKE.META.IGNORE_PATHS to facilitate local additions to the paths
which should be ignored by meta_oodate().

Ignore any files in /proc/

resync from head

New compiler doesn't like foo()

Refine the effect of .OODATE on command comparison.
Rather than apply it to the whole script, just the current command line
is affected.   This allows a trick like ${.OODATE:M.NOMETA_CMP}
to cause command comparison to be supressed for one command only.

After removing an entry from missingFiles,
set ln=NULL after Lst_Remove() to make it clear we won't use it
after this point.

If built with filemon support, set .MAKE.PATH_FILEMON to _PATH_FILEMON
so that makefiles can test for it.

resync with head

sync with head

meta_oodate: rename ignoreOODATE to needOODATE.
if target uses ${.OODATE} in its script;
1. we canno compare commands
2. if we decide it is out-of-date .OODATE must not be empty.
   The only option is to set it to .ALLSRC

sync with head

gcc 4.2 doesn't like externs inside functions

PR/45383: Henning Petersen: Swapped lseek args

meta_oodate should ignore makeDependfile

In meta mode, we create .meta files for most targets.
These capture all the interesting data - useful for debugging.
In such cases there is no need to replicate commands in the build log.
Rather than run the entire build .SILENT, allow meta mode to set that flag
per target iff a .meta file is created.
Normal behavior is retained for targets where no .meta file is created,
ensuring that no build data is lost.

If a target is explicitly flagged .META,
a missing .meta file means it is out-of-date.

Catchup with rmind-uvmplock merge.

After each strsep(&p) check that p is valid.
If not the .meta file is corrupted (build interrupted?)
issue a warning and declare target oodate.

Use %zu rather than cast

size_t on amd64 doesn't like %u, use a cast.

Add .MAKE.META.BAILIWICK - to identify the scope of make's control.
meta_oodate: if a file that was written or linked within our bailiwick,
but outside of .OBJDIR is missing, add it to missingFiles.
If we get to the end of the .meta file without seeing it [re]moved,
then consider the target out-of-date.

Call meta_job_finish() from meta_job_error(), so that if
the .ERROR target copies the .meta file, it has been completed.
meta_job_finish() is safe to call again.

Add NetBSD tags

Check for extra commands as soon as we reach CWD,
so we return faster (if oodate) and give a more accurate debug message.

Use curdirOk as the token

When .MAKE.MODE inlcudes 'meta'; 'curdir=true' enables creating .meta
files in .CURDIR - such as when running make in .OBJDIR with a generated
makefile.

Only need to getcwd() once

CWD should be compared against cwd

Sync with HEAD

main.c now exports curdir[] so we can use it.

Sync with HEAD

When making a target that involves creating itermediate dirs,
realpath() will fail until after those dirs exist.
Try to ensure the same meta file name is used regardless.

Sync with HEAD

tmplen should be static

Invalidate meta_fname when we finish a job, so that in compat
mode, we do not report a previous .meta file as being involved
in an error.

meta_oodate(): we should track 'latestdir' on a per process basis.
Also, only consider target out of date if a not-found file is outside
'CWD'.  Ignore anything in "/tmp/" or tmpdir.

When a source file moves, make will ignore the stale dependency,
but if the file in question is one that needs to be compiled (.c or .cc),
it still hands the bogus name to the compiler.

If Dir_MTime() cannot find such a file (gn->iParents is not empty),
see if the basename can be found via .PATH, and if so set gn->path to
the found file.   This prevents the stale path being given to the
compiler.

In meta_oodate(), if a referenced file no longer exists, consider the
target out-of-date.

Also, if meta_oodate() decides a target is out-of-date, and it
it uses .OODATE in its commands, we need .OODATE recomputed.
Undo our call to Make_DoAllVar() so that the call from Make_OODate()
will do the right thing.

tools/make/buildmake.sh compiles *.c, make sure meta.c does nothing
unless USE_META is defined.

We do not need filemon(9) to have a .meta file

Add meta.c which implements "meta" mode for make.
In this mode, a .meta file is created for each target, capturing
the expanded commands used, any command output, and if filemon(9)
is available, a record of system calls which are of interest.
Not enabled unless USE_META=yes is set when building make.
Also, if FILEMON_H exists, meta.c will be compiled to use filemon(9).