![[BACK]](/icons/back.gif){kind=icon}
Up to [cvs.NetBSD.org] / src / bin / sh
Request diff between arbitrary revisions
Keyword substitution: kv
Default branch: MAIN
Fix processing of unknown variable expansion types.
Our shell is (was) one of the last not to do this correctly.
Expansions are supposed to happen only when the command in which
they occur is being executed, not while it is being parsed.
If the expansion only happens them, errors should only be
detected then.
Make it work like that (I saw after I fixed this that FreeBSD
had done it, long ago, almost the same way - it is kind of an
obvious thing to do).
This will allow code like
if test it is shell X
then
commands using shell X specific expansion ops
else if it is shell Y
then
commands using shell Y specific expansion ops
else ...
fi
Previously expansion errors were detected while parsing, so
if we're not shell X, and don't implement something that it
does (some extension to the standard) that would have generated
a parser syntax error, and the script could not be executed
(despite the line with the error never being executed).
Note that this change does not handle all such possible
extensions, just this one. Others are much harder.
One side effect of this change is that sh will now continue
reading a variable expansion until it locates the terminating
'}' (in ${var} forms) regardless of how broken it obviously
is (to our shell) whereas previously it would have bailed out
as soon as an oddity was spotted.
bin: fix lint warning "effectively discards 'const'" For example: src/bin/ed/io.c(339): warning: call to 'strchr' effectively discards 'const' from argument [346] No binary change.
Implement expandvar() : runs var/arith/cmdsub expansions on var value expandvar() is like expandenv() and expandstr() - each expands variable values in different contexts, expandenv() when processing environment vars at startup (and there is no existing state to lose), but is always using unsafe data, so never runs command substitutions, expandstr() while in the middle of processing commands (mostly to expand the prompt variables, so there is probably half a command tree in the process of being built) - it uses the promptcmds option to decide whether to run command substitutions. expandvar() is for intermediate situations, where a variable is to be used during processing. It will run command substitutions if the variable value concerned was not imported from the environment. There are currently no uses of expandvar(), so this change alters nothing in the shell - but there will be in the near future.
Work around a probably gcc12 bug in detecting "potentially clobbered" variables after longjmp() for some architectures (sh3 at least). This should allow the workaround to disable those warnings for this file to be removed. In the affected function the extra var & assignment added should simply be deleted by any good optimiser, but if not, it doesn't matter, as performance of this function (expandonstack()) is almost irrelevant.
The great shell trailing whitespace cleanup of 2023... Inspired by private e-mail comments from mouse@ NFCI.
fix various typos in comments.
Avoid generating error messages implying that user errors are illegal.
s/desireable/desirable/ in comments.
s/existance/existence/ in comment.
Fix value of ${LINENO} in "for" commands.
This affects (as best I can tell) only uses of ${LINENO} in PS4
when -x is enabled (and perhaps only when the list contains no
expansions). "for" like "case" (which was already handled) is
special in that it generates trace output before actually executing
any kind of simple command.
Improve the solution for the 2nd access to a fd which shouldn't
be available ("13") issue reported by Jan Schaumann on netbsd-users.
This fixes a bug in the earlier fix (a day or so ago) which could allow the
shell's idea of which fd range was in use by the script to get wildly
incorrect, but now also actually looks to see which fd's are in use as
renamed other user fd's during the lifetime of a redirection which needs
to be able to be undone (most redirections occur after a fork and are
permanent in the child process). Attempting to access such a fd (as with
attempts to access a fd in use by the shell for its own purposes) is treated
as an attempt to access a closed fd (EBADF). Attempting to reuse the fd
for some other purpose (which should be rare, even for scripts attempting
to cause problems, since the shell generally knows which fds the script
wants to use, and avoids them) will cause the renamed (renumbered) fd
to be renamed again (moved aside to some other available fd), just as
happens with the shell's private fds.
Also, when a generic fd is required, don't give up because of EMFILE
or similar unless there are no available fds at all (we might prefer >10
or bigger, but if there are none there, use anything). This avoids
redirection errors when ulimit -n has been set small, and all the fds >10
that are available have been used, but we need somewhere to park the old
user of a fd while we reuse that fd for the redirection.
Deal with some issues where fds intended only for internal use by the shell were available for manipulation by scripts (or the user). These issues were reported by Jan Schaumann on netbsd-users. The first allows the user to reference sh internal fds, and is a simple fix - any sh internal fd is simply treated as if it were closed when referenced by the script. These fds can be discovered by examining /proc/N/fd so it is not difficult for a script to discover which fd it should attempt to access. The second allows the user to reference a user level fd which is one that is normally available to it, but at a point where it should no longer be visible (when that fd has been redirected, for a built in command, so the original fd needs to be saved so it can be restored, the saving fd should not be accessible). It is not as easy for the script to determine which fd to attempt here, as the relevant one exists only during the lifetime of a built-in command (and similar), but there are ways in some cases (aside from looking at /proc from another process). Fix this one by watching which fds the user script is attempting to use, and avoid using those as temporary fds. This is possible in this case as we know what command is being run, before we need to save the fds it uses. That's different from the earlier case where when the shell allocates its fds we have no idea what it might reference later. Also clean up a couple of other minor code issues (NFC intended) that I noticed while here...
Fix a bug with here document processing reported on the austin group list by oguzismailuysal@gmail.com (2021-09-08) (applies to all ash descendant shells). There were places in the parser where newline tokens were handled without reading any pending here documents (leaving those until a later newline token). Eg: in : << do | for x in xxx do do echo $x done The here doc text (<<do) should start immediately after the next newline (which is after xxx). But it wasn't happening until after the following newline (after the line containing only "do"). Bizarrely, this : << do | for x in xxx do echo $x do done "worked" because of that. For other cases that also failed, see the hard_cases test case in src/tests/bin/sh/t_here.sh Note that there's nothing magic about the particular end delimiter word chosen here, any will do, using the ones selected for the tests (and shown here) simply makes it all more mysterious! The one here is the exact case reported. Fix this by reading the here docs when processing newline tokens when they are encountered in other than the "normal" way. Whether this catches every possibility is unknown currently - may never be known for certain, but there are no more I can currently think of. No pullups needed, this isn't a significant enough bug (ie: no one actually writes code like this) to warrant that.
For now, probably forever, prohibit unquoted $ and ` in the names of
functions being defined (they can still be included if quoted).
If we parsed the way POSIX specifies (leaving the exact input text of
$ and ` expansions unaltered, until required to be expanded) this would
not be needed, as the name of a function being defined does not underbo
parameter, command, or arith expansions, so xxx$3() { : ; } would just
work. But for many reasons we don't do that (and are unlikely to ever,
though maintaing both forms might be an option someday) - which led to
very obscure behaviour (if sh were compiled in DEBUG mode, even an abort())
and certainly nothing useful. So just prohibit these uses for now.
(A portable function name must be a "name" so this makes no difference
at all to posix compat applications/scripts).
A doc update is pending (the updated sh.1 also contains updates in other
areas not yet appropriate to commit).
Remove extra semicolon.
Ooops, restore accidently removed files from merge mishap
Sync with HEAD
Merge changes from current as of 20200406
Correct a typo in a comment, 08x0 was meant to be 0x80 (duh!). NFC.
Sync with HEAD
Fix an (apparent) ancient ash bug, that was apparently fixed sometime
in the past, but managed to re-surface...
The expression "${0+\}}" should expand to "}" not "\}"
Almost all other shells handle it that way (incl FreeBSD & dash).
Issue pointed out by Martijn Dekker.
Add ATF sub-tests for the 4 old var expand operators (${var+word}
${var-word} ${var-word} and ${var?word} - including the forms
with the ':' included) and amongst those tests include test cases
for this issue, so if the bug tries to appear again, we can squash
it quicker. (The newer pattern matching operators are already
well tested as part of testing patterns.)
Finish the fixes from Feb 4 for handling of random data that matches the internal CTL* chars. The earlier fixes handled CTL* char values in var expansions, but not in various other places they can occur (positional parameters, $@ $* -- even potentially $0 and ~ expansions, as well as byte strings generated from a \u in a $'' string). These should all be correctly handled now. There is a new ISCTL() macro to make the test, rather than using the old BASESYNTAX[c]==CCTL form (which us still a viable alternative) as the new way allows compiler optimisations, and less mem references, so it should be smaller and faster. Also, be sure in all cases to remove any CTLESC (or other) CTL* chars from all strings before they are made available for any external use (there was one case missed - which didn't matter when we weren't bothering to escape the CTL* chars at all.) XXX pullup-8 (will need to be via a patch) along with the Feb 4 fixes.
Add a check that the file descriptor mentioned in a N> or N< type redirect operator is within range of what the code tree node can hold. Currently this is a no-op change (the new error can never occur) as the code already checks that N is in range for an int (and errors if not) and the field in the node in which we store N is also an int, so we cannot overflow - but fd's do not really need to be that big (the max a typical kernel supports is < 10000) so this just adds validation in case it ever happens that we decide we can save some node size (ie: sh memory) by making that field smaller. Note this is parse time error detection, and has no bearing upon the execution time error that will occur if a script attempts to use an fd that exceeds the process's max fd limit. NFCI (for now anyway.)
PR bin/53919 Suppress shell error messages while expanding $ENV (which also causes errors while expanding $PS1 $PS2 and $PS4 to be suppressed as well). This allows any random garbage that happens to be in ENV to not cause noise when the shell starts (which is effectively all it did). On a parse error (for any of those vars) we also use "" as the result, which will be a null prompt, and avoid attempting to open any file for ENV. This does not in any way change what happens for a correctly parsed command substitution (either when it is executed when permitted for one of the prompts, or when it is not (which is always for ENV)) and commands run from those can still produce error output (but shell errors remain suppressed).
Sync with HEAD
lexical analysis fixes. This fixes the tests just committed in
src/tests/bin/sh/t_here.sh
The "magicq" magic was all wrong - it cannot be simply a parameter
to readtoken1() as its value needs to alter during that routine
(eg: when magicq is set - processing here doc text, or whatever)
and we encountered ${var%pattern} "magicq" needs to be off for
"pattern" - and it wasn't.
To handle this magicq needs to be included in the token stack struct,
and simply init'd from the arg to readtoken1 (which we rename).
Then it can be manipulated as required.
Once we no longer have that problem, some other issues can be cleaned
up as well (some of this unbelievably fragile code was attempting to
cope with this in various ad-hoc - and mostly broken - ways).
Also, remove the magicq parameter from parsebackq() - it was not
used (at all) and should never be, a command substitution, wherever
it appears, always starts a new parsing context. How that applies
to old style command substitutions is less clear, but until we see
some real examples where we're not doing the right thing (slightly
less likely now than before ... nothing has changed here in the
way command substitutions are parsed, but quoting in general is
slightly better) I don't plan on worrying about it.
There are a couple of other minor cleanups, which make no actual
difference (like adding () around the use of the parameter in the
RETURN macro ... which is generally better, but makes no difference
here as the param is always a simple constant.
All the current ATF tests pass.
NFCI - DEBUG mode only change. Add tracing of lexical analyser operations. This is deliberately kept out of the normal "all on" set as it makes a *lot* of noise when enabled (especially in verbose mode) - but when needed, it helps (evidence for which is coming soon). As usual, no doc, you need the sources (and of course, a specially built sh to even be able to enable it.)
Fix an amazing crazy botch (of mine) when expanding prompt strings (PS1 etc) which, if the shell were already exiting, and a prompt were to be expanded (which only really happens if -x is enabled, and an exit trap is set, so the commands in the trap need PS4 expanded and written, last thing, before the shell exits) the shell would instead simply exit when it finished expanding PS4 (before even writing it, or the xtrace output). There were more conditions required to set up the environment for this to actually occur (it seems to only happen when the exit trap is set in a function, called in a command substitution) but that's unimportant, the code was nonsense. Problem noticed by Martijn Dekker. XXX pullup -8
Synch with HEAD
pgetc_linecont() needs to use pgetc() rather than pgetc_macro() so the fake char returned by the latter when an alias ends (which is there so we can correctly avoid alias recursion) is correctly ignored where it is not wanted.
A similar fix to that added in 1.169 of eval.c, but here for when processing command substitutions. If there is an error while processing, the any pending queued input should be discarded. From FreeBSD.
Sync with HEAD, resolve a few conflicts
PR standards/42829 Implement parameter and arithmetic expansion of $ENV before using it as the name of a file from which to read startup commands for the shell. This continues to happen for all interactive shells, and non-interactive shells for which the posix option is not set (-o posix). On any actual error, or if an attempt is made to use command substitution, then the value of ENV is used unchanged as the file name. The expansion complies with POSIX XCU 2.5.3, though that only requires parameter expansion - arithmetic expansion is an extension (but for us, it is much easier to do, than not to do, and it allows some weird stuff, if you're so inclined....) Note that there is no ~ expansion (use $HOME).
Pull up following revision(s) (requested by kre in ticket #1663): bin/sh/parser.c: revision 1.152 (via patch) PR bin/53712 Avoid crash from redirect on null compound command.
comment out unused.
Yet another foray into the mysterious world of $@ -- this time
to fix the (unusual) idiom "${1+$@}" (the quotes are part of it).
This seems to have broken about 5 or 6 years ago (somewhere
between -6 and -7), I believe.
Note this is not the same as "$@" and also not the same as ${1+"$@"}
(much more common idioms) which both worked.
Also attempt to deal with "" more correctly, especially when it
appears adjacent to "$@" (or one of the similar constructs.)
This stuff is still all as ugly and hackish (and fragile) as is
possible to imagine, but in an effort to allow some of the weirdness
to eventually go away, the parser output has been made more
regular and all quoted (parts of) words always now start with
CTLQUOTEMARK and end with CTLQUOTEEND regardless of where the
quotes appear.
This allows us to tell the difference between """$@" and "$@"
which was impossible before - yet they are required to generate
different output when there are no args (when "$@" simply vanishes).
Needless to say that change had ramifications all over the place.
To simplify any similar change in the future, there are some new
macros that can generally be used to detect the "noise" data when
processing words, rather than open coding that every time (which
meant that there would *always* be one which missed getting
updated...)
Several other bugs (of my making, and older ones) are also fixed.
The aim is that (aside from anything that is detecting the cases
that were broken before - which were all unlikely uses of sh
syntax) these changes should have no external visible impact.
Sure...
Revamp aliases - as dumb an idea as they are, if we're going to have them, they should work as documented, not cause core dumps, reference after free, incorrect replacements, failing to implement alias after alias, ... The big comment that ended: This is a good idea ------- ***NOT*** and the hack it was describing are gone. Note that most of this was from original CVS version 1.1 code (ie: came from the original import, even before 4.4-Lite was merged. That is, May 1994. And no-one in 24.5 years noticed (or at least complained about) all the bugs (or at least, most of them)). With these changes, aliases ought to work (if you can call it that) as they are expected to by POSIX. Now if only we could get POSIX to delete them (or make them optional)... Changes partly inspired by similar changes made by FreeBSD, (as was the previous change to alias.c, forgot ack in commit log for that one, apologies) but done a little differently, and perhaps with a slightly better outcome.
Rename the internal function "makename" to "makeword" to better reflect what it actually does (makearg would have been an alternative). While here, notice the one remaining place where it should have been used, but was left open coded, and consume that one. NFCI.
NFC. Need a grain of const
Sync with HEAD, resolve a couple of conflicts
Rationalise (slightly) the way that expansions are processed
to hide meta-characters in the result when the expansion was
in (double) quotes, and so should not be further processed.
Most of this has been OK for a long while, but \ needs hiding
as well, which complicates things, as \ cannot simply be hidden
in the syntax tables as one of the group of random special characters.
This was fixed earlier for simple variable expansions, but
every variety has its own code path ($var uses different code
than $n which is different than $(...), which is different
again from ~ expansions, and also from what $'...' produces).
This could be fixed by moving them all to a common code path,
but that's harder than it seems. The form in which the data
is made available differs, so one common routine would need
a whole bunch of different "get the next char or indicate end"
methods - probably via passing in an accessor function.
That's all a lot of churn, and would probably slow the shell.
Instead, just make macros for doing the standard tests, and
use those instead of open coding (differently) each time.
This way some of the code paths don't end up forgetting to
handle '\' (which is different than all the others).
This removes one optimisation ... when no escaping is needed
(like just $var (unquoted) where magic chars (think '*') in
the value are intended to remain magic), the code avoided doing
two tests for each char ("do we need escapes" and "is this char
one that needs escaping") by choosing two different syntax
tables (choice made outside the loop) - one of which never
returns the magic "needs escaping" result, and the other does
when appropriate, and then just avoiding the "do we need escapes"
test for each character processed. Then when '\' was fixed,
there needed to be another test for it, as it cannot (for other
reasons) be the same as all the others for which "this char
need escaping" is true. So that added a 2nd test for each char...
Not all the code paths were updated. Hence the bugs...
nb: this is all rarely seen in the wild, so it is no big
surprised that no-one ever noticed.
Now the "use two different syntax tables" is gone (the two
returned the same for '\' which is why '\' needed special
processing) - and in order to avoid two tests for each
char (plus the \ test) we duplicate the loops, one of which
tests each char to see if it needs an escape, the 2nd just
copies them. This should be faster in the "no escapes"
code path (though that is not the point) and perhaps also
in the "escapes needed" path (no indirect reference to
the syntax table - though that would probably be in a
register) but makes the code slightly bigger. For /bin/sh
the text segment (on amd64) has grown by 48 bytes. But
it still uses the same number of 512 byte pages (and hence
also any bigger page size). The resulting file size
(/bin/sh) is identical before and after. So is /rescue/sh
(or /rescue/anything-else).
Pull up following revision(s) (requested by kre in ticket #1086): bin/sh/parser.c: revision 1.152 PR bin/53712 Avoid crash from redirect on null compound command.
PR bin/53712 Avoid crash from redirect on null compound command.
Allow shells forked to run command substitutions while expanding prompts to exit when they're done, rather than forcing them to turn into interactive shells and start reading input ... Completes a part of the previous changes (just 10+ weeks late...) Should fix the prompt expansion issue reported by Caóc on current-users.
Pull up following revision(s) via patch (requested by kre in ticket #1015): bin/sh/expand.c: revision 1.124 bin/sh/expand.c: revision 1.127 bin/sh/parser.c: revision 1.148 bin/sh/parser.c: revision 1.149 bin/sh/syntax.c: revision 1.6 bin/sh/syntax.h: revision 1.9 (partial) First pass at fixing some of the more arcane pattern matching possibilities that we do not currently handle all that well. This mostly means (for now) making sure that quoted pattern magic characters (as well as quoted sh syntax magic chars) are properly marked, so they remain known as being quoted, and do not turn into pattern magic. Also, make sure that an unquoted \ in a pattern always quotes whatever comes next (which, unlike in regular expressions, includes inside [] matches), - Part 2 of pattern matching (glob etc) fixes. Attempt to correctly deal with \ (both when it is a literal, in appropriate cases, and when it appears as CTLESC when it was detected as a quoting character during parsing). In a pattern, in sh, no quoted character can ever be anything other than a literal character. This is quite different than regular expressions, and even different than other uses of glob matching, where shell quoting is not an issue. In something like ls ?\*.c the ? is a meta-character, the * is a literal (it was quoted). This is nothing new, sh has handled that properly for ever. But the same happens with VAR='?\*.c' and ls $VAR which has not always been handled correctly. Of course, in ls "$VAR" nothing in VAR is a meta-character (the entire expansion is quoted) so even the '\' must match literally (or more accurately, no matching happens - VAR simply contains an "unusual" filename). But if it had been ls *"$VAR" then we would be looking for filenames that end with the literal 5 characters that make up $VAR. The same kinds of things are requires of matching patterns in case statements, and sub-strings with the % and # operators in variable expansions. While here, the final remnant of the ancient !! pattern matching hack has been removed (the code that actually implemented it was long gone, but one small piece remained, not doing any real harm, but potentially wasting time - if someone gave a pattern which would once have invoked that hack.)
Sync with HEAD Resolve a couple of conflicts (result of the uimin/uimax changes)
Pull up following revision(s) (requested by kre in ticket #988): bin/sh/parser.c: revision 1.147 bin/sh/var.c: revision 1.70 bin/sh/mystring.c: revision 1.18 bin/sh/options.c: revision 1.53 bin/sh/histedit.c: revision 1.53 Remove atoi() Mostly use number() (no longer implemented using atoi()) when an unsigned integer is required, but use strtoXXX() when a conversion is wanted, without the possibility or error (like setting OPTIND and RANDOM). Always init OPTIND to 1 when sh starts (overriding anything in environ.)
PR bin/48875 (is related, and ameliorated, but not exactly "fixed")
Import a whole set of tree evaluation enhancements from FreeBSD.
With these, before forking, the shell predicts (often) when all it will
have to do after forking (in the parent) is wait for the child and then
exit with the status from the child, and in such a case simply does not
fork, but rather allows the child to take over the parent's role.
This turns out to handle the particular test case from PR bin/48875 in
such a way that it works as hoped, rather than as it did (the delay there
was caused by an extra copy of the shell hanging around waiting for the
background child to complete ... and keeping the command substitution
stdout open, so the "real" parent had to wait in case more output appeared).
As part of doing this, redirection processing for compound commands gets
moved out of evalsubshell() and into a new evalredir(), which allows us
to properly handle errors occurring while performing those redirects,
and not mishandle (as in simply forget) fd's which had been moved out
of the way temporarily.
evaltree() has its degree of recursion reduced by making it loop to
handle the subsequent operation: that is instead of (for any binop
like ';' '&&' (etc)) where it used to
evaltree(node->left);
evaltree(node->right);
return;
it now does (kind of)
next = node;
while ((node = next) != NULL) {
next = NULL;
if (node is a binary op) {
evaltree(node->left);
if appropriate /* if && test for success, etc */
next = node->right;
continue;
}
/* similar for loops, etc */
}
which can be a good saving, as while the left side (now) tends to be
(usually) a simple (or simpleish) command, the right side can be many
commands (in a command sequence like a; b; c; d; ... the node at the
top of the tree will now have "a" as its left node, and the tree for
b; c; d; ... as its right node - until now everything was evaluated
recursively so it made no difference, and the tree was constructed
the other way).
if/while/... statements are done similarly, recurse to evaluate the
condition, then if the (or one of the) body parts is to be evaluated,
set next to that, and loop (previously it recursed).
There is more to do in this area (particularly in the way that case
statements are processed - we can avoid recursion there as well) but
that can wait for another day.
While doing all of this we keep much better track of when the shell is
just going to exit once the current tree is evaluated (with a new
predicate at_eof() to tell us that we have, for sure, reached the end
of the input stream, that is, this shell will, for certain, not be reading
more command input) and use that info to avoid unneeded forks. For that
we also need another new predicate (have_traps()) to determine of there
are any caught traps which might occur - if there are, we need to remain
to (potentially) handle them, so these optimisations will not occur (to
make the issue in PR 48875 appear again, run the same code, but with a
trap set to execute some code when a signal (or EXIT) occurs - note that
the trap must be set in the appropriate level of sub-shell to have this
effect, any caught traps are cleared in a subshell whenever one is created).
There is still work to be done to handle traps properly, whatever
weirdness they do (some of which is related to some of this.)
These changes do not need man page updates, but 48875 does - an update
to sh.1 will be forthcoming once it is decided what it should say...
Once again, all the heavy lifting for this set of changes comes directly
(with thanks) from the FreeBSD shell.
XXX pullup-8 (but not very soon)
Sync with HEAD
Part 2 of pattern matching (glob etc) fixes. Attempt to correctly deal with \ (both when it is a literal, in appropriate cases, and when it appears as CTLESC when it was detected as a quoting character during parsing). In a pattern, in sh, no quoted character can ever be anything other than a literal character. This is quite different than regular expressions, and even different than other uses of glob matching, where shell quoting is not an issue. In something like ls ?\*.c the ? is a meta-character, the * is a literal (it was quoted). This is nothing new, sh has handled that properly for ever. But the same happens with VAR='?\*.c' and ls $VAR which has not always been handled correctly. Of course, in ls "$VAR" nothing in VAR is a meta-character (the entire expansion is quoted) so even the '\' must match literally (or more accurately, no matching happens - VAR simply contains an "unusual" filename). But if it had been ls *"$VAR" then we would be looking for filenames that end with the literal 5 characters that make up $VAR. The same kinds of things are requires of matching patterns in case statements, and sub-strings with the % and # operators in variable expansions. While here, the final remnant of the ancient !! pattern matching hack has been removed (the code that actually implemented it was long gone, but one small piece remained, not doing any real harm, but potentially wasting time - if someone gave a pattern which would once have invoked that hack.)
First pass at fixing some of the more arcane pattern matching possibilities that we do not currently handle all that well. This mostly means (for now) making sure that quoted pattern magic characters (as well as quoted sh syntax magic chars) are properly marked, so they remain known as being quoted, and do not turn into pattern magic. Also, make sure that an unquoted \ in a pattern always quotes whatever comes next (which, unlike in regular expressions, includes inside [] matches),
Remove atoi() Mostly use number() (no longer implemented using atoi()) when an unsigned integer is required, but use strtoXXX() when a conversion is wanted, without the possibility or error (like setting OPTIND and RANDOM). Always init OPTIND to 1 when sh starts (overriding anything in environ.)
Pull up following revision(s) (requested by kre in ticket #804):
bin/sh/parser.c: revision 1.146
PR bin/53201
Don't synerr on
${var-anything
more}
The newline in the middle of the var expansion is permitted.
Bug reported by Martijn Dekker from his modernish tests.
XXX pullup-8
Sync with HEAD
PR bin/53201
Don't synerr on
${var-anything
more}
The newline in the middle of the var expansion is permitted.
Bug reported by Martijn Dekker from his modernish tests.
XXX pullup-8
Pull up following revision(s) (requested by kre in ticket #355): bin/sh/parser.c: revision 1.145 PR bin/52715 Correct a (relatively harmless) use after free in prompt expansion processing [detected by asan.] Relatively harmless: as (while incorrect) the way the data is (was) used more or less guaranteed that the buffer contents would be unaltered until well after they are (were) no longer wanted (this is the expanded prompt string, it is just output (or copied into libedit internal storage) and forgotten. This should make no visible difference to anyone (not using asan or similar.)
PR bin/52715 Correct a (relatively harmless) use after free in prompt expansion processing [detected by asan.] Relatively harmless: as (while incorrect) the way the data is (was) used more or less guaranteed that the buffer contents would be unaltered until well after they are (were) no longer wanted (this is the expanded prompt string, it is just output (or copied into libedit internal storage) and forgotten. This should make no visible difference to anyone (not using asan or similar.) XXX pullup -8
Add support for $'...' quoting (based upon C "..." strings, with \ expansions.)
Implementation largely obtained from FreeBSD, with adaptations to meet the
needs and style of this sh, some updates to agree with the current POSIX spec,
and a few other minor changes.
The POSIX spec for this ( http://austingroupbugs.net/view.php?id=249 )
[see note 2809 for the current proposed text] is yet to be approved,
so might change. It currently leaves several aspects as unspecified,
this implementation handles those as:
Where more than 2 hex digits follow \x this implementation processes the
first two as hex, the following characters are processed as if the \x
sequence was not present. The value obtained from a \nnn octal sequence
is truncated to the low 8 bits (if a bigger value is written, eg: \456.)
Invalid escape sequences are errors. Invalid \u (or \U) code points are
errors if known to be invalid, otherwise can generate a '?' character.
Where any escape sequence generates nul ('\0') that char, and the rest of
the $'...' string is discarded, but anything remaining in the word is
processed, ie: aaa$'bbb\0ccc'ddd produces the same as aaa'bbb'ddd.
Differences from FreeBSD:
FreeBSD allows only exactly 4 or 8 hex digits for \u and \U (as does C,
but the current sh proposal differs.) reeBSD also continues consuming
as many hex digits as exist after \x (permitted by the spec, but insane),
and reject \u0000 as invalid). Some of this is possibly because that
their implementation is based upon an earlier proposal, perhaps note 590 -
though that has been updated several times.
Differences from the current POSIX proposal:
We currently always generate UTF-8 for the \u & \U escapes. We should
generate the equivalent character from the current locale's character set
(and UTF8 only if that is what the current locale uses.)
If anyone would like to correct that, go ahead.
We (and FreeBSD) generate (X & 0x1F) for \cX escapes where we should generate
the appropriate control character (SOH for \cA for example) with whatever
value that has in the current character set. Apart from EBCDIC, which
we do not support, I've never seen a case where they differ, so ...
Pull up following revision(s) (requested by kre in ticket #199): bin/sh/input.c: revision 1.61 bin/sh/parser.c: revision 1.143 PR bin/52458 Avoid mangling history when editing is enabled, and the prompt contains a \n Also, allow empty input lines into history when they are being appended to a previous (partial) command (but not when they would just make an empty entry) . For all the gory details, see the PR. Note nothing here actually makes prompts containing \n work correctly when editing is enabled, that's a libedit issue, which will be addressed some other time.
PR bin/52458 Avoid mangling history when editing is enabled, and the prompt contains a \n Also, allow empty input lines into history when they are being appended to a previous (partial) command (but not when they would just make an empty entry). For all the gory details, see the PR. Note nothing here actually makes prompts containing \n work correctly when editing is enabled, that's a libedit issue, which will be addressed some other time.
PR bin/48498 PR bin/52426 Don't ignore unexpected reserved words after ';' Don't allow any random token type as a case stmt pattern, only a word. Those are ancient ash bugs and do not affect correct scripts. Don't ignore redirects in a case stmt list where the list is nothing but redirects (if the pattern matches, the redirects should be performed). That was introduced when a redirect only case stmt list was allowed (older shells had generated a syntax error.) Random cleanups/refactoring taken from or inspired by the FreeBSD sh parser ... use makename() consistently to create a NARG node - we were using it in a couple of places but most NARG node creation was open coded. Introduce consumetoken() (from FreeBSD) to handle the fairly common case where exactly one token type must come next, and we need to check that, and skip past it when found (or error) and linebreak() (new) to handle places where optional \n's are permitted. Both previously open coded. Simplify list() by removing its second arg, which was only ever used when handling the end of a `` (old style command substitution). Simply move the code from inside list() to just after its call in the `` case (from FreeBSD.)
Pull up following revision(s) (requested by kre in ticket #103):
bin/kill/kill.c: 1.28
bin/sh/Makefile: 1.111-1.113
bin/sh/arith_token.c: 1.5
bin/sh/arith_tokens.h: 1.2
bin/sh/arithmetic.c: 1.3
bin/sh/arithmetic.h: 1.2
bin/sh/bltin/bltin.h: 1.15
bin/sh/cd.c: 1.49-1.50
bin/sh/error.c: 1.40
bin/sh/eval.c: 1.142-1.151
bin/sh/exec.c: 1.49-1.51
bin/sh/exec.h: 1.26
bin/sh/expand.c: 1.113-1.119
bin/sh/expand.h: 1.23
bin/sh/histedit.c: 1.49-1.52
bin/sh/input.c: 1.57-1.60
bin/sh/input.h: 1.19-1.20
bin/sh/jobs.c: 1.86-1.87
bin/sh/main.c: 1.71-1.72
bin/sh/memalloc.c: 1.30
bin/sh/memalloc.h: 1.17
bin/sh/mknodenames.sh: 1.4
bin/sh/mkoptions.sh: 1.3-1.4
bin/sh/myhistedit.h: 1.12-1.13
bin/sh/nodetypes: 1.16-1.18
bin/sh/option.list: 1.3-1.5
bin/sh/parser.c: 1.133-1.141
bin/sh/parser.h: 1.22-1.23
bin/sh/redir.c: 1.58
bin/sh/redir.h: 1.24
bin/sh/sh.1: 1.149-1.159
bin/sh/shell.h: 1.24
bin/sh/show.c: 1.43-1.47
bin/sh/show.h: 1.11
bin/sh/syntax.c: 1.4
bin/sh/syntax.h: 1.8
bin/sh/trap.c: 1.41
bin/sh/var.c: 1.56-1.65
bin/sh/var.h: 1.29-1.35
An initial attempt at implementing LINENO to meet the specs.
Aside from one problem (not too hard to fix if it was ever needed) this version
does about as well as most other shell implementations when expanding
$((LINENO)) and better for ${LINENO} as it retains the "LINENO hack" for the
latter, and that is very accurate.
Unfortunately that means that ${LINENO} and $((LINENO)) do not always produce
the same value when used on the same line (a defect that other shells do not
share - aside from the FreeBSD sh as it is today, where only the LINENO hack
exists and so (like for us before this commit) $((LINENO)) is always either
0, or at least whatever value was last set, perhaps by
LINENO=${LINENO}
which does actually work ... for that one line...)
This could be corrected by simply removing the LINENO hack (look for the string
LINENO in parser.c) in which case ${LINENO} and $((LINENO)) would give the
same (not perfectly accurate) values, as do most other shells.
POSIX requires that LINENO be set before each command, and this implementation
does that fairly literally - except that we only bother before the commands
which actually expand words (for, case and simple commands). Unfortunately
this forgot that expansions also occur in redirects, and the other compound
commands can also have redirects, so if a redirect on one of the other compound
commands wants to use the value of $((LINENO)) as a part of a generated file
name, then it will get an incorrect value. This is the "one problem" above.
(Because the LINENO hack is still enabled, using ${LINENO} works.)
This could be fixed, but as this version of the LINENO implementation is just
for reference purposes (it will be superseded within minutes by a better one)
I won't bother. However should anyone else decide that this is a better choice
(it is probably a smaller implementation, in terms of code & data space then
the replacement, but also I would expect, slower, and definitely less accurate)
this defect is something to bear in mind, and fix.
This version retains the *BSD historical practice that line numbers in functions
(all functions) count from 1 from the start of the function, and elsewhere,
start from 1 from where the shell started reading the input file/stream in
question. In an "eval" expression the line number starts at the line of the
"eval" (and then increases if the input is a multi-line string).
Note: this version is not documented (beyond as much as LINENO was before)
hence this slightly longer than usual commit message.
A better LINENO implementation. This version deletes (well, #if 0's out)
the LINENO hack, and uses the LINENO var for both ${LINENO} and $((LINENO)).
(Code to invert the LINENO hack when required, like when de-compiling the
execution tree to provide the "jobs" command strings, is still included,
that can be deleted when the LINENO hack is completely removed - look for
refs to VSLINENO throughout the code. The var funclinno in parser.c can
also be removed, it is used only for the LINENO hack.)
This version produces accurate results: $((LINENO)) was made as accurate
as the LINENO hack made ${LINENO} which is very good. That's why the
LINENO hack is not yet completely removed, so it can be easily re-enabled.
If you can tell the difference when it is in use, or not in use, then
something has broken (or I managed to miss a case somewhere.)
The way that LINENO works is documented in its own (new) section in the
man page, so nothing more about that, or the new options, etc, here.
This version introduces the possibility of having a "reference" function
associated with a variable, which gets called whenever the value of the
variable is required (that's what implements LINENO). There is just
one function pointer however, so any particular variable gets at most
one of the set function (as used for PATH, etc) or the reference function.
The VFUNCREF bit in the var flags indicates which func the variable in
question uses (if any - the func ptr, as before, can be NULL).
I would not call the results of this perfect yet, but it is close.
Unbreak (at least) i386 build .... I have no idea why this built for me on
amd64 (problem was missing prototype for snprintf witout <stdio.h>)
While here, add some (DEBUG mode only) tracing that proved useful in
solving another problem.
Set the line number before expanding args, not after. As the line_number
would have usually been set earlier, this change is mostly an effective
no-op, but it is better this way (just in case) - not observed to have
caused any problems.
Undo some over agressive fixes for a (pre-commit) bug that did not
need these changes to be fixed - and these cause problems in another
absurd use case. Either of these issues is unlikely to be seen by
anyone who isn't an idiot masochist...
PR bin/52280
removescapes_nl in expari() even when not quoted,
CRTNONL's appear regardless of quoting (unlike CTLESC).
New sentence, new line. Whitespace.
Improve the (new) LINENO section, markup changes (with thanks to wiz@ for
assistace) and some better wording in a few placed.
I am an idiot... revert the previous unintended commit.
Remove some left over baggage from the LINENO v1 implementation that
didn't get removed with v2, and should have. This would have had
(I think, without having tested it) one very minor effect on the way
LINENO worked in the v2 implementation, but my guess is it would have
taken a long time before anyone noticed...
Correct spelling in comments of DEBUG only code...
(Perhaps) temporary fix to pkgtools (cwrappers) build (configure).
Expanding `` containing \ \n sequences looks to have been giving
problems. I don't think this is the correct fix, but it will do
no worse harm than (perhaps) incorrectly calculating LINENO in this
kind of (rare) circumstance. I'll look and see if there should be
a better fix later.
s/volatile/const/ -- wonderful how opposites attract like this.
NFC (normal use) - DEBUG only change, when showing empty arg list don't
omit terminating \n.
Free stack memory in a couple of obscure cases where it wasn't
being done (one in probably dead code that is never compiled, the other
in a very rare error case.) Since it is stack memory it wasn't lost
in any case, just held longer than needed.
Many internal memory management type fixes.
PR bin/52302 (core dump with interactive shell, here doc and error
on same line) is fixed. (An old bug.)
echo "$( echo x; for a in $( seq 1000 ); do printf '%s\n'; done; echo y )"
consistently prints 1002 lines (x, 1000 empty ones, then y) as it should
(And you don't want to know what it did before, or why.) (Another old one.)
(Recently added) Problems with ~ expansion fixed (mem management related).
Proper fix for the cwrappers configure problem (which includes the quick
fix that was done earlier, but extends upon that to be correct). (This was
another newly added problem.)
And the really devious (and rare) old bug - if STACKSTRNUL() needs to
allocate a new buffer in which to store the \0, calculate the size of
the string space remaining correctly, unlike when SPUTC() grows the
buffer, there is no actual data being stored in the STACKSTRNUL()
case - the string space remaining was calculated as one byte too few.
That would be harmless, unless the next buffer also filled, in which
case it was assumed that it was really full, not one byte less, meaning
one junk char (a nul, or anything) was being copied into the next (even
bigger buffer) corrupting the data.
Consistent use of stalloc() to allocate a new block of (stack) memory,
and grabstackstr() to claim a block of (stack) memory that had already
been occupied but not claimed as in use. Since grabstackstr is implemented
as just a call to stalloc() this is a no-op change in practice, but makes
it much easier to comprehend what is really happening. Previous code
sometimes used stalloc() when the use case was really for grabstackstr().
Change grabstackstr() to actually use the arg passed to it, instead of
(not much better than) guessing how much space to claim,
More care when using unstalloc()/ungrabstackstr() to return space, and in
particular when the stack must be returned to its previous state, rather than
just returning no-longer needed space, neither of those work. They also don't
work properly if there have been (really, even might have been) any stack mem
allocations since the last stalloc()/grabstackstr(). (If we know there
cannot have been then the alloc/release sequence is kind of pointless.)
To work correctly in general we must use setstackmark()/popstackmark() so
do that when needed. Have those also save/restore the top of stack string
space remaining.
[Aside: for those reading this, the "stack" mentioned is not
in any way related to the thing used for maintaining the C
function call state, ie: the "stack segment" of the program,
but the shell's internal memory management strategy.]
More comments to better explain what is happening in some cases.
Also cleaned up some hopelessly broken DEBUG mode data that were
recently added (no effect on anyone but the poor semi-human attempting
to make sense of it...).
User visible changes:
Proper counting of line numbers when a here document is delimited
by a multi-line end-delimiter, as in
cat << 'REALLY
END'
here doc line 1
here doc line 2
REALLY
END
(which is an obscure case, but nothing says should not work.) The \n
in the end-delimiter of the here doc (the last one) was not incrementing
the line number, which from that point on in the script would be 1 too
low (or more, for end-delimiters with more than one \n in them.)
With tilde expansion:
unset HOME; echo ~
changed to return getpwuid(getuid())->pw_home instead of failing (returning ~)
POSIX says this is unspecified, which makes it difficult for a script to
compensate for being run without HOME set (as in env -i sh script), so
while not able to be used portably, this seems like a useful extension
(and is implemented the same way by some other shells).
Further, with
HOME=; printf %s ~
we now write nothing (which is required by POSIX - which requires ~ to
expand to the value of $HOME if it is set) previously if $HOME (in this
case) or a user's directory in the passwd file (for ~user) were a null
STRING, We failed the ~ expansion and left behind '~' or '~user'.
Changed the long name for the -L option from lineno_fn_relative
to local_lineno as the latter seemed to be marginally more popular,
and perhaps more importantly, is the same length as the peviously
existing quietprofile option, which means the man page indentation
for the list of options can return to (about) what it was before...
(That is, less indented, which means more data/line, which means less
lines of man page - a good thing!)
Cosmetic changes to variable flags - make their values more suited
to my delicate sensibilities... (NFC).
Arrange not to barf (ever) if some turkey makes _ readonly. Do this
by adding a VNOERROR flag that causes errors in var setting to be
ignored (intended use is only for internal shell var setting, like of "_").
(nb: invalid var name errors ignore this flag, but those should never
occur on a var set by the shell itself.)
From FreeBSD: don't simply discard memory if a variable is not set for
any reason (including because it is readonly) if the var's value had
been malloc'd. Free it instead...
NFC - DEBUG changes, update this to new TRACE method.
KNF - white space and comment formatting.
NFC - DEBUG mode only change - convert this to the new TRACE() format.
NFC - DEBUG mode only change - complete a change made earlier (marking
the line number when included in the trace line tag to show whether it
comes from the parser, or the elsewhere as they tend to be quite different).
Initially only one case was changed, while I pondered whether I liked it
or not. Now it is all done... Also when there is a line tag at all,
always include the root/sub-shell indicator character, not only when the
pid is included.
NFC: DEBUG related comment change - catch up with reality.
NFC: DEBUG mode only change. Fix botched cleanup of one TRACE().
"b" more forgiving when sorting options to allow reasonable (and intended)
flexibility in option.list format. Changes nothing for current option.list.
Now that excessive use of STACKSTRNUL has served its purpose (well, accidental
purpose) in exposing the bug in its implementation, go back to not using
it when not needed for DEBUG TRACE purposes. This change should have no
practical effect on either a DEBUG shell (where the STACKSTRNUL() calls
remain) or a non DEBUG shell where they are not needed.
Correct the initial line number used for processing -c arg strings.
(It was inheriting the value from end of profile file processing) - I didn't
notice before as I usually test with empty or no profile files to avoid
complications. Trivial change which should have very limited impact.
Fix from FreeBSD (applied there in July 2008...)
Don't dump core with input like sh -c 'x=; echo >&$x' - that is where
the word after a >& or <& redirect expands to nothing at all.
Another fix from FreeBSD (this one from April 2009).
When processing a string (as in eval, trap, or sh -c) don't allow
trailing \n's to destroy the exit status of the last command executed.
That is:
sh -c 'false
'
echo $?
should produce 1, not 0.
It is amazing what nonsense appears to work sometimes... (all my nonsense too!)
Two bugs here, one benign because of the way the script is used.
The other hidden by NetBSD's sort being stable, and the data not really
requiring sorting at all...
So as it happens these fixes change nothing, but they are needed anyway.
(The contents of the generated file are only used in DEBUG shells, so
this is really even less important than it seems.)
Another ancient (highly improbable) bug bites the dust. This one
caused by incorrect macro usage (ie: using the wrong one) which has
been in the sources since version 1.1 (ie: forever).
Like the previous (STACKSTRNUL) bug, the probability of this one
actually occurring has been infinitesimal but the LINENO code increases
that to infinitesimal and a smidgen... (or a few, depending upon usage).
Still, apparently that was enough, Kamil Rytarowski discovered that the
zsh configure script (damn competition!) managed to trigger this problem.
source .editrc after we initialize so that commands persist!
Make arg parsing in kill POSIX compatible with POSIX (XBD 2.12) by
parsing the way getopt(3) would, if only it could handle the (required)
-signumber and -signame options. This adds two "features" to kill,
-ssigname and -lstatus now work (ie: one word with all of the '-', the
option letter, and its value) and "--" also now works (kill -- -pid1 pid2
will not attempt to send the pid1 signal to pid2, but rather SIGTERM
to the pid1 process group and pid2). It is still the case that (apart
from --) at most 1 option is permitted (-l, -s, -signame, or -signumber.)
Note that we now have an ambiguity, -sname might mean "-s name" or
send the signal "sname" - if one of those turns out to be valid, that
will be accepted, otherwise the error message will indicate that "sname"
is not a valid signal name, not that "name" is not. Keeping the "-s"
and signal name as separate words avoids this issue.
Also caution: should someone be weird enough to define a new signal
name (as in the part after SIG) which is almost the same name as an
existing name that starts with 'S' by adding an extra 'S' prepended
(eg: adding a SIGSSYS) then the ambiguity problem becomes much worse.
In that case "kill -ssys" will be resolved in favour of the "-s"
flag being used (the more modern syntax) and would send a SIGSYS, rather
that a SIGSSYS. So don't do that.
While here, switch to using signalname(3) (bye bye NSIG, et. al.), add
some constipation, and show a little pride in formatting the signal names
for "kill -l" (and in the usage when appropriate -- same routine.) Respect
COLUMNS (POSIX XBD 8.3) as primary specification of the width (terminal width,
not number of columns to print) for kill -l, a very small value for COLUMNS
will cause kill -l output to list signals one per line, a very large
value will cause them all to be listed on one line.) (eg: "COLUMNS=1 kill -l")
TODO: the signal printing for "trap -l" and that for "kill -l"
should be switched to use a common routine (for the sh builtin versions.)
All changes of relevance here are to bin/kill - the (minor) changes to bin/sh
are only to properly expose the builtin version of getenv(3) so the builtin
version of kill can use it (ie: make its prototype available.)
Properly support EDITRC - use it as (naming) the file when setting
up libedit, and re-do the config whenever EDITRC is set.
Get rid of workarounds for ancient groff html backend.
Simplify macro usage.
Make one example more like a real world possibility (it still isn't, but
is closer) - though the actual content is irrelevant to the point being made.
Add literal prompt support this allows one to do:
CA="$(printf '\1')"
PS1="${CA}$(tput bold)${CA}\$${CA}$(tput sgr0)${CA} "
Now libedit supports embedded mode switch sequence, improve sh
support for them (adds PSlit variable to set the magic character).
NFC: DEBUG only change - provide an externally visible (to the DEBUG sh
internals) interface to one of the internal (private to trace code) functions
Include redirections in trace output from "set -x"
Implement PS1, PS2 and PS4 expansions (variable expansions, arithmetic
expansions, and if enabled by the promptcmds option, command substitutions.)
Implement a bunch of new shell environment variables. many mostly useful
in prompts when expanded at prompt time, but all available for general use.
Many of the new ones are not available in SMALL shells (they work as normal
if assigned, but the shell does not set or use them - and there is no magic
in a SMALL shell (usually for install media.))
Omnibus manual update for prompt expansions and new variables. Throw in
some random cleanups as a bonus.
Correct a markup typo (why did I not see this before the prev commit??)
Sort options (our default is 0..9AaBbZz).
Fix markup problems and a typo.
Make $- list flags in the same order they appear in sh(1)
Do a better job of detecting the error in pkgsrc/devel/libbson-1.6.3's
configure script, ie: $(( which is intended to be a sub-shell in a
command substitution, but is an arith subst instead, it needs to be
written $( ( to do as intended. Instead of just blindly carrying on to
find the missing )) somewhere, anywhere, give up as soon as we have seen
an unbalanced ')' that isn't immediately followed by another ')' which
in a valid arith subst it always would be.
While here, there has been a comment in the code for quite a while noting a
difference in the standard between the text descr & grammar when it comes to
the syntax of case statements. Add more comments to explain why parsing it
as we do is in fact definitely the correct way (ie: the grammar wins arguments
like this...).
DEBUG and white space changes only. Convert TRACE() calls for DEBUg mode
to the new style. NFC (when not debugging sh).
Mostly DEBUG and white space changes. Convert DEEBUG TRACE() calls to
the new format. Also #if 0 a function definition that is used nowhere.
While here, change the function of pushfile() slightly - it now sets
the buf pointer in the top (new) input descriptor to NULL, instead of
simply leaving it - code that needs a buffer always (before and after)
must malloc() one and assign it after the call. But code which does not
(which will be reading from a string or similar) now does not have to
explicitly set it to NULL (cleaner interface.) NFC intended (or observed.)
DEBUG changes: convert DEBUG TRACE() calls to new format.
ALso, cause exec failures to always cause the shell to exit with
status 126 or 127, whatever the cause. 127 is intended for lookup
failures (and is used that way), 126 is used for anything else that
goes wrong (as in several other shells.) We no longer use 2 (more easily
confused with an exit status of the command exec'd) for shell exec failures.
DEBUG only changes. Convert the TRACE() calls in the remaining files
that still used it to the new format. NFC.
Fix a reference after free (and consequent nonsense diagnostic for
attempts to set readonly variables) I added in 1.60 by incompletely
copying the FreeBSD fix for the lost memory issue.
Do a better job of detecting the error in pkgsrc/devel/libbson-1.6.3's configure script, ie: $(( which is intended to be a sub-shell in a command substitution, but is an arith subst instead, it needs to be written $( ( to do as intended. Instead of just blindly carrying on to find the missing )) somewhere, anywhere, give up as soon as we have seen an unbalanced ')' that isn't immediately followed by another ')' which in a valid arith subst it always would be. While here, there has been a comment in the code for quite a while noting a difference in the standard between the text descr & grammar when it comes to the syntax of case statements. Add more comments to explain why parsing it as we do is in fact definitely the correct way (ie: the grammar wins arguments like this...).
Implement PS1, PS2 and PS4 expansions (variable expansions, arithmetic expansions, and if enabled by the promptcmds option, command substitutions.)
Another ancient (highly improbable) bug bites the dust. This one caused by incorrect macro usage (ie: using the wrong one) which has been in the sources since version 1.1 (ie: forever). Like the previous (STACKSTRNUL) bug, the probability of this one actually occurring has been infinitesimal but the LINENO code increases that to infinitesimal and a smidgen... (or a few, depending upon usage). Still, apparently that was enough, Kamil Rytarowski discovered that the zsh configure script (damn competition!) managed to trigger this problem.
Many internal memory management type fixes. PR bin/52302 (core dump with interactive shell, here doc and error on same line) is fixed. (An old bug.) echo "$( echo x; for a in $( seq 1000 ); do printf '%s\n'; done; echo y )" consistently prints 1002 lines (x, 1000 empty ones, then y) as it should (And you don't want to know what it did before, or why.) (Another old one.) (Recently added) Problems with ~ expansion fixed (mem management related). Proper fix for the cwrappers configure problem (which includes the quick fix that was done earlier, but extends upon that to be correct). (This was another newly added problem.) And the really devious (and rare) old bug - if STACKSTRNUL() needs to allocate a new buffer in which to store the \0, calculate the size of the string space remaining correctly, unlike when SPUTC() grows the buffer, there is no actual data being stored in the STACKSTRNUL() case - the string space remaining was calculated as one byte too few. That would be harmless, unless the next buffer also filled, in which case it was assumed that it was really full, not one byte less, meaning one junk char (a nul, or anything) was being copied into the next (even bigger buffer) corrupting the data. Consistent use of stalloc() to allocate a new block of (stack) memory, and grabstackstr() to claim a block of (stack) memory that had already been occupied but not claimed as in use. Since grabstackstr is implemented as just a call to stalloc() this is a no-op change in practice, but makes it much easier to comprehend what is really happening. Previous code sometimes used stalloc() when the use case was really for grabstackstr(). Change grabstackstr() to actually use the arg passed to it, instead of (not much better than) guessing how much space to claim, More care when using unstalloc()/ungrabstackstr() to return space, and in particular when the stack must be returned to its previous state, rather than just returning no-longer needed space, neither of those work. They also don't work properly if there have been (really, even might have been) any stack mem allocations since the last stalloc()/grabstackstr(). (If we know there cannot have been then the alloc/release sequence is kind of pointless.) To work correctly in general we must use setstackmark()/popstackmark() so do that when needed. Have those also save/restore the top of stack string space remaining. [Aside: for those reading this, the "stack" mentioned is not in any way related to the thing used for maintaining the C function call state, ie: the "stack segment" of the program, but the shell's internal memory management strategy.] More comments to better explain what is happening in some cases. Also cleaned up some hopelessly broken DEBUG mode data that were recently added (no effect on anyone but the poor semi-human attempting to make sense of it...). User visible changes: Proper counting of line numbers when a here document is delimited by a multi-line end-delimiter, as in cat << 'REALLY END' here doc line 1 here doc line 2 REALLY END (which is an obscure case, but nothing says should not work.) The \n in the end-delimiter of the here doc (the last one) was not incrementing the line number, which from that point on in the script would be 1 too low (or more, for end-delimiters with more than one \n in them.) With tilde expansion: unset HOME; echo ~ changed to return getpwuid(getuid())->pw_home instead of failing (returning ~) POSIX says this is unspecified, which makes it difficult for a script to compensate for being run without HOME set (as in env -i sh script), so while not able to be used portably, this seems like a useful extension (and is implemented the same way by some other shells). Further, with HOME=; printf %s ~ we now write nothing (which is required by POSIX - which requires ~ to expand to the value of $HOME if it is set) previously if $HOME (in this case) or a user's directory in the passwd file (for ~user) were a null STRING, We failed the ~ expansion and left behind '~' or '~user'.
(Perhaps) temporary fix to pkgtools (cwrappers) build (configure). Expanding `` containing \ \n sequences looks to have been giving problems. I don't think this is the correct fix, but it will do no worse harm than (perhaps) incorrectly calculating LINENO in this kind of (rare) circumstance. I'll look and see if there should be a better fix later.
Remove some left over baggage from the LINENO v1 implementation that didn't get removed with v2, and should have. This would have had (I think, without having tested it) one very minor effect on the way LINENO worked in the v2 implementation, but my guess is it would have taken a long time before anyone noticed...
Undo some over agressive fixes for a (pre-commit) bug that did not need these changes to be fixed - and these cause problems in another absurd use case. Either of these issues is unlikely to be seen by anyone who isn't an idiot masochist...
A better LINENO implementation. This version deletes (well, #if 0's out)
the LINENO hack, and uses the LINENO var for both ${LINENO} and $((LINENO)).
(Code to invert the LINENO hack when required, like when de-compiling the
execution tree to provide the "jobs" command strings, is still included,
that can be deleted when the LINENO hack is completely removed - look for
refs to VSLINENO throughout the code. The var funclinno in parser.c can
also be removed, it is used only for the LINENO hack.)
This version produces accurate results: $((LINENO)) was made as accurate
as the LINENO hack made ${LINENO} which is very good. That's why the
LINENO hack is not yet completely removed, so it can be easily re-enabled.
If you can tell the difference when it is in use, or not in use, then
something has broken (or I managed to miss a case somewhere.)
The way that LINENO works is documented in its own (new) section in the
man page, so nothing more about that, or the new options, etc, here.
This version introduces the possibility of having a "reference" function
associated with a variable, which gets called whenever the value of the
variable is required (that's what implements LINENO). There is just
one function pointer however, so any particular variable gets at most
one of the set function (as used for PATH, etc) or the reference function.
The VFUNCREF bit in the var flags indicates which func the variable in
question uses (if any - the func ptr, as before, can be NULL).
I would not call the results of this perfect yet, but it is close.
An initial attempt at implementing LINENO to meet the specs.
Aside from one problem (not too hard to fix if it was ever needed) this version
does about as well as most other shell implementations when expanding
$((LINENO)) and better for ${LINENO} as it retains the "LINENO hack" for the
latter, and that is very accurate.
Unfortunately that means that ${LINENO} and $((LINENO)) do not always produce
the same value when used on the same line (a defect that other shells do not
share - aside from the FreeBSD sh as it is today, where only the LINENO hack
exists and so (like for us before this commit) $((LINENO)) is always either
0, or at least whatever value was last set, perhaps by
LINENO=${LINENO}
which does actually work ... for that one line...)
This could be corrected by simply removing the LINENO hack (look for the string
LINENO in parser.c) in which case ${LINENO} and $((LINENO)) would give the
same (not perfectly accurate) values, as do most other shells.
POSIX requires that LINENO be set before each command, and this implementation
does that fairly literally - except that we only bother before the commands
which actually expand words (for, case and simple commands). Unfortunately
this forgot that expansions also occur in redirects, and the other compound
commands can also have redirects, so if a redirect on one of the other compound
commands wants to use the value of $((LINENO)) as a part of a generated file
name, then it will get an incorrect value. This is the "one problem" above.
(Because the LINENO hack is still enabled, using ${LINENO} works.)
This could be fixed, but as this version of the LINENO implementation is just
for reference purposes (it will be superseded within minutes by a better one)
I won't bother. However should anyone else decide that this is a better choice
(it is probably a smaller implementation, in terms of code & data space then
the replacement, but also I would expect, slower, and definitely less accurate)
this defect is something to bear in mind, and fix.
This version retains the *BSD historical practice that line numbers in functions
(all functions) count from 1 from the start of the function, and elsewhere,
start from 1 from where the shell started reading the input file/stream in
question. In an "eval" expression the line number starts at the line of the
"eval" (and then increases if the input is a multi-line string).
Note: this version is not documented (beyond as much as LINENO was before)
hence this slightly longer than usual commit message.
When we record an arithmetic expression ($(( ))) as being quoted, what matters is the quoting state just before we switch into arithmetic syntax parsing mode, not the state after... This fixes the regiression introduced earlier today (UTC) where quoted arithmetic expressions were being subjected to word splitting.
Fixes to shell expand (that is, $ stuff) from FreeBSD (implemented
differently...)
In particular ${01} is now $1 not $0 (for ${0any-digits})
${4294967297} is most probably now ""
(unless you have a very large number of params)
it is no longer an alias for $1 (4294967297 & 0xFFFFFFFF) == 1
$(( expr $(( more )) stuff )) is no longer the same as
$(( expr (( more )) stuff )) which was sometimes OK, as in:
$(( 3 + $(( 2 - 1 )) * 3 ))
but not always as in:
$(( 1$((1 + 1))1 ))
which should be 121, but was an arith syntax error as
1((1 + 1))1
is meaningless.
Probably some more. This also sprinkles a little const, splits a big
func that had 2 (kind of unrelated) purposes into two simpler ones,
and avoids some (semi-dubious) modifications (and restores) in the input
string to insert \0's when they were needed.
NFC (normal builds): DEBUG only change - convert parser to newer trace method. parser tracing is useful when debugging the parser (which admittedly is fairly often...) but there is a lot of it, and it gets in the way when looking at something else. Now we can turn it off when not wanted.
More standard (and saner) implementation of the ! reserved word.
Unless the shell is compiled with the (compilation time) option
BOGUS_NOT_COMMAND (as in CFLAGS+=-DBOGUS_NOT_COMMAND) which it
will not normally be, the ! command (reserved word) will only
be permitted at the start of a pipeline (which includes the
degenerate pipeline with no '|'s in it of course - ie: a simple cmd)
and not in the middle of a pipeline sequence (no "cmd | ! cmd" nonsense.)
If the latter is really required, then "cmd | { ! cmd; }" works as
a standard equivalent.
In POSIX mode, permit only one ! ("! pipeline" is ok. "! ! pipeline" is not).
Again, if needed (and POSIX conformance is wanted) "! { ! pipeline; }"
works as an alternative - and is safer, some shells treat "! ! cmd" as
being identical to "cmd" (this one did until recently.)
Resolve conflicts from previous merge (all resulting from $NetBSD keywork expansion)
NFC: changes to comments only - expand/add comments relating to ${#...}
parsing, and all its peculiarities.
Fix some parser weirdness...
${#VAR:-foo} (or any other modifier on ${#VAR} is a syntax error.
On the other hand ${##} is not, nor is ${##13} though they mean
quite different things (the latter is an idiom everyone should learn,
... $# except we refuse to admit the possibility that it is 13...
Even I cannot explain what ${#-foo} used to do, but it wasn't sane!
(It should be just $# as $# is never unset, but ...)
Shell syntax is truly a wondrous thing!
Sync with HEAD
NFC: Whitespace, KNF, and (some) consistency.
If we are going to permit ! ! pipeline (And for now the other places where ! is permitted) we should at least generate the logically correct exit status: ! ! (exit 5); echo $? should print 1, not 5. ksh and bosh do it this way - and it makes sense. bash and the FreeBSD sh echo "5" (as did we until now.) dash, zsh, yash all enforce the standard syntax, and prohibit this.
Remove bogus extra \n from syntax error message.
Implement the ';&' (used instead of ';;') case statement list terminator which causes fall through the to command list of the following pattern (wuthout evaluating that pattern). This has been approved for inclusion in the next major version of the POSIX standard (Issue 8), and is implemented by most other shells. Now all form a circle and together attempt to summon the great wizd in the hopes that his magic spells can transform the poor attempt at documenting this feature into something rational...
Fix the heredoc line counting bug that I caused when the heredoc processing was changed just over a year ago (rev 1.111).
Deal with \newline line continuations more correctly.
They can occur anywhere (*anywhere*) not only where it
happens to be convenient to the parser...
This fix from FreeBSD (thanks again folks).
To make this work, pushstring()'s signature needed to change to allow a
const char * as its string arg, which meant sprinkling some const other
places for a brighter appearance (and handling fallout).
All this because I wanted to see what number would come from
echo $\
{\
L\
I\
N\
E\
N\
O\
}
and was surprised at the result! That works now...
The bug would also affect stuff like
true &\
& false
and all kinds of other uses where the \newline occurred in the
"wrong" place.
An ATF test for sh syntax is coming... (sometime.)
Pull up following revision(s) (requested by kre in ticket #1212):
bin/sh/parser.c: revision 1.114 via patch
PR bin/51027 - fix the parsing of references to shell parameters
when given without braces (ie: $2 etc). Only the first 9 shell
parameters ($1 .. $9) and the special parameter ($0) can be
referenced this way, $10 is ${1}0 not ${10}. Make it so.
This bug brought to notice by Sven Mascheck's web pages which
discuss (among other things) the history of this (and other ash
based) shells .. see http://www.in-ulm.de/~mascheck/ (from kre@)
Pull up following revision(s) (requested by kre in ticket #1397):
bin/sh/parser.c: revision 1.114 via patch
PR bin/51027 - fix the parsing of references to shell parameters
when given without braces (ie: $2 etc). Only the first 9 shell
parameters ($1 .. $9) and the special parameter ($0) can be
referenced this way, $10 is ${1}0 not ${10}. Make it so.
This bug brought to notice by Sven Mascheck's web pages which
discuss (among other things) the history of this (and other ash
based) shells .. see http://www.in-ulm.de/~mascheck/ (from kre@)
PR bin/51145 PR bin/48489
More fixes to the shell parser to prevent empty simple commands (where
empty means no significant text at all) - as discussed on tech-userlevel
this still allows { } (which can be useful in function definitions, not
really anywhere else though) except in posix mode. ( ) now generates
a syntax error, as should any other place where commands are required but
nothing is present. (nb, redirections, var assignments, even var expansions
that expand to nothing, are all OK, and avoid the error - just comments, or
whits space, are not.) This is (aside from allowing { } at all) all in
accordance with the posix spec.
PR bin/48489 -- Shell "simple commands" are now not allowed to be completely empty, they must have something (var assignment, redirect, or command, or multiple of those) to avoid a syntax error. This matches the requirements of the grammar in the standard. Correct the parser (using FreeBSD's sh as a guide) and update the man page to remove text that noted a couple of places this was previously OK. OK christos@
Allow function names to be any shell word not containing '/'. This allows anything that could be a filesystem command to be implemented as a function instead. The restriction on '/' is because of the way that functions are (required to be) searched for relative to PATH searching - a function with a name containing '/' could never be executed, so simply prohibit defining such a thing. ok christos@
Fix handing of user file descriptors outside the 0..9 range. Also, move (most of) the shell's internal use fd's to much higher values (depending upon what ulimit -n allows) so they are less likely to clash with user supplied fd numbers. A future patch will (hopefully) avoid this problem completely by dynamically moving the shell's internal fds around as needed. (From kre@)
Allow a heredoc to be positioned outside a `` command substitution. POSIX just says "here docs begin after the next newline [token]". Nothing about "provided it is inside any `` the redirect operator appears in... As best I can tell, NetBSD now has the only shell to handle this "correctly" (which raises the question whether it is correct - but if not, only erroneous scripts are affected.) This is required by some (probably broken) autoconfigure related scripts. (from kre@)
After discussions with Jilles Tjoelker (FreeBSD shell) and following a suggestion from him, the way the fix to PR bin/50993 was implemented has changed a little. There are three steps involved in processing a here document, reading it, parsing it, and then evaluating it before applying it to the correct file descriptor for the command to use. The third of those is not related to this problem, and has not changed. The bug was caused by combining the first two steps into one (and not doing it correctly - which would be hard that way.) The fix is to split the first two stages into separate events. The original fix moved the 2nd stage (parsing) to just immediately before the 3rd stage (evaluation.) Jilles pointed out some unwanted side effects from doing it that way, and suggested moving the 2nd stage to immediately after the first. This commit makes that change. The effect is to revert the changes to expand.c and parser.h (which are no longer needed) and simplify slightly the change to parser.c. (from kre@)
PR bin/51027 - fix the parsing of references to shell parameters
when given without braces (ie: $2 etc). Only the first 9 shell
parameters ($1 .. $9) and the special parameter ($0) can be
referenced this way, $10 is ${1}0 not ${10}. Make it so.
This bug brought to notice by Sven Mascheck's web pages which
discuss (among other things) the history of this (and other ash
based) shells .. see http://www.in-ulm.de/~mascheck/ (from kre@)
This is an internally visible change - no effect visible to the user is expected. With the previous commits to parser.c, we no longer need to handle reading here documents in the (massive) readtoken1() function. That allows its code to be simplified and made easier to read and understand (several goto's goto goto heaven. RIP) (from kre@)
Move the parseredir internal subroutine out of readtoken1() into being a real function of its own (also inspired by FreeBSD - though for this one other sh differences require slightly different code.) (from kre@)
PR bin/50993 - this is a significant rewrite of the way that here documents are processed. Now, when first detected, they are simply read (the only change made to the text is to join lines ended with a \ to the subsequent line, otherwise end marker detection does not work correctly (for here docs with an unquoted endmarker only of course.) This patch also moves the "internal subroutine" for looking for the end marker out of readtoken1() (which had to happen as readtoken1 is no longer reading the here doc when it is needed) - that uses code mostly taken from FreeBSD's sh (thanks!) and along the way results in some restrictions on what the end marker can be being removed. We still do not allow all we should. (from kre@)
Cease "support" for <redirect> fn() { ...
Any redirect (or redirects) before a function definition were
allowed by the parser, but otherwise totally ignored. The standard
syntax does not permit redirects there, now, neither do we. (from kre@)
Finish constifying the new parsebackquote() function. Save a variable or two... Should change nothing. (from kre@)
General KNF and source code cleanups, avoid scattering the magic string " \t\n" all over the place, slightly improved syntax error messages, restructured some of the code for clarity, don't allow IFS to be imported through the environment, and remove the (never) conditionally compiled ATTY option. Apart from one or two syntax error messages, and ignoring IFS if present in the environment, this is intended to have no user visible changes. (from kre@)
fix constness (from kre)
Move the command substitution "internal subroutine" part of readtoken1() into a real function of its own (inspired by a similar change made by FreeBSD - the other internal routines they moved out are expected to move out here as well soon.) This change helps avoid gcc 5.3 demanded (not required!) volatile noise which would slow down parsing. (from kre)
sprinkle more volatile (needed for the rescue build with gcc-5.3)
put back some volatile, gcc complains (x86_64)
Remove most volatility from readtoken1() restoring execution speed to approximately where it was before the changes made in version 1.96 (which assumed this would eventually happen.)
Bug fixes to handling of unterminated here documents (they should be, and now are, a syntax error), and miscellaneous other minor cleanups. (from kre)
remove useless casts
CID 1354293: handle EOF
Fix quoting inside heredoc's. (from kre)
PR bin/43469 - correctly handle quoting of the pattern part of ${var%pat}
type expansions. (from kre)
Fix for PR bin/48631 - allow commands controlled by case statements to be nothing more than redirects (from kre)
Finish the fix for PR/48631 - that is, make the parser correctly handle the token syntax it really should be handling (including some that posix does not require, but is right anyway.) This is quite similar to, and to some extent inspired by the way the FreeBSD sh parser.c works, but the actual implementation is quite different. (from kre)
KNF / whitespace fixes. No changes of substance. (from kre)
PR/50827: Richard Hansen: Fix default variable assignment with arithmetic, from kre.
remove unused assignment
Eat trailing backslash like bash and pdksh (not zsh). CBACK+CEOF = TEOF
Rebase to HEAD as of a few days ago.
PR/49125: Havard Eidnes: /bin/sh does not support redirecting to or from FDs > 9 According to: http://pubs.opengroup.org/onlinepubs/009604599/utilities/xcu_chap02.html#tag_02_07 Redirection support for fds > 9 is optional but allowed.
sync with head.
for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.
this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
whitespace fixes
There was a case where \n did not increase plinno
clarify further.
explain the previous fix.
allow case statement without any patterns.
add stdio.h
add crude $LINENO support for FreeBSD
sync with head
Initialize two variables for clang.
sync with head
include <limits.h> for CHAR_MIN/CHAR_MAX
PR/43597: Don't break from parsing word tokens in we are in double quotes.
Fixes: sh -c 'echo "${foo:="first-word"} second-word"'
NULL does not need a cast
PR/44229: Henning Petersen: Remove dup check for whitespace.
revert again, since this breaks libtool amongst other things.
PR/43469: Antii Kantee: test/util/sh/t_expand:strip fails.
Bring back fixes from revision 1.75:
- Fix a couple of bugs to make the following two echo statements print the
same output as they should:
line='#define bindir "/usr/bin" /* comment */'
echo "${line%%/\**}"
echo ${line%%/\**}
1. ISDBLQUOTE() was not working properly for non VSNORMAL expansions because
varnest was incremented before the variable was completely parsed. Add
an insub adjustment to keep track of that.
2. When we have a quoted backslash, we either need to escape twice, because
one level of escaping will be stripped later (in the variable substitution
case) or simply enter the backslash.
revert previous. breaks other stuff.
- Fix a couple of bugs to make the following two echo statements print the
same output as they should:
line='#define bindir "/usr/bin" /* comment */'
echo "${line%%/\**}"
echo ${line%%/\**}
1. ISDBLQUOTE() was not working properly for non VSNORMAL expansions because
varnest was incremented before the variable was completely parsed. Add
an insub adjustment to keep track of that.
2. When we have a quoted backslash, we need to escape twice, because one
level of escaping will be stripped later. (XXX: Do that when insub == 1
only?)
- Make macros statements
fix -Wsign-compare issues
and if you don't succeed twice, try again.
Try a different fix for PR/11317: Don't ignore errors in list().
Break PR/11317 again. The counting parentheses fix does not handle case statements properly. Fixes PR/39873.
PR/11317: Hubert Feyrer: Recognize mismatched parentheses inside old style command substitution.
Sync with wrstuden-revivesa-base-2.
Fix here documents that end abruptly without NL before EOF. (Andy Shevchenko)
sync with head.
Detect unmatched quotes inside old style command substitution. echo `"`
sync with head.
sync with HEAD
Change spaces to tabs for consistency with adjacent lines.
sync with HEAD
convert __attribute__s to applicable cdefs.h macros
Sync w/ NetBSD-4-RC_1
Pull up following revision(s) (requested by dsl in ticket #718): bin/sh/parser.c: revision 1.64 PR/35410: Valeriy E. Ushakov: /bin/sh mishandles shell function definitions with function names that are not plain words 1. remove the escape annotations from the function name. 2. check if the function has a valid name before storing it.
Pull up following revision(s) (requested by dsl in ticket #719): bin/sh/parser.c: revision 1.65 Fix sh -c 'true && ! true | false; echo $?' Add some more TRACE((...)) calls to aid such debugging. Fixes PR bin/36435 Clearly no one tried this test when the changes of rev 1.31 and 1.44 were done!
Fix sh -c 'true && ! true | false; echo $?' Add some more TRACE((...)) calls to aid such debugging. Fixes PR bin/36435 Clearly no one tried this test when the changes of rev 1.31 and 1.44 were done!
PR/35410: Valeriy E. Ushakov: /bin/sh mishandles shell function definitions with function names that are not plain words 1. remove the escape annotations from the function name. 2. check if the function has a valid name before storing it.
sprinkle volatile.
Prefix "Syntax Error" with the program name, if the command name is not available. At least this way we get an idea of what program gives us the message.
PR/33775: YAMAMOTO Takashi: /bin/sh doesn't accept "(" before case patterns
quell GCC 4.1 uninitialised variable warnings. XXX: we should audit the tree for which old ones are no longer needed after getting the older compilers out of the tree..
Back out previous, amongst other things it breaks $((0x10))
Treat $((x)) as equivalent to $(($x)) - posix seems to require this now.
Fix treatment of ' inside a 'here document' with a quoted EOF marker. Fixes a breakage from the previous version.
Correctly apply IFS to unquoted text in ${x-text}.
Fixes PR/26058 and the 'for i in ${x-a b c}; do ...' and ${x-'a b' c}.
I can't find a PR for the latter problem.
Regression test goind in shortly.
Move UCB-licensed code from 4-clause to 3-clause licence. Patches provided by Joel Baker in PR 22249, verified by myself.
Fixes from David Laight: - ansification - format of output of jobs command (etc) - job identiers %+, %- etc - $? and $(...) - correct quoting of output of set, export -p and readonly -p - differentiation between nornal and 'posix special' builtins - correct behaviour (posix) for errors on builtins and special builtins - builtin printf and kill - set -o debug (if compiled with DEBUG) - cd src obj (as ksh - too useful to do without) - unset -e name, remove non-readonly variable from export list. (so I could unset -e PS1 before running the test shell...)
implement noclobber. From Ben Harris, with minor tweaks from me. Two unimplemented comments to go. Go Ben!
enami convinced me that it is a good idea to use the first word of the allocated area.
Since we should be able to handle nested double quotes, don't use the syntax maps to determine the beginning and end quotes (kill CENDQUOTE). Handle single quotes opening and closing via checking the current syntax map. Keep a bitmap of doublequote state one bit per variable nesting level. For the first 32 nested double quotes, we don't need any additional memory, but for more we allocate dynamically.
back this directory up a day, systems won't even boot (rc.subr splodes)
suggested back-to-the-drawing-board test: $ echo "${PWD:-notlikely}"
PR/15579: Alan Barrett: }'s inside variable specs were taken into account
even if quoted:
foo=${foo:-"'{}'"}; echo $foo
would display '{'} instead of '{}'.
Pull up revision 1.48 (requested by hubertf): Prevent alias expansion in case labels, where it is unwanted.
Fix switch alias handling. Inspired from FreeBSD, but corrected to handle alias expansion inside the switch as appropriate. This is achieved by a flag noalias which is turned on and off in as we parse. In the following example [1] and [0] indicate the value of noalias. [0] case <expr> in [1] <lit> ) [0] <expr> ;; [1] <lit> ) [0] <expr> ;; ... [1] esac [0] FreeBSD does: [0] case <expr> in [1] <lit> ) <expr> ;; <lit> ) <expr> ;; ... esac [0] This handles the following shell script: alias a=ls case $1 in a) echo a; a;; f) echo f;; *) echo default;; esac
PR/12533: Koji Mori: eval gets misparsed when it has a trailing semi-colon.
remove redundant declarations and nexted externs.
un-__P functions declared in parser.h. host programs include parser.h, and so it shouldn't use __P. (this should probably be done better, by not declaring the parser functions in headers used by host programs, but this works well enough.)
Fix bin/9184, bin/9194, bin/9265, bin/9266 Exitcode and negation problems (From Martin Husemann)
compile with WARNS = 2
PR/4966: Joel Reicher: Implement <> redirections which are documented in the man page.
Patches from Tor Egge (via Havard Eidnes) to fix various bugs in field splitting and combining. (Note: Some of this are not strictly bugs, but differences between traditional Bourne shell and POSIX.)
Fixed memory leak on old style command substitution such as sh -c 'echo `echo foo`' . The memory allocated with ckmalloc() at parser.c:1349:readtoken1() (search for "done:" label) was never freed. I changed this to use 'string stack' framework of Ash. Note that a string on string stack is properly freed on exception and end of command parsing, and no explicit free or signal handlings required. See TOUR for an overview, and memalloc.[ch] for details of string stack.
Be more retentive about use of NOTREACHED and noreturn.
Sync with trunk, per request of christos.
- change "extern" variables into int's - remove extern'd variables not actually referenced - don't use char as an array index
Fix Sparcworks warnings.
Fix compiler warnings.
NO_HISTORY->SMALL
Pullup request 1.33 -> 1.34 from Christos Zoulas.
"Fix parsing problem introduced in the previous ${10} fix, where $#digit
or $digit# would get mis-parsed as a positional parameter."
Fix parsing problem introduced in the previous ${10} fix, where $#digit
or $digit# would get mis-parsed as a positional parameter.
Update /bin/sh from trunk per request of Christos Zoulas. Fixes many bugs.
enable parsing of multi-digit positional arguments i.e. ${10} works and it
is not a bad substitution.
kill 'register'
- fix parsing bug reported by mycroft: ! was only recognized in the beginning of pipelines. Now we are recognizing it everywhere like ksh, so: ! if [ a = b ]; then ! echo c && ! true; else ! echo d; fi works.
PR/2808: - detect eof inside backquotes - handle && and || shortcircuits properly - tokens.def -> tokens.h (from FreeBSD)
Close PR/2384 backquoted backslash-newline was not eaten.
- parser.c: Fix prompting in old style backquote expansion. Fixes PR/2139 and many user complaints why the shell hangs in echo "`" - eval.c: Fix exitstatus invalid resetting in `if' statements were: if (exit 3); then echo foo $? else echo bar $? fi printed 'bar 0' instead of bar 3
- fix PR1620, -DNO_HISTORY did not work. - restore parsing state after parsing old style command substitution. The ';' in '`echo z;`' broke the following: for i in 1; do cat > /dev/tty << __EOF__ `echo z;` __EOF__ done cVS: Enter Log. Lines beginning with `CVS: ' are removed automatically
Fixed bug where ${#} was interpreted incorrectly as ${#var} and thus
misparsed. Keith Bostic reported it.
Merge in my changes from vangogh, and fix the x=`false`; echo $? == 0 bug.
convert to new RCS id conventions.
I added the documented in the manual but not implemented variable expansions:
${#WORD}
${WORD%PAT}
${WORD%%PAT}
${WORD#PAT}
${WORD##PAT}
clean up further. more patches from Jim Jegers
Eliminate uses of some obsolete functions.
update from trunk
Fix a core dump and another parse error related to null commands.
from trunk: Fix some problems with empty commands.
Fix some problems with empty commands.
From Christos: 1. Fix `-' quoting in [ ] expressions. 2. Fix expansion of variables in redirections
Add RCS ids.
Fixed another case statement parsing bug that was introduced by my last fix :-(. Thanks to Theo for letting me know about it...
Re-worked case statement parsing. With luck, this fixes bug #268.
a few more things to omit when NO_HISTORY defined. from noel@cs.oberlin.edu
Fix backslash parsing within backquoted string
Mark Weaver's background/list fix (for bug #236)
sync with 4.4lite
44lite code
from jim wilson: (command): Handle TEOF like TNL.
Make `>file;' work.
Add RCS identifiers.
Another change from Jim Wilson.
IEEE 1003.2 (D11.2.2.3) requires that the system's true and false be accessed instead of searching $PATH. The best way to satisfy this requirement is to make them builtins. True was allready builtin, this patch adds false.
Jim "wilson@moria.cygnus.com" Wilson's patches to make C News (and other things) work.
Fixed incorrect calls to longjmp.
changed "Id" to "Header" for rcsids
added rcs ids to all files
initial import of 386bsd-0.1 sources
Initial revision