Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0001006 [1003.1(2013)/Issue7+TC1] Shell and Utilities Editorial Omission 2015-11-13 08:23 2019-10-21 09:43
Reporter rhansen View Status public  
Assigned To
Priority normal Resolution Accepted As Marked  
Status Applied  
Name Richard Hansen
Organization BBN
User Reference
Section 2.5.3 PS1
Page Number 2326
Line Number 73833
Interp Status Approved
Final Accepted Text Note: 0003310
Summary 0001006: PS1 should be subject to command substitution and arithmetic expansion
Description The specification of PS1 only says that it is subject to parameter expansion. A non-exhaustive survey of shells suggests that most implementations also perform command substitution and arithmetic expansions on PS1. These additional expansions ought to be included.
Desired Action On page 2326 lines 73832-73834 (XCU 2.5.3 PS1), change:
PS1 Each time an interactive shell is ready to read a command, the value of this variable shall be subjected to parameter expansion and written to standard error.
to:
PS1 Each time an interactive shell is ready to read a command, the value of this variable shall be subjected to parameter expansion, command substitution, and arithmetic expansion, and the results shall be written to standard error.
Tags tc3-2008
Attached Files

- Relationships
related to 0001038Applied 1003.1(2013)/Issue7+TC1 Specification relies upon unspecified behaviour 
related to 0001511Applied Issue 8 drafts LINENO description claims to be a User Portability Utilities option var, but has no margin tag or shading 

-  Notes
(0002906)
joerg (reporter)
2015-11-13 10:34

Looks like a security problem.

This kind of expansions are risky unless they cannot be inherited via the environment and the dot scripts have been verified to be writable only by the user who needs to be the owner.
(0002907)
kre (reporter)
2015-11-13 13:51

The text for PS1 also says ...

   The shell shall replace each instance of the character '!' in PS1 with
   the history file number of the next command to be typed.

but it doesn't give any indication of how the two replacements are intended
to interact with each other.

For example, after executing PS1='[$!] ' if the current history file number
(for the next command) is 2, and the last command executed (history file number 1) was sleep 100 & which runs as pid 1234, just what is supposed to be
printed as the prompt?

Does the ! expand first, producing PS1='[$2] ' ? Or does the parameter
expansion occur first, resulting in PS1='[1234] ' ? Or something else?

If parameter expansion is first, and it produces a ! as part of the value
expanded, is that ! expanded to give the history file number, or only a !
that is literally in PS1 ?

As it is now, this whole thing is too underspecified to be useful, and it would
be entirely reasonable for a shell to simply echo $PS1 with no expansion at all.
(0002909)
user229
2015-11-13 15:44
edited on: 2015-11-13 16:04

Regarding kre's question: ksh93 and zsh in ksh-mode do the latter; mksh does the former. Ksh93's behavior certainly seems more useful. Incidentally, I noticed that $! when there is no process is a blank string in ksh93 and "0" in zsh. It's not clear what the intended behavior is in this case.

Other shells I tested (bash, dash) lack the !-substitution feature.

(0002911)
geoffclare (manager)
2015-11-13 16:11

bash --posix does !-substitution, and does it before parameter expansion.
(0002913)
shware_systems (reporter)
2015-11-13 16:33

By the longest-match lexical scanning convention, I would think '$!' being a parameter expansion is more 'correct', in the absence of any statement saying '!' will be scanned for first like with tilde expansions. Changing the text to "the character ! (that does not follow a $)" would also make it explicit.
(0002914)
user229
2015-11-13 16:36
edited on: 2015-11-13 16:49

Just to add another data point - ksh93 and zsh do !-substitution _after_ expansion; e.g. x='!' PS1='$x>', or PS1='`printf \\041`>', is equivalent to PS1='!>'. So no examples of those I surveyed do it "simultaneously", even though the theoretical "simultaneous" behavior would probably be the least surprising.

And since the standard already specifies parameter expansion, this is a problem in the existing standard, rather than merely in the proposal to add command substitution and arithmetic expressions.

(0002918)
kre (reporter)
2015-11-13 18:21

Re: this is a problem in the existing standard, rather than merely n the proposal

Apologies if I did things the wrong way, I an unused to the correct process
here, if some other way of reporting the issue exists, please let me know
(you can do that via e-mail to kre@munnari.oz.au to avoid clogging this forum.)

However, the issue isn't irrelevant to the proposal - if $() or $(())
expansion are added to PS1 processing, the issue of what happens with !
becomes rather more important that the (rather contrived) use of $! in PS1

Since ! is not an unusual character to use in command args, and if less
common perhaps, is legal in arith exprs (as unary ! or part of !=) users
would need to know whether they need to enter those !'s as !! or not.
(0002919)
kre (reporter)
2015-11-13 18:29

Re: Changing the text to "the character ! (that does not follow a $)" would also make it explicit.

That really solves nothing. Consider instead ${!} or ${VAR%!*} or
a zillion other possible ways that using ! in a var expansion expression
might occur.

The actual order of evaluation needs to be made clear ... or given that the
aim should be to specify what shells actually do (rather than attempt to
invent what they should do) and given the different implementation approaches
that have been demonstrated here, perhaps the correct approach is simply to
say that it is implementation defined (or even undefined) what happens when
both expansions are attempted.
(0002920)
user229
2015-11-13 18:31

I'm not an "insider" here, I don't know if there are any process issues (If anything I'd think it would just be another bug report), I was just pointing out that there's still the matter of how !-substitution interacts with parameter expansion (it doesn't just affect $!, but also any variable which may contain ! in its value, or any command with it in its output if this is added), just to make sure the problem doesn't "go away" by rejecting this on these grounds. Sorry for the confusion.
(0002932)
rhansen (manager)
2015-11-14 01:39

Regarding the security concern, the standard already requires support for
PS1='${PS1+$(some command here)}'
so adding command substitution should not open up any new vulnerabilities.

This is relevant: https://github.com/rhansen/clonepwn [^]
(0002933)
rhansen (manager)
2015-11-14 01:40
edited on: 2015-11-14 01:44

I agree that the specification for ! is unclear. We could file a new bug report to deal with !, but it might be easier to just address everything together in this bug report.

Another data point: dash doesn't do ! expansion in PS1.

We've heard that ksh does ! expansion after the other expansions and bash --posix does ! expansion before the other expansions. Has anyone found an implementation that does ! expansion simultaneously with the other expansions?

Some additional issues:
  • If we assume that ! expansion happens simultaneously with the other expansions in a beginning-to-end fashion, does ! expansion recurse? For example, should PS1='${PS1+!!}' yield a single exclamation point or two? What about PS1='$(printf %s !!)'?
  • If ! expansion happens simultaneously and recurses, how should ! be interpreted in arithmetic expansion? Should PS1='$((! - 1))' be just like PS1='$(($! - 1))' or should it yield the previous history number?
  • There is no definition of "history file number" and there is no cross-reference to the sh extended description section, so the expansion of ! is not as clear as it should be.
  • The last sentence of the descriptions of LINENO, PPID, PS1, PS2, and PS4 say that those variables only have meaning if the User Portability Utilities option is supported, but there is no UP shading anywhere. I prefer the approach in the description of the ENV variable.


These are the questions I think readers need to be able to answer (and it's OK if the answer is "unspecified" as long as its clearly unspecified):
  • which expansions are performed? (just parameter and !, or also command and arithmetic?)
  • when does ! expansion occur? (before, after, or simultaneous?)
  • if ! expansion is simultaneous:
    • how is PS1 is parsed? (where do ambiguous ! characters belong? beginning-to-end longest match?)
    • does ! expansion recurse into the other types of expansions?
    • if ! expansion recurses, how is ! handled in arithmetic expansions? (as if it was $! or is it the history number?)


Note that we can resolve this one way for Issue 7 (e.g., make it clear that the answer to every question above is "unspecified") and a different way for Issue 8.

Here's some example wording that would leave the behavior unspecified everywhere, except I assume that ! expansion never happens simultaneously with the other expansions (otherwise the wording gets really complicated):
On page 2326 lines 73832-73841, replace the entire description of PS1 with:
[UP]PS1 The processing of the PS1 shell variable shall be supported if the system supports the User Portability Utilities option.[/UP]

Each time an interactive shell is ready to read a command, the value of this variable shall be subjected to parameter expansion ([xref to Section 2.6.2]) and exclamation-mark expansion (see below). Whether the value is also subjected to command substitution ([xref to Section 2.6.3]) or arithmetic expansion ([xref to Section 2.6.4]) or both is unspecified.

The expansions shall be performed in two passes, where the result of the first pass is input to the second pass. One of the passes shall perform only the exclamation-mark expansion described below. The other pass shall perform the other expansion(s) according to the rules in [xref to Section 2.6]. Which of the two passes is performed first is unspecified.

The default value shall be <tt>"$ "</tt>. For users who have specific additional implementation-defined privileges, the default may be another, implementation-defined value.

Exclamation-mark expansion: The shell shall replace each instance of the <exclamation-mark> character (<tt>'!'</tt>) with the history file number ([xref to sh extended description]) of the next command to be typed. Escaping the <exclamation-mark> character with another <exclamation-mark> character (that is, <tt>"!!"</tt>) shall place the literal <exclamation-mark> character in the prompt.


I would like things to be a bit more specified than this, however. In particular, I want support for command substitution and arithmetic expansion to be required:
Each time an interactive shell is ready to read a command, the value of this variable shall be subjected to parameter expansion ([xref to Section 2.6.2]), command substitution ([xref to Section 2.6.3]), arithmetic expansion ([xref to Section 2.6.4]), and exclamation-mark expansion (see below).


(0002934)
rhansen (manager)
2015-11-14 01:50

Oh, it looks like PS2 and PS4 might also need command substitution and arithmetic expansion. Fortunately they don't do ! expansion.
(0002939)
geoffclare (manager)
2015-11-14 10:06

(Response to Note: 0002932)
My interpretation of the current standard is that it requires
PS1='${PS1+$(echo foo)}'
to result in the prompt written to stderr being the literal string "$(echo foo)", and that is what /usr/xpg4/bin/sh on Solaris and /bin/sh on HP-UX do.
(0002940)
kre (reporter)
2015-11-14 13:43
edited on: 2015-11-15 07:55

Re: how is ! handled in arithmetic expansions? (as if it was $! or is it the history number?)

If it is/was meant to be handled as $! then no shell I have access to (which
does not include ksh93 or, right now, dash) conforms in any case (that is,
regardless of it being assigned to PS1, just as an arith expr). ! in arith
exprs is (apart from some shells where it is being treated as a history lookup
request, and as quoting is disabled in $(()) that seems difficult to avoid
cleanly) entirely seen as the unary ! operator, even in cases where that clearly
is not rational like $((!)). The same is true for '?', except for zsh which
does turn $((?)) into $? it appears (it might handle $((!)) as well if I could
work our how to stop it using the ! as a history lookup request.)

I believe that, as ! and ? are sh variables, and contain valid integer
constants (at least assuming a background process has previously been
started so the pid will be in $! and a process has terminated so $? has
an exit status) 2.6.4 requires $((!)) to be ${!} and $((?)) to be ${?}
regardless of almost no shells implementing that, preferring instead to
treat the ! as unary not (or history request) and ? as a part of the ?:
operator.

This is something else that ought to be fixed. I suspect 2.6.4 should
probably be restricted to apply only to var names that start with an
alpha char...

[I also tested $((@)) and $((*)) - after "set -- 5" so $@ and $* are both 5)
and those also don't work in any shell I can test - not even zsh handles those.]
[I had forgotten $# ... tested that one now too, again, only zsh of the shells
I have access to treats $((#)) as ${#}, most of the others give a syntax error,
but mksh treats $((#)) as 0 (regardless of the value of $#) .. I am guessing
it might be treating the # as a comment indicator.]

In any case, the issue for arith expansion in PS1 (if it gets added) is
more whether $(( ! x )) turns into $(( 50 x )) or something (and so,
invalid) or whether it means "not x", rather than whether it means one of
$(( 50 x )) or $(( 1234 x )) (where history number == 50, $! == 1234)

(0002941)
user229
2015-11-14 14:01

> it might handle $((!)) as well if I could work our how to stop it using the ! as a history lookup request.

echo $((\!)) in zsh mode seems to work for this, and gives "zsh: bad math expression: operand expected at end of string".

Whether PS1='$((!))' is handled as the history number or a syntax error depends on the shell, same as in the other cases - in ksh and zsh it is not, and in bash and mksh it is. No existing implementation seems to "recurse" anyway except in the trivial version of the second expansion applying to the output of the first.
(0002944)
shware_systems (reporter)
2015-11-14 17:26
edited on: 2015-11-14 17:27

How about:
"The shell shall convert occurences of '!' not escaped by '\' and not part of a '$' prefixed expansion to decimal digits representing the next history index number, along with any parameter expansions, etc., in left-to-right order. It is implementation-defined whether a parameter expansion is available that also provides the history index number. The results of expansions shall not be rescanned for '!' substitutions. If an expansion produces an error condition a space character shall be used as the expansion result."

Using the above,
PS1=ID=! \! $!
would produce
ID=123 ! <value of special param !>
as output.

In the context of PS1, '$$' could be a standard-defined special parameter expansion of the history id inside expansions too. ${$} would still be a syntax error in this case. It would allow a $(($$ - 1)) expansion to report the prior index, at least.

(0002945)
user229
2015-11-14 21:03
edited on: 2015-11-14 21:04

$$ already means the process ID. Why does it have to be a single character, anyway, not something like ${.sh.histnum}? And as-is, ! is escaped in PS1 with !!, not \!. Really the whole ! feature seems a little bit "80s" and not how the feature "ability to put history number in the prompt" would be implemented in a modern shell that already had parameter expansion (AFAIK this is a csh-ism originally?)

(0002946)
rhansen (manager)
2015-11-15 03:26
edited on: 2015-11-15 03:43

(Response to Note: 0002939)

2.5.3 currently says that PS1 is subjected to parameter expansion. Section 2.6.2 (Parameter Expansion) says that word in ${parameter+word} shall be subjected to tilde expansion, parameter expansion, command substitution, and arithmetic expansion. So my interpretation is that
PS1='${PS1+$(echo foo)}'
should yield foo, not $(echo foo).

(0002947)
rhansen (manager)
2015-11-15 04:22

Regarding ! in arithmetic expansions: It slipped my mind that ! was also a unary operator. :) So $((! - 1)) should always yield 0 (logical 'not' of negative one).

This aspect of the problem only becomes relevant if ! expansion is concurrent with the other expansions and that it recurses. These are two big ifs, particularly because I don't think any existing shells do ! expansion at the same time as the other expansions. I think we should table this aspect of the problem until concurrent ! expansion becomes a real possibility for standardization.

I agree that 2.6.4 is not as clear as it should be regarding variables without the $. We should address that, though probably in a different bug report.
(0002948)
kre (reporter)
2015-11-15 07:28

Re: This aspect of the problem only becomes relevant if ! expansion is concurrent

What concerns me is whether (assuming I want the unary logical negation operator,
and assuming $(()) processing is added to PS1 processing) whether I need to
write
      PS1='$(( ! - 1 )): '
or
      PS1='$(( !! - 1 )): '
in order to guarantee that I get "0: " as the prompt, and not the history
number decremented by one. Concurrent processing of $ and ! is not really
the issue (though were it ever attempted, if it is even really possible, I
agree things get quite hairy.)

I will end my brief involvement with this issue with a plea to sh implementers.

Please, if you do what the standard requires currently (handling both var
expansion and ! expansion) then do the ! expansion first, and never after the
var expansion. Doing it the other way means that in the much more common
setting for PS1 than has been being discussed here, of PS1='${PWD} $ '
(or similar), if I change to a directory with ! in its name, instead of
getting $PWD printed in the prompt, I get $PWD with the ! replaced by the
history number. That's not useful.

To avoid that I would need to assume $() processing in PS1 expansion, and then
do PS1='$(printf "%s\n" "${PWD}" | sed 's/!/!!/g') $ '
(or some similar contortion) which is ugly, and would be slow...

Even worse, if the standard is vague about whether ! or $ expansions happen
first, then with some shells, that might turn into
     PS1="$(printf "%s\n" "${PWD}" | sed 's/50/!/g') $ '
which is not at all useful - in this case it obviously is not a problem that
a ! in the directory name doesn't get changed to !! (we would not want that)
but changing the history number to a ! would be bizarre.

Since PS1 can be (and often is) exported through the environment from shell
to shell, this simple task, of including the current directory in the prompt
would become a nightmare of hellish proportions, if implementors don't find
a way to fix this in some rational fashion (which can then be standardised once
it is agreed.)
(0002949)
kre (reporter)
2015-11-15 09:14
edited on: 2015-11-15 09:31

I thought I was finished, but I just noticed another issue that relates to
the specification of PS1 ...

As has been being discussed. PS1 is subject to parameter expansion. The
default value of PS1 is also fixed in 2.5.3 (sorry, I have no idea how to
obtain line numbers from the doc...) as:
              The default value shall be "$ ".

This combination (while what we are familiar with, especially the default)
is weird, as 2.6 says ...

           The '$' character is used to introduce parameter expansion,
           command substitution, or arithmetic evaluation. If an unquoted
           '$' is followed by a character that is not one of the following:
[list omitted, but space is not one of them]
            the result is unspecified.

When PS1 is expanded, the $ in the default value is unquoted, and is not
followed by one of the list of characters that give the $ a defined
interpretation, hence the standard has specified an unspecified value.
This cannot be good...

Note that the quotes on the assignment to PS1 (if any) cannot be part of
the value of PS1 (which would lead to the $ being quoted, and hence not
subject to the "the result is unspecified" text, as otherwise when PS1 is
assigned as PS1='value' no expansions of value would be permitted, and
we know that is not what happens, nor is it the intent.

Alternatively, perhaps the intent of the "default shall be" was as if
              PS1='"$ "'
had been executed, making the $ in PS1 (by default) quoted. If that's
how it is supposed to be read, then every shell I have tested (uniformly)
gets it wrong, and treats the prompt as containing two " characters (in
addition to the $ and space) in that case. I don't believe that can be correct.

I would suggest changing 2.6 to explicitly allow white space or "end-of-string"
after a $, with the interpretation that such a $ is to be treated literally.

It might also be necessary to change step 5 in 2.3 (token recognition) to
something like

      If the current character is an unquoted '`', or unquoted '$' that is
      not followed by a <blank> or <newline>, the shell shall identify ...

I think (I hope) that with that small addition, the rules all end up working
as expected, and the $ would either be appended to the end of the current
word (and then the blank or newline would delimit that token) or become a
token of its own.

(0002950)
user229
2015-11-15 10:44

> making the $ in PS1 (by default) quoted.

Even if that were the case (and even if quotes worked within PS1, as you noted they do not), parameter expansion still happens within double quotes. And the language about being unspecified behavior for other characters [including, technically, space] is repeated in 2.6.2: "If the parameter is not enclosed in braces, and is a name, the expansion shall use the longest valid name (see XBD Name), whether or not the variable represented by that name exists. Otherwise, the parameter is a single-character symbol, and behavior is unspecified if that character is neither a digit nor one of the special parameters (see Special Parameters)." which is linked from the section on double quotes.
(0002952)
geoffclare (manager)
2015-11-16 09:38

(Response to Note: 0002946)

By a strict reading, perhaps you are right, but it is obviously not the intention.

2.6.2 explains parameter expansion in the normal shell command context. When applied to PS1, where the only word expansion performed is parameter expansion, it should be understood as only requiring tilde expansion, command substitution, and arithmetic expansion within a parameter expansion to be performed if the parameter expansion is in a context where those expansions are performed.

I think that this is the only reasonable interpretation. An interpretation which requires that PS1='$(echo foo)' yields '$(echo foo)' but PS1='${PS1+$(echo foo)}' yields 'foo' makes no sense.
(0002953)
joerg (reporter)
2015-11-16 11:22

Let me again mention security here.

Do you remember the "Shellshock" problem?

This was more broadly exploitable but the basic method still applies to a shell that expands variables like PS1.

Note that $ENV is also a potential problem. It does not do command substitition but it allows to enforce to run arbitrary commands from arbitrary scripts if injected into an interactive shell.

For all these "nice" features, I like to see a security discussion and related rationale section in the standard - together with methods to defend against atacks.

The current Bourne Shell e.g. implements "persistent aliases" that are stored in $HOME/.globals but this file is only loaded in case that it is owned by the uid of the shell and only writable by this uid, or if (needed for "su"-ing) the calling program uses the option "-o aliasowner=otherid".
(0002954)
joerg (reporter)
2015-11-16 11:34

Re Note 2945:

I am against mentioning history offset numbers in the standard as the current maintained Bourne Shell comes with a LRU history editor, I designed in 1982 and that moves an old command to the end of the command list if you re-execute it.

Having history offsets in the standard that are required to have a useful meaning would prevent to implement a history editor that some people feel to be superior to the csh model.

For the same reason, I recommened the deletion of the "fc" command already.
(0002955)
joerg (reporter)
2015-11-16 13:41

Re Note 2933:

I am not sure whether dash could ever get a POSIX certification as it
does not support multi byte characters. The missing multi-byte character
support even seems to be the cause for it's relatively good performance.

Dash always tries to implement the minimal set of features.
(0002956)
user229
2015-11-16 13:54

To my understanding, the real issue with Shellshock is that it applied to variables with arbitrary names and therefore variables which might appear on a whitelist (LANG) or have a well-defined meaning associated with remote input (HTTP_COOKIE, SSH_COMMAND). Saying that no program should allow even a narrow set of environment variables to control anything significant about its behavior is a massive overreaction. You might as well say that it's dangerous for HOME to be an environment variable.

As for your other note, I don't see why having the standard specify the conventional history model would prevent an implementation from providing it and some other model in parallel.
(0002957)
joerg (reporter)
2015-11-16 14:01

Of course, Shellshock was a more general attack on the shell, but it may be a wise idea to forbid to import shell variables from the calling environment if these variables are subject to command substitution.

In theory, I could implement access to previous history commands via offsets, but the command that is e.g. 3 items above the current may be different from what people expect when they assume the csh model. So the main problem is that this feature cannot result in portable behavior.
(0002958)
user229
2015-11-16 14:15

Or the calling environment can take some responsibility for what variables it passes through from a remote caller. Shellshock was special because it worked with *any* variable name.

Why not maintain two lists in parallel, one on the old model and one on the new one, and use the "old model" list for commands that use offsets?
(0002959)
joerg (reporter)
2015-11-16 14:20

The reason why the new LRU model was introduced is that it allows to hold more different entries from the history even when a single command is repeated many times. There is no option for the old model anymore.
(0002960)
user229
2015-11-16 14:28

If you're not interested in allowing users to carry assumptions about interactive behavior from one system to another, in favor of a bleeding-edge new feature that nobody else has, maybe the "User Portability" option is not for you.
(0002961)
joerg (reporter)
2015-11-16 14:45
edited on: 2015-11-16 14:46

If there are still implementations that support only a model from 1979 even though there is a more convenient (for typical users) method since more than 30 years, we have at least to ask why the older model could make it into the standard.

The POSIX standard is not about to standardize behavior where it is known that different existing implementations disagree since more than 30 years.

(0002962)
user229
2015-11-16 15:06

How is it more convenient? You just said it was about saving storage space. I have lots of storage space. If I want it to hold more entries I can increase the limit.
(0002963)
joerg (reporter)
2015-11-16 15:11
edited on: 2015-11-16 15:16

You may like to play with an implementation of the old history concept,
issue a single command several dozen times and then try to move through
the history using "cursor up" or trying to use a history offset from
memory without first listing the actual history.

BTW: when the new cursor history editor was implemented at H.Berthold AG
in August 1984, it took me some time to convince people to allow the
new model. But after it was implemented and tested in the field,
nobody did like to get the old model back.

(0002964)
user229
2015-11-16 15:18

If the history number is in PS1 and the command is still on the screen then it doesn't have to be from memory. And I don't see why you couldn't use the new model for cursor up while still using the history offset/number for commands that explicitly accept one.
(0002967)
shware_systems (reporter)
2015-11-17 21:07

Re: 2955
Afaik, without the multi-byte support dash could get a base POSIX certification, but not an XSI/UNIX one where localedef provided.
(0003023)
mirabilos (reporter)
2016-01-13 22:03

mksh expands ! first because it was (at pdksh times) believed to be what the standard dictates, however, it allows escaping them, so $!! expands to the value of $!.

@kre: $((# is special in mksh, it starts an unsigned integer arithmetics block. In general, only parameters that start with a letter (or underscore) can be used inside $((…)) blocks without preceding $ sign. So you need $(($#)) instead, and $(($*)) works:

tg@blau:~ $ x() { echo $(($*)); }; x 1 + 2
3
(0003031)
kre (reporter)
2016-01-22 16:42

Re note 3023:

Thanks for the explanation of $((# ...)) in mksh. That explains the
behaviour I saw. Also I don't think there's any doubt that $ expansions
happen inside $(( )) - all shells do that (all I am aware of).

To fix one error I introduced - in my note 2940, I said ...
     I believe that, as ! and ? are sh variables, and contain valid integer
     constants
which is, of course, incorrect - ! and ? are not sh variables, they are
special parameters, the syntax for their use is just similar to that for
variables. Only variables are expected to be expanded without a $ inside
a $(( )) expression, so the shells that do not expand special parameters are
doing the right thing (and the one that does is bizarre.)

I am kind of surprised that none of the language lawyers here caught that
error (of mine.)
(0003310)
geoffclare (manager)
2016-07-28 15:57
edited on: 2017-01-19 16:27

Interpretation response
------------------------

The standard does not speak to this issue, and as such no conformance
distinction can be made between alternative implementations based on this.
This is being referred to the sponsor.

Rationale:
-------------
The standard does not match existing practice for several current shell
implementations.

Notes to the Editor (not part of this interpretation):
-------------------------------------------------------

On page 2326 lines 73832-73841, replace the entire description of PS1 with:
[UP]PS1 The processing of the PS1 shell variable shall be supported if the system supports the User Portability Utilities option.[/UP]

Each time an interactive shell is ready to read a command, the value of this variable shall be subjected to parameter expansion (see [xref to Section 2.6.2]) and exclamation-mark expansion (see below). Whether the value is also subjected to command substitution (see [xref to Section 2.6.3]) or arithmetic expansion (see [xref to Section 2.6.4]) or both is unspecified. After expansion, the value shall be written to standard error.

The expansions shall be performed in two passes, where the result of the first pass is input to the second pass. One of the passes shall perform only the exclamation-mark expansion described below. The other pass shall perform the other expansion(s) according to the rules in [xref to Section 2.6]. Which of the two passes is performed first is unspecified.

The default value shall be <tt>"$ "</tt>. For users who have specific additional implementation-defined privileges, the default may be another, implementation-defined value.

Exclamation-mark expansion: The shell shall replace each instance of the <exclamation-mark> character (<tt>'!'</tt>) with the history file number (see [xref to sh extended description]) of the next command to be typed. An <exclamation-mark> character escaped by another <exclamation-mark> character (that is, <tt>"!!"</tt>) shall expand to a single <exclamation-mark> character.

On page 2326 lines 73842-73846, replace the entire description of PS2 with:
[UP]PS2 The processing of the PS2 shell variable shall be supported if the system supports the User Portability Utilities option.[/UP]

Each time the user enters a <newline> prior to completing a command line in an interactive shell, the value of this variable shall be subjected to parameter expansion (see [xref to Section 2.6.2]). Whether the value is also subjected to command substitution (see [xref to Section 2.6.3]) or arithmetic expansion (see [xref to Section 2.6.4]) or both is unspecified. After expansion, the value shall be written to standard error. The default value shall be <tt>"> "</tt>.

On page 2326 lines 73847-73851, replace the entire description of PS4 with:
[UP]PS4 The processing of the PS4 shell variable shall be supported if the system supports the User Portability Utilities option.[/UP]

When an execution trace (set −x) is being performed in an interactive shell, before each line in the execution trace, the value of this variable shall be subjected to parameter expansion (see [xref to Section 2.6.2]). Whether the value is also subject to command substitution (see [xref to Section 2.6.3]) or arithmetic expansion (see [xref to Section 2.6.4]) or both is unspecified. After expansion, the value shall be written to standard error. The default value shall be <tt>"+ "</tt>.

On page 3191 line 106776 append to the description of ENV:
The file specified by ENV need not be processed if the file can be written by any user other than the user identified by the real (and effective) user ID of the shell process.


(0003328)
kre (reporter)
2016-08-06 12:41

I object to this interpretation being applied until after issue 1038 is
resolved.

Accidentally specifying undefined behaviour is an unfortunate error that
should be corrected (which is what 1038 is requesting). Deliberately
specifying undefined behaviour after the problem is known is unconscionable.

Note: the relevant wording here is not incorrect, nor is it undesireable,
the fix for the problem should be applied elsewhere - but before this text
is incorporated.

While I am here though, and separate from the above, the lack of specification
or the order for the interpretation of the ! in PS1 wrt other expansions is
truly unfortunate, and makes all kinds of things impossible to achieve
portably. It would be really nice to gradually move to a state where all
this becomes rational - accessing the history event number should be achieved
via a shell variable (or even a new special parameter if desired) rather than
via this abominable ! hack. Shell implementors who currently have this !
expansion in their shell, please consider providing an alternative, and
then deprecate use of ! in later versions.
(0003336)
rhansen (manager)
2016-08-11 16:10
edited on: 2016-08-11 16:11

> I object to this interpretation being applied until after issue 1038 is resolved.

This bug report and 0001038 touch the same text, but they are logically independent and thus the order we resolve them doesn't really matter. We will get to 0001038 eventually; when we do, the proposed text will say something like "After applying the changes from 0001006, change ... to ...".

(0003337)
geoffclare (manager)
2016-08-11 16:13

Note: 0003310 was edited during the 2016-08-11 teleconference to add the PS2 and PS4 changes.
(0003338)
rhansen (manager)
2016-08-11 16:38

> [...] the lack of specification or the order for the interpretation
> of the ! in PS1 wrt other expansions is truly unfortunate, and makes
> all kinds of things impossible to achieve portably. It would be
> really nice to gradually move to a state where all this becomes
> rational [...]

I completely agree. Unfortunately, we can't fix it here in this bug report because this bug is targeting an Issue 7 TC. What we can and should do is file a new bug report targeting Issue 8.
(0003339)
rhansen (manager)
2016-08-11 16:45
edited on: 2016-08-11 23:59

There's another change to the proposed wording that we may want to make:
Escaping the <exclamation-mark> character with another <exclamation-mark> character (that is, <tt>"!!"</tt>) shall place the literal <exclamation-mark> character in the prompt.
should be something like:
An <exclamation-mark> character escaped by another <exclamation-mark> character (that is, <tt>"!!"</tt>) shall expand to a single <exclamation-mark> character.
because the exclamation point expansion might happen first and thus the ! might not end up in the prompt.

(0003340)
kre (reporter)
2016-08-11 16:48

wrt note 3336:

    This bug report and 0001038 touch the same text,

Actually, no, they don't. The text here (and/or that which it is replacing)
provides the motivation that requires 1038 to be fixed, but the fix for 1038
will go elsewhere, I hope.

    but they are logically independent

that was my point. They are not. In the text in note 3310 (the resolution)
the default value of PS1 is stated to be "$ " (which is obviously what it
should be) and that it is subject to parameter expansion.

As soon as you specify that, you're specifying undefined behaviour, as a
space after a '$' is explicitly unspecified (see section 2.6) That is,
given the required default value of PS1 and its required processing, the
shell is permitted to do anything at all. You cannot seriously mean that?

    We will get to 0001038 eventually;

I'm sure you will, and that's fine - but until that is resolved, you cannot
in good conscience decide to incorporate this text. After all, when 1038
is discussed. the resolution may be that a space after a '$' should remain
unspecified, and that it should not be used in any conforming application.
That would mean that PS1="$ " would be both non-conforming, and required.

All that is required is for this issue be suspended until after 1038 is
dealt with, after which what should be in this text can be determined
(and most likely, I hope, what is there now will be fine ... aside from that
! nonsense, which is a different issue entirely.)
(0003341)
shware_systems (reporter)
2016-08-11 17:09
edited on: 2016-08-11 17:37

To me the intent is default value applies when expansion fails or succeeds but the expansion is of zero length, so is a result value, not an input to a retry of expansion. As a result value no processing of the "$" is implied. An implementation may specify as a privileges appropriate alternative a value as an input, with the current wording.

How about:
Instead of: The default value shall be <tt>"$ "</tt>.

Use:
If PS1 is unset or a zero length value after expansion, the shell shall use the string <tt>"$ "</tt> as the sequence to output, without evaluation for expansion, as the default value.

(0003342)
kre (reporter)
2016-08-11 17:41

Re note 3341

First, don't even consider treating a zero length PS1 (or PS2 or PS4)
as the same as unset - setting the prompt to null is very useful in
some circumstances (like when you're scripting and ddon't want
junk corrupting the script).

Apart from that, that could be a solution, but why? Do you really want
to make it unspecified if someone explicitly sets PS1 to its defined
default value? To me that makes no sense. Nor would it be anything
like traditional for a shell to leave PS1 undefined and then just output '$ '
in that case - while not exactly safe, I have seen scripts that examine
the value of PS1 to decide if they're running as root or not - and other
scripts that save and restore it.

Seems simpler to me to simply make space after '$' stop being unspecified,
and have it mean a literal $ followed by space (as it always has.) That is
just fix the wording in 2.6 (and I think somewhere else) and be done with
it - it doesn't have to be solved in this issue, just because this one was
raised first.
(0003343)
rhansen (manager)
2016-08-11 19:40

>> but they are logically independent
>
> that was my point. They are not. In the text in Note: 0003310 (the resolution)
> the default value of PS1 is stated to be "$ " (which is obviously what it
> should be) and that it is subject to parameter expansion.
>
> As soon as you specify that, you're specifying undefined behaviour

Yes, but the standard already said that PS1 was subject to parameter expansion and that the default value is "$ ". The changes in Note: 0003310 do not change the nature of the flaw described in 0001038. That's what I mean by logically independent.
(0003345)
shware_systems (reporter)
2016-08-11 22:07

The idea is, for interactive use, something standard gets visibly displayed. Both unset and zero length don't have any chars that can be construed as their value to attempt expansion with, or will be visible. It is little different from IFS using it's default value when zero length, so there is precedent. From the user's perspective I feel that is simplest. Otherwise there's no visible means to tell a previous application may have finished processing if it does not output a "Yes, I'm done" message of its own. When processing a script PS1 isn't evaluated anyways. PS4 might be, but this refinement is for PS1 because the default uses '$' and the others don't. Also, expansion is limited to the value PS1 is set to. The text does not say PS1 shall be set to the default value, which would imply "$ " is expected to be evaluated, just that this is the value that will be used (for output). Someone can use a zero display width string as prompt, such as <DEL> or <ACK> chars, if they don't want anything visible. Saying "no prompt is visible if the expansion result is zero length" can be rolled into privileges appropriate alternative too.

As it is, tokenization of "$ " is specified in XCU 2.3 by items 5 and 7, 8, and 10 for a conforming shell. Item 9 covers the "$#" sequence. Item 5 makes '$' a candidate for expansion, yes, but the immediately following ' ' makes it a non-candidate trailer to the current WORD token, by Item 7, as the specified behavior. Similarly for other following chars, if not a legal expansion continuation that changes the state from expansion candidate to pending expansion, the '$' and following char are simply appended to the current token value as text by Item 8 or start a new token by Item 10. Those are the first rules that apply since the conditions for Item 5 to apply aren't met. Where XCU 2.6 is in error is the list of continuing chars just before XCU 2.6.1 is correct, but to me it should say "the current token is appended to as if the $ was escaped, plus any non-delimiter char, per XCU 2.3" not "the result is unspecified". Unspecified leaves open the possibility for extensions when not in conforming mode, granted, or saying it is a syntax error, but I see the behavior as defined for strictly conforming.
(0003520)
ajosey (manager)
2016-12-15 18:10

Interpretation proposed: 15 Dec 2016
(0003526)
geoffclare (manager)
2016-12-19 09:37

Note: 0003339 was added before the interpretation review started. I believe it is a valid point and should be addressed as part of the review.
(0003556)
geoffclare (manager)
2017-01-19 16:28

Note: 0003310 has been edited to implement the change suggested in Note: 0003339
(0003557)
ajosey (manager)
2017-01-20 10:57

Revised Interpretation proposed: 20 January 2017
(0003567)
ajosey (manager)
2017-02-22 08:31

Interpretation approved: 22 Feb 2017
(0003614)
eblake (manager)
2017-03-09 16:41

We may need to reopen this, to allow shells to perform implementation-defined treatment of any PS1 value that contains literal \ (bash) or leading ~ (ksh).

- Issue History
Date Modified Username Field Change
2015-11-13 08:23 rhansen New Issue
2015-11-13 08:23 rhansen Name => Richard Hansen
2015-11-13 08:23 rhansen Organization => BBN
2015-11-13 08:23 rhansen Section => 2.5.3 PS1
2015-11-13 08:23 rhansen Page Number => 2326
2015-11-13 08:23 rhansen Line Number => 73833
2015-11-13 08:23 rhansen Interp Status => ---
2015-11-13 10:34 joerg Note Added: 0002906
2015-11-13 13:51 kre Note Added: 0002907
2015-11-13 15:44 user229 Note Added: 0002909
2015-11-13 16:04 user229 Note Edited: 0002909
2015-11-13 16:11 geoffclare Note Added: 0002911
2015-11-13 16:33 shware_systems Note Added: 0002913
2015-11-13 16:36 user229 Note Added: 0002914
2015-11-13 16:40 user229 Note Edited: 0002914
2015-11-13 16:47 user229 Note Edited: 0002914
2015-11-13 16:49 user229 Note Edited: 0002914
2015-11-13 16:49 user229 Note Edited: 0002914
2015-11-13 18:21 kre Note Added: 0002918
2015-11-13 18:29 kre Note Added: 0002919
2015-11-13 18:31 user229 Note Added: 0002920
2015-11-14 01:39 rhansen Note Added: 0002932
2015-11-14 01:40 rhansen Note Added: 0002933
2015-11-14 01:41 rhansen Note Edited: 0002933
2015-11-14 01:44 rhansen Note Edited: 0002933
2015-11-14 01:50 rhansen Note Added: 0002934
2015-11-14 10:06 geoffclare Note Added: 0002939
2015-11-14 13:43 kre Note Added: 0002940
2015-11-14 13:49 kre Note Edited: 0002940
2015-11-14 14:01 user229 Note Added: 0002941
2015-11-14 17:26 shware_systems Note Added: 0002944
2015-11-14 17:27 shware_systems Note Edited: 0002944
2015-11-14 21:03 user229 Note Added: 0002945
2015-11-14 21:04 user229 Note Edited: 0002945
2015-11-15 03:26 rhansen Note Added: 0002946
2015-11-15 03:43 rhansen Note Edited: 0002946
2015-11-15 04:22 rhansen Note Added: 0002947
2015-11-15 07:28 kre Note Added: 0002948
2015-11-15 07:55 kre Note Edited: 0002940
2015-11-15 09:14 kre Note Added: 0002949
2015-11-15 09:25 kre Note Edited: 0002949
2015-11-15 09:27 kre Note Edited: 0002949
2015-11-15 09:31 kre Note Edited: 0002949
2015-11-15 10:44 user229 Note Added: 0002950
2015-11-16 09:38 geoffclare Note Added: 0002952
2015-11-16 11:22 joerg Note Added: 0002953
2015-11-16 11:34 joerg Note Added: 0002954
2015-11-16 13:41 joerg Note Added: 0002955
2015-11-16 13:54 user229 Note Added: 0002956
2015-11-16 14:01 joerg Note Added: 0002957
2015-11-16 14:15 user229 Note Added: 0002958
2015-11-16 14:20 joerg Note Added: 0002959
2015-11-16 14:28 user229 Note Added: 0002960
2015-11-16 14:45 joerg Note Added: 0002961
2015-11-16 14:46 joerg Note Edited: 0002961
2015-11-16 15:06 user229 Note Added: 0002962
2015-11-16 15:11 joerg Note Added: 0002963
2015-11-16 15:14 joerg Note Edited: 0002963
2015-11-16 15:15 joerg Note Edited: 0002963
2015-11-16 15:16 joerg Note Edited: 0002963
2015-11-16 15:18 user229 Note Added: 0002964
2015-11-17 21:07 shware_systems Note Added: 0002967
2016-01-13 22:03 mirabilos Note Added: 0003023
2016-01-22 16:42 kre Note Added: 0003031
2016-07-28 15:57 geoffclare Note Added: 0003310
2016-07-28 15:58 geoffclare Interp Status --- => Pending
2016-07-28 15:58 geoffclare Final Accepted Text => Note: 0003310
2016-07-28 15:58 geoffclare Status New => Interpretation Required
2016-07-28 15:58 geoffclare Resolution Open => Accepted As Marked
2016-07-28 15:58 geoffclare Tag Attached: tc3-2008
2016-07-28 15:59 geoffclare Note Edited: 0003310
2016-08-06 12:41 kre Note Added: 0003328
2016-08-11 15:29 Don Cragun Relationship added related to 0001038
2016-08-11 16:08 geoffclare Note Edited: 0003310
2016-08-11 16:10 rhansen Note Added: 0003336
2016-08-11 16:11 rhansen Note Edited: 0003336
2016-08-11 16:13 geoffclare Note Added: 0003337
2016-08-11 16:38 rhansen Note Added: 0003338
2016-08-11 16:42 rhansen Note Edited: 0003310
2016-08-11 16:45 rhansen Note Added: 0003339
2016-08-11 16:48 kre Note Added: 0003340
2016-08-11 17:09 shware_systems Note Added: 0003341
2016-08-11 17:17 shware_systems Note Edited: 0003341
2016-08-11 17:26 shware_systems Note Edited: 0003341
2016-08-11 17:36 shware_systems Note Edited: 0003341
2016-08-11 17:37 shware_systems Note Edited: 0003341
2016-08-11 17:41 kre Note Added: 0003342
2016-08-11 19:40 rhansen Note Added: 0003343
2016-08-11 22:07 shware_systems Note Added: 0003345
2016-08-11 23:59 rhansen Note Edited: 0003339
2016-12-15 18:10 ajosey Interp Status Pending => Proposed
2016-12-15 18:10 ajosey Note Added: 0003520
2016-12-19 09:37 geoffclare Note Added: 0003526
2017-01-19 16:27 geoffclare Note Edited: 0003310
2017-01-19 16:28 geoffclare Note Added: 0003556
2017-01-20 10:57 ajosey Note Added: 0003557
2017-02-22 08:31 ajosey Interp Status Proposed => Approved
2017-02-22 08:31 ajosey Note Added: 0003567
2017-03-09 16:41 eblake Note Added: 0003614
2019-10-21 09:43 geoffclare Status Interpretation Required => Applied
2021-08-12 09:32 geoffclare Relationship added related to 0001511


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker