Austin Group Defect Tracker

Aardvark Mark IV

Main | My View | View Issues | Change Log | Docs
  

Viewing Issue Simple Details [ Jump to Notes ] [ Issue History ] [ Print ]
ID Category Severity Type Date Submitted Last Update
0001342 [1003.1(2016/18)/Issue7+TC2] Shell and Utilities Objection Error 2020-05-05 08:50 2024-06-11 09:08
Reporter kre View Status public  
Assigned To
Priority normal Resolution Accepted As Marked  
Status Closed  
Name Robert Elz
Organization
User Reference
Section XCU 2.2.3
Page Number 2346
Line Number 74721
Interp Status Approved
Final Accepted Text Note: 0005123
Summary 0001342: Aliases in command substitutions are handled differently when the command subst is quoted vs not quoted
Description A 1997 interpretation resulted in the addition of the words:

   not including the alias substitutions in Section 2.3.1,

to the rules for processing a command substitution embedded within double
quotes (XCU 2.2.3).

No similar change was made to section 2.3 (bullet point 5) which handles
the general rules for token recognition when a command substitution is
detected, unquoted.

This means that different rules apply to recognising a command substitution
that happens in a double-quoted string than one that appears unquoted.

This is neither rational, nor what shells implement.

Apart from the unfortunate difference made to the two cases, the 1997
interpretation was almost certainly correct for its time - the relevant shells
of the mid 1990's did not expand aliases while searching for the terminating ')'
of a $() command substitution (backquoted command substitutions are much
simpler and have none of these issues). Shells of the time didn't parse
the command substitution at all while searching for the ')' - they relied
upon '(' and ')' counting.

In the intervening 20-something years that technique has been shown to be
fatally flawed, not only do (some - unlikely but possible) aliases potentially
break this, but case statements (not using the optional '(' preceding the
pattern) and here documents (which can contain almost anything - but can
only be recognised as being a here document by parsing the accompanying
command) break this simple scheme - shells must fully parse a $() command
substitution to have any hope of correctly finding the terminating ')' in
all cases.

Note that whether the shell retains the results of this parse, or simply
copies the command substitution string literally, saving it to be parsed
again later, is immaterial to this issue - the correct ')' must be located
to properly terminate the command substitution, whatever is done with the
text of the command substitution, or the results of parsing it, during that
process.
Desired Action In line 74271, on page 2346 of XCU 2.2.3 (2016 edition) delete the words:

    not including the alias substitutions in Section 2.3.1,

This is needed regardless of any other changes that might be made, as even
if it is (was) the correct behaviour, it was applied in the wrong place.

That is all that is needed to handle this issue.

However, as an option, in XCU 2.3, bullet point 5, on page 2348, after
the sentence (lines 74774-6):

    While processing the characters, if instances of expansions or quoting are
    found nested within the substitution, the shell shall recursively process
    them in the manner specified for the construct that is found.

add a new sentence:

    It is unspecified whether aliases [xref XCU 2.3.1] shall be expanded
    while performing this process.

It needs to be unspecified, rather than "shall not process" as most shells
do process aliases at this point, as they are when the command substitution
is eventually parsed and executed, if different rules apply strange hard
to fathom errors can occur.

However, not all shells do, so making it unspecified might be the wise choice
for now - even if such shells tend to be broken in this regard. Since alias
processing when the command substitution was in a double quoted string was
(or is) prohibited by the standard, suddenly making it mandated when there are,
or might be, some shells which actually implemented this prohibition would
also seem harsh.

If this option is adopted, two further changes should be made.

In the application notes (wherever they are for this) there should be an
admonition

     Applications shall not include aliases which contain unbalanced
     syntax components in any $() command substitution.

What that means can be expanded if deemed appropriate. The idea is that
trivial aliases "alias l=ls" (etc) are harmless, and not a problem, it is
only ones like "alias switch=case", "alias subshell='('", or
"alias forever='while true do;'" that cause problems. The "unbalanced"
is because an alias like "alias set13='(exit 13)'" is not a problem,
despite having parentheses in its value.

And second, in the Future Directions (wherever that is for sh) it should
say

     A future version of this standard may require alias expansion
     while scanning for the terminating ')' in a $() command substitution.

Last, some kind of explanation of all that happened here should be added
in the rationale (somewhere in XRAT I assume).


An alternative option open to the group who decides these things would be
to simply delete aliases from the standard completely. Since I doubt this
one will happen, I won't bother providing the changes needed to implement it.
Tags tc3-2008
Attached Files

- Relationships
related to 0000953Closedajosey 1003.1(2013)/Issue7+TC1 Alias expansion is under-specified 

-  Notes
(0004856)
geoffclare (manager)
2020-05-05 10:57

 We also need to do something about the statement for $() command substitution that "Any valid shell script can be used for command". This is not true because a shell script which defines an alias which changes the syntax, and then uses it, might not be parsed correctly without the alias command being executed.
For example a script containing:
alias subshell='('
subshell echo foo )
subshell echo bar )
works fine (note you need POSIXLY_CORRECT set for bash), but:
echo $(
alias subshell='('
subshell echo foo )
subshell echo bar )
)
does not.
(0004859)
kre (reporter)
2020-05-05 15:36

 Re Note: 0004856

I suspect that's already handled by the (new, I forget which bug it is)
wording that specifies when alias commands take effect.

This is just the same issue as prevents

if [ $SHELL = myshell ]; then
    alias xx=mycommand
    xx foo bar
fi

from working. Or from defining, and then using, an alias in a function.

Aliases are evil - let's delete them!!!
(0004860)
geoffclare (manager)
2020-05-05 15:52

 Re Note: 0004859

Bug 0000953 does indeed change that text, but the new text still has the same problem. It now says:
It is unspecified whether command is parsed and executed as a program (as for a shell script) or is parsed as a single compound_list that is executed after the entire command has been parsed. With the $(command) form any valid program can be used for command, except a program consisting solely of redirections which produces unspecified results.
(0004863)
kre (reporter)
2020-05-05 17:43

 OK, we can add text that makes it clear (just in case it wasn't
already obvious) that the entire command in a command substitution is
parsed before any of it is executed. And given that, alias commands
in the command substitution cannot work - the command substitution is
a subshell environment, so the parent shell cannot be affected, and
any aliases defined there get defined too late to be used within the
command substitution.

Alias in a sub-shell block '(' code ')' have the same uselessness.

Aliases in general are largely useless, let's delete them!!!
(0005123)
geoffclare (manager)
2020-11-30 17:04

 Interpretation response
------------------------
The standard states that alias substitutions are applied when finding the terminating ')' of a command substitution when the command substitution is not within double quotes, but they are not applied when the command substitution is within double quotes, and conforming implementations must conform to this. However, concerns have been raised about this which are being referred to the sponsor.

Rationale:
-------------
Historically some shells used simple parenthesis counting to find the terminating ')' and therefore did not account for aliases. However, such shells never conformed to POSIX, which has always required recursive parsing (see XCU 2.3 item 5). It is likely that observing the behavior of these non-conforming shells was what led to the erroneous addition of "not including the alias substitutions in Section 2.3.1" to XCU 2.2.3 after PASC interpretation 1003.2 #157.

Notes to the Editor (not part of this interpretation):
-------------------------------------------------------

On page 2346 line 74721 section 2.2.3, delete:
not including the alias substitutions in Section 2.3.1,

On page 2357 line 75182 section 2.6.3, change:
Command substitution allows the output of a command to be substituted in place of the command name itself. Command substitution shall occur when the command is enclosed as follows
to:
Command substitution allows the output of one or more commands to be substituted in place of the commands themselves. Command substitution shall occur when command(s) are enclosed as follows

On page 2357 line 75190 section 2.6.3, change:
with the standard output of the command
to:
with the standard output of the command(s)

On page 2358 lines 75202-75204 section 2.6.3, after applying bug 953, change:
It is unspecified whether command is parsed and executed as a program (as for a shell script) or is parsed as a single compound_list that is executed after the entire command has been parsed. With the $(command) form any valid program can be used for command, except a program consisting solely of redirections which produces unspecified results.
to:
It is unspecified whether the commands string is parsed and executed incrementally as a program (as for a shell script), or is parsed as a single compound_list that is executed after the string has been completely parsed. In addition, it is unspecified whether the terminating ')' of the $(commands) form can result from alias substitution. With the $(commands) form any syntactically correct program can be used for commands, except that:

  • If the commands string consists solely of redirections, the results are unspecified.

  • If the commands string is parsed as a single compound_list, before any commands are executed, alias and unalias commands in commands have no effect during parsing (see [xref to 2.3.1]). Strictly conforming applications shall ensure that the commands string does not depend on alias changes taking effect incrementally as would be the case if parsed and executed as a program.

  • The behavior is unspecified if the terminating ')' is not present in the token containing the command substitution; that is, if the ')' is expected to result from alias substitution.

Throughout section 2.6.3 change any remaining occurrences of "command" to "commands" or "the commands string" as appropriate.

On page 3730 line 127967 section C.2.6.3, change:
   
While the newer "$()" form can process any kind of valid embedded script, the backquoted form ...
to:
While the newer "$()" form can process any kind of valid embedded script (with a few caveats; see below), the backquoted form ...

On page 3730 line 127985 section C.2.6.3, change:
If command is of the form ...
to:
If the commands string is of the form ...

On page 3730 line 127990 section C.2.6.3, after:
... it is unspecified to have a script within "$()" that has only redirections.
add the following new paragraph:
In POSIX.2-1992 the $(commands) form of command substitution only had unspecified behavior for a commands string consisting solely of redirections. However, two additional unspecified cases have since been added with relation to aliases:

  1. Implementations are permitted to parse the entire commands string before executing any of it, and in this case alias and unalias commands in commands have no effect during parsing. For example, the following commands:
    alias foo='echo "hello globe"'
echo $(alias foo='echo "Hello World"';foo)

    produce the output "hello globe" if the entire commands string is executed as an entire command and produce the output "Hello World" if the commands string is executed incrementally.

  2. Although existing aliases are required to be expanded when the shell parses the input that follows the "$(" in order to find the terminating ')' (see [xref to 2.3 Token Recognition]), it is unspecified whether the terminating ')' can result from alias substitution. For example, with this script:
    alias foo="echo foo )"
echo $(foo ; echo bar

    some shells output lines containing <tt>foo</tt> and <tt>bar</tt> whereas other shells report a syntax error because they do not find a terminating ')' for the command substitution.
(0005137)
agadmin (administrator)
2020-12-02 17:04

 Interpretation proposed: 2 December 2020
(0005190)
agadmin (administrator)
2021-01-04 12:14

 Interpretation approved: 4 Jan 2021

- Issue History
Date Modified Username Field Change
2020-05-05 08:50 kre New Issue
2020-05-05 08:50 kre Name => Robert Elz
2020-05-05 08:50 kre Section => XCU 2.2.3
2020-05-05 08:50 kre Page Number => 2346
2020-05-05 08:50 kre Line Number => 74721
2020-05-05 10:57 geoffclare Note Added: 0004856
2020-05-05 15:36 kre Note Added: 0004859
2020-05-05 15:52 geoffclare Note Added: 0004860
2020-05-05 15:53 geoffclare Relationship added related to 0000953
2020-05-05 17:43 kre Note Added: 0004863
2020-11-30 17:04 geoffclare Note Added: 0005123
2020-11-30 17:07 geoffclare Interp Status => Pending
2020-11-30 17:07 geoffclare Final Accepted Text => Note: 0005123
2020-11-30 17:07 geoffclare Status New => Interpretation Required
2020-11-30 17:07 geoffclare Resolution Open => Accepted As Marked
2020-11-30 17:07 geoffclare Tag Attached: tc3-2008
2020-12-02 17:04 agadmin Interp Status Pending => Proposed
2020-12-02 17:04 agadmin Note Added: 0005137
2021-01-04 12:14 agadmin Interp Status Proposed => Approved
2021-01-04 12:14 agadmin Note Added: 0005190
2021-01-29 11:21 geoffclare Status Interpretation Required => Applied
2024-06-11 09:08 agadmin Status Applied => Closed

Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker