Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0000351 [1003.1(2008)/Issue 7] Shell and Utilities Objection Error 2010-11-09 18:42 2023-10-28 05:20
Reporter eblake View Status public  
Assigned To ajosey
Priority normal Resolution Accepted As Marked  
Status Applied  
Name Eric Blake
Organization Red Hat
User Reference ebb.export
Section 2.9.1 Simple Commands
Page Number 2316
Line Number 73089
Interp Status Approved
Final Accepted Text See Note: 0000865
Summary 0000351: certain shell special built-ins should expand arguments in assignment context
Description Historical behavior of ksh88 was that 'export' and 'readonly' can treat
arguments in an assignment context. From the ksh88 man page:

| Words that are in the format of a parameter assignment,
| are expanded with the same rules as a parameter assignment.
| This means that
| tilde substitution is performed after the
| .B =
| sign and word splitting and file name generation are not
| performed.

The standard even assumes this is the case, in the EXAMPLES section of
export at line 74320:

| Set and export the PATH variable:
| export PATH=/local/bin:$PATH

However, the standard fails to document this, so a current strict reading
would require that if $PATH contains any whitespace or globbing
characters, then the result might be the exporting of more than just PATH,
or a syntax error if the expansion results in a non-name argument, and
that the above example should have been written

export PATH=/local/bin:"$PATH"

to avoid word splitting and file name expansion.

Additionally, the ksh behavior is documented as treating arguments as
assignment context only if they could be recognized as assignment context
in isolation. In isolation, the use of quoting rejects a word from being
recognized as an assignment, so for words after export, the same logic
applies where the use of quoting can make a word subject to traditional
field splitting and no special tilde expansion.

Current dash behavior is strictly compliant, but it is inconsistent with
historical behavior; for example:

$ ksh -c 'set "1 b=2" "3 d=4"; export a=$1 \c=$2; echo "$a.$b.$c.$d"'
1 b=2..3.4
$ dash -c 'set "1 b=2" "3 d=4"; export a=$1 \c=$2; echo "$a.$b.$c.$d"'
1.2.3.4

This proposal mandates the historical ksh behavior, even though it renders
some existing implementations non-compliant; so it must be sent down the
interpretations track. However, it is worth inclusion in the TC1
revision, rather than deferring to a future revision, even though it will
require some implementations to make fixes. For example:

$ ksh -c 'HOME=/tmp; export \a=~; echo "$a"'
~
$ mksh -c 'HOME=/tmp; export \a=~; echo "$a"'
/tmp
Here, ksh correctly recognizes that the quoting of the name a means that
the argument to export is not in the form of an assignment; therefore, no
tilde expansion takes place. mksh mistakenly performs the tilde
expansion.

$ bash --posix -c 'HOME=/tmp; command export a=~; echo "$a"'
~
$ ksh -c 'HOME=/tmp; command export a=~; echo "$a"'
/tmp
Since the 'command' utility is not a special built-in, and this proposal
only assignment context if the command word is recognized as a special
built-in, the bash behavior is correct and ksh is in error for treating
a=~ in an assignment context.

There is one thing that this proposal intentionally leaves unspecified,
in order to allow two styles of existing behavior:

$ mksh -c 'HOME=/tmp b=export; $b a=~; echo "$a"'
/tmp
$ ksh -c 'HOME=/tmp b=export; $b a=~; echo "$a"'
~

Here, mksh recognizes the first command word as 'export', and performs
assignment-style tilde expansion; but ksh performs textual recognition
(see also 0000053 for an example of ksh recognizing trap via textual
recognition), so none of the subsequent arguments are recognized as an
assignment context, and no tilde expansion is performed.

Finally, the proposal intentionally allows shells to implement other
special built-ins as extensions, such as 'typeset' or 'local', that can
also treat arguments as being in an assignment context.
Desired Action At line 73093 (XCU 2.9.1 step 2), add:

If the command name is recognized as a special built-in utility that can
affect variable assignments, then any remaining words that would be
recognized as a variable assignment in isolation shall be expanded as
a variable assignment (tilde expansion after the first <equals-sign> and
after any unquoted <colon>, parameter expansion, command substitution,
arithmetic expansion, and quote removal, but no field splitting or
pathname expansion); while words that would not be a variable assignment
in isolation are subject to regular expansion. For all other command
names, all subsequent words are subject to regular expansion (tilde
expansion for only a leading <tilde>, parameter expansion, command
substitution, arithmetic expansion, field splitting, pathname expansion,
and quote removal).

After line 73100, add a paragraph:

When determining whether a command name is present and represents a
special built-in that would require subsequent words to be considered for
assignment expansion, an implementation may use only lexical analysis.
For example, it is unspecified whether 'var=export; $var a=~' sets the
variable a to a literal <tilde> or to the value of $HOME, since lexical
analysis sees '$var' rather than 'export' as the command name.

At line 73820 (XCU 2.14), change "two" to "three".

At line 73827, add a paragraph:

3. When a special built-in that modifies variables is recognized as the
command word, subsequent words shall be considered for expansion in an
assignment context (allowing multiple tilde expansion, and inhibiting
field splitting and pathname expansion).

At line 74276 (XCU export DESCRIPTION), add a sentence:

During parsing of the simple command, if export was recognized as the
command word, then subsequent words of the form name=word shall be expanded
in an assignment context.

At line 74360 (XCU readonly DESCRIPTION), add a sentence:

During parsing of the simple command, if readonly was recognized as the
command word, then subsequent words of the form name=word shall be expanded
in an assignment context.

At line 81561 (XCU command APPLICATION USAGE), add:

Similarly:

command export a=~

suppresses the parsing of a=~ in an assignment context, such that the
variable a will contain a literal <tilde> rather than the contents of
$HOME.

At line 124658 (XRAT C.2.9.1), add:

Expansion of words in an assignment context following the command word can
only occur for special built-ins, and only when the word can be used as a
variable assignment in isolation.

For example, this code sequence exports the single variable a with the
value '1 b=2', but invokes make with the macro a set to '1' and b set
to '2', since make is not a special built-in:

$ set '1 b=2'
$ export a=$1
$ make a=$1

Conversely, this code sequence exports two variables, a set to '1' and
b set to '2', because the use of quoting means that the word could not
be recognized as a variable assignment, and regular expansion rules
require that field splitting occurs on the unquoted expansion of $1:

$ set '1 b=2'
$ export \a=$1

Implementations are permitted to provide extensions of other special
built-ins that can cause assignment context expansion, such as
'typeset' or 'local'.
Tags issue8
Attached Files

- Relationships
related to 0001393Applied Issue 8 drafts 'command' should not be treated as a declaration utility 
related to 0000352Closedajosey 1003.1(2008)/Issue 7 behavior when assignments precede export 
related to 0001172Applied 1003.1(2016/18)/Issue7+TC2 shell tilde expansion. Clarify ~"user" ~user@domain ~$user ~$(logname) ~-1... 
related to 0001535Applied Issue 8 drafts Poor description of declaration (all really) utility argument processing 
related to 0001785Resolved Issue 8 drafts Conflict in specification of processing of declaration utilities 

-  Notes
(0000613)
eblake (manager)
2010-11-09 18:46

Given the behavior of ksh 'command export a=~', and the on-list comments from
David Korn:

ksh93 has the concept of declaration commands and these are recognized
while the script is tokenized. Changing this would complicate extensions
in ksh93 such as user defined types which are declaration commands that
contain declaration commands.

It may be worth an alternative proposal that defines the notion of a
declaration command, states that parsing arguments in an assignment context
shall happen for all simple commands that are recognized as a declaration
command whether or not they are special built-ins, then mandates that
'export', 'readonly' are declaration commands, that it is
implementation-defined whether 'command' is a declaration command, and that
implementations may define extensions that allow the user to introduce
other declaration commands.
(0000618)
eblake (manager)
2010-11-18 16:43

Mail sequence 14988 proposed this in response to note 613:

New definition in XBD 3:
Declaration Command
A utility which can take arguments that cause variable assignments (of
the form varname=value) which will persist in the current shell
environment. During parsing, when the shell recognizes a declaration
command as the command name, then subsequent arguments that would be a
valid variable assignment in isolation are subject to different
expansion rules (field splitting and pathname expansion are suppressed,
and tilde expansion occurs after the <equals-sign> and any unquoted
<colon>). Arguments which are not a valid variable assignment in
isolation are processed according to normal argument expansion rules.

The following standard utilities are declaration commands: export,
readonly, and certain instances of command. An implementation may
provide other declaration commands.

At line 3276, XBD 4.22 Variable Assignment, add:
Assignment context occurs in the cmd_prefix portion of a shell simple
command, as well as in arguments of a recognized declaration command.

At line 71964, XCU 1.4 Utility Description Defaults DESCRIPTION, add:
A standard utility shall not be treated as a declaration command unless
explicitly stated in this section.

At line 73093 (XCU 2.9.1 step 2), add:
If the command name is recognized as a declaration command, then any
remaining words that would be recognized as a variable assignment in
isolation shall be expanded as a variable assignment (tilde expansion
after the first <equals-sign> and after any unquoted <colon>, parameter
expansion, command substitution, arithmetic expansion, and quote
removal, but no field splitting or pathname expansion); while words that
would not be a variable assignment in isolation are subject to regular
expansion. For all other command names, all subsequent words are subject
to regular expansion (tilde expansion for only a leading <tilde>,
parameter expansion, command substitution, arithmetic expansion, field
splitting, pathname expansion, and quote removal).

After line 73100, add a paragraph:
When determining whether a command name is declaration command, an
implementation may use only lexical analysis. It is unspecified whether
assignment context will be used if the command name is not known prior
to the expansion of the first field which is not a variable assignment
or redirection.

At line 74276 (XCU export DESCRIPTION), add:
The export special built-in shall be a declaration command. Therefore,
during parsing of a simple command, if export is recognized as the
command word, then subsequent words of the form name=word shall be
expanded in an assignment context.

At line 74360 (XCU readonly DESCRIPTION), add:
The readonly special built-in shall be a declaration command.
Therefore, during parsing of a simple command, if readonly is recognized
as the command word, then subsequent words of the form name=word shall
be expanded in an assignment context.

At line 81462 (XCU command DESCRIPTION), add:
The command utility shall be a declaration command if the command_name
argument is recognized as a declaration command. Therefore, during
parsing of a simple command, if command is recognized as the command
word and a declaration command is recognized as the next argument, then
subsequent words of the form name=word shall be expanded in an
assignment context.

At line 81588 (XCU command APPLICATION USAGE), add:
The command utility takes on the parsing behavior of the command that it
is wrapping. Therefore,
 command command export a=~
is recognized as a declaration command, and sets the variable a to the
value of $HOME because it performs tilde-expansion of an assignment
context; while
 command echo a=~
outputs the literal string "a=~" because regular expansion can only
perform tilde-expansion at the beginning of the word. However, the
shell need only perform lexical analysis of the command_name argument
when deciding if command should be treated as a declaration command;
therefore, with:
 var=export; command $var a=~
it is unspecified whether the word 'a=~' is handled in an assignment
context or as a regular expansion.

At line 124658 (XRAT C.2.9.1), add:
Expansion of words in an assignment context following the command word
can only occur for declaration commands, and only when the word can be
used as a variable assignment in isolation.

For example, this code sequence exports the single variable a with the
value '1 b=2', but invokes make with the macro a set to '1' and b set
to '2', since make is not a declaration command:

$ set '1 b=2'
$ export a=$1
$ make a=$1

Conversely, this code sequence exports two variables, a set to '1' and b
set to '2', because the use of quoting means that the word could not be
recognized as a variable assignment, and regular expansion rules require
that field splitting occurs on the unquoted expansion of $1:

$ set '1 b=2'
$ export \a=$1

Likewise, this code sequence will not be parsed in assignment context,
but is still required to export the variable named foo with the value 1:

$ var=foo
$ export $var=1

Implementations are permitted to provide extensions that serve as
declaration commands, such as 'typeset' or 'local', or even a way to
define a function that can behave as a declaration command.

Declaration commands are only required to be recognized via lexical
analysis; if any expansions are required before the command name is
known, then it is unspecified whether subsequent arguments will be
treated with an assignment context during expansion. For example, it is
unspecified whether 'var=export; $var a=~' sets the variable a to a
literal <tilde> or to the value of $HOME, since lexical analysis sees
'$var' rather than 'export' as the command name.
(0000865)
Don Cragun (manager)
2011-06-23 15:28
edited on: 2011-08-25 15:25

Interpretation response
------------------------
The standard states that the current behavior of ksh93 does not conform, and conforming implementations must conform to this. However, concerns have been raised about this which are being referred to the sponsor.

Rationale:
-------------
Some shells behave as specified; some others match ksh93 behavior; and others have gone different routes.

Notes to the Editor (not part of this interpretation):
-------------------------------------------------------
Make the changes suggested in Note: 0000943.

(0000933)
ajosey (manager)
2011-08-10 15:24
edited on: 2011-08-26 08:55

Interpretation proposed 10 August 2011 for final 30 day review -- see note 946, which has the review clock being restarted.

(0000942)
geoffclare (manager)
2011-08-25 15:23

From mailing list item 16402 ...

Jilles Tjoelker raised some issues a while ago with the resolution of
bug 351 which should be addressed during the 30 day review ...

Jilles Tjoelker <jilles@stack.nl> wrote, on 25 Jun 2011:
>
> > New definition in XBD 3:
> > Declaration Command
> > A utility which can take arguments that cause variable assignments (of
> > the form varname=value) which will persist in the current shell
> > environment. During parsing, when the shell recognizes a declaration
> > command as the command name, then subsequent arguments that would be a
> > valid variable assignment in isolation are subject to different
> > expansion rules (field splitting and pathname expansion are suppressed,
> > and tilde expansion occurs after the <equals-sign> and any unquoted
> > <colon>). Arguments which are not a valid variable assignment in
> > isolation are processed according to normal argument expansion rules.
>
> The "During parsing" bit is only really needed for kshisms like
> readonly constants=(1 2 4)
> and conflicts with the later "It is unspecified whether assignment
> context will be used if the command name is not known..." if the
> implementation chooses to use assignment context in that case.

Yes, specifying "during parsing" forbids recognition at a later
stage, which we want to allow.

> > The following standard utilities are declaration commands: export,
> > readonly, and certain instances of command. An implementation may
> > provide other declaration commands.
>
> The wording blurs the distinction between "utility" and "command".
> Perhaps it helps to use "declaration utility" instead of "declaration
> command"; "certain instances of command" needs to be replaced with
> something like "command, under certain conditions".

I agree.

> > At line 81462 (XCU command DESCRIPTION), add:
> > The command utility shall be a declaration command if the command_name
> > argument is recognized as a declaration command. Therefore, during
> > parsing of a simple command, if command is recognized as the command
> > word and a declaration command is recognized as the next argument, then
> > subsequent words of the form name=word shall be expanded in an
> > assignment context.
>
> I read the first sentence to mean that also things like
> command -p export a=~
> command -- export a=~
> command -p -- export a=~
> get the special treatment. However, the second sentence denies it again
> and that's also what ksh93 does.
>
> Allowing the options would be more orthogonal but not particularly
> useful and the implementations could become uglier.

I can see that there is no need to handle "command -p" since declaration
utilities are always builtins, but I'm a little surprised that
"command -- export ..." doesn't work in ksh93. Nevertheless, since it
is existing practice we should allow the ksh93 behaviour.

I have attempted to make the necessary changes to address all these
points in the following note (an updated version of note 618).
(0000943)
geoffclare (manager)
2011-08-25 15:24

New definition in XBD 3:
Declaration Utility
A utility which can take arguments that cause variable assignments (of
the form varname=value) which will persist in the current shell
environment. When the shell recognizes a declaration utility as the
command name, subsequent arguments that would be a
valid variable assignment in isolation are subject to different
expansion rules (field splitting and pathname expansion are suppressed,
and tilde expansion occurs after the <equals-sign> and any unquoted
<colon>). Arguments which are not a valid variable assignment in
isolation are processed according to normal argument expansion rules.

The following standard utilities are declaration utilities: export,
readonly, and, under certain conditions, command. An implementation may
provide other declaration utilities.

At line 3276, XBD 4.22 Variable Assignment, add:
Assignment context occurs in the cmd_prefix portion of a shell simple
command, as well as in arguments of a recognized declaration utility.

At line 71964, XCU 1.4 Utility Description Defaults DESCRIPTION, add:
A standard utility shall not be treated as a declaration utility unless
explicitly stated in this section.

At line 73093 (XCU 2.9.1 step 2), add:
If the command name is recognized as a declaration utility, then any
remaining words that would be recognized as a variable assignment in
isolation shall be expanded as a variable assignment (tilde expansion
after the first <equals-sign> and after any unquoted <colon>, parameter
expansion, command substitution, arithmetic expansion, and quote
removal, but no field splitting or pathname expansion); while words that
would not be a variable assignment in isolation shall be subject to regular
expansion. For all other command names, all subsequent words shall be
subject to regular expansion (tilde expansion for only a leading <tilde>,
parameter expansion, command substitution, arithmetic expansion, field
splitting, pathname expansion, and quote removal).

After line 73100, add a paragraph:
When determining whether a command name is a declaration utility, an
implementation may use only lexical analysis. It is unspecified whether
assignment context will be used if the command name would only
become recognized as a declaration utility after word expansions.

At line 74276 (XCU export DESCRIPTION), add:
The export special built-in shall be a declaration utility. Therefore,
if export is recognized as the command word of a simple command, then
subsequent words of the form name=word shall be expanded in an
assignment context. See Section 2.9.1.

At line 74360 (XCU readonly DESCRIPTION), add:
The readonly special built-in shall be a declaration utility.
Therefore, if readonly is recognized as the command word of a simple
command, then subsequent words of the form name=word shall be expanded
in an assignment context. See Section 2.9.1.

At line 81462 (XCU command DESCRIPTION), add:
The command utility shall be treated as a declaration utility if the
first argument passed to the utility is recognized as a declaration utility.
In this case, subsequent words of the form name=word shall be expanded
in an assignment context. See Section 2.9.1.

At line 81588 (XCU command APPLICATION USAGE), add:
The command utility takes on the expansion behavior of the command that it
is wrapping. Therefore, in
 command command export a=~
command is recognized as a declaration utility, and the command sets the
variable a to the value of $HOME because it performs tilde-expansion
of an assignment context; while
 command echo a=~
outputs the literal string "a=~" because regular expansion can only
perform tilde-expansion at the beginning of the word. However, the
shell need only perform lexical analysis of the next argument when
deciding if command should be treated as a declaration utility;
therefore, with:
 var=export; command $var a=~
and
 command -- export a=~
it is unspecified whether the word 'a=~' is handled in an assignment
context or as a regular expansion.

At line 124658 (XRAT C.2.9.1), add:
Expansion of words in an assignment context following the command word
can only occur for declaration utilities, and only when the word can be
used as a variable assignment in isolation.

For example, this code sequence exports the single variable a with the
value '1 b=2', but invokes make with the macro a set to '1' and b set
to '2', since make is not a declaration utility:

$ set '1 b=2'
$ export a=$1
$ make a=$1

Conversely, this code sequence exports two variables, a set to '1' and b
set to '2', because the use of quoting means that the word could not be
recognized as a variable assignment, and regular expansion rules require
that field splitting occurs on the unquoted expansion of $1:

$ set '1 b=2'
$ export \a=$1

Likewise, this code sequence will not be parsed in assignment context,
but is still required to export the variable named foo with the value 1:

$ var=foo
$ export $var=1

Implementations are permitted to provide extensions that serve as
declaration utilities, such as 'typeset' or 'local', or even a way to
define a function that can behave as a declaration utility.

Declaration utilities are only required to be recognized via lexical
analysis; if any expansions are required before the command name is
known, or before the first argument to the command utility is known,
then it is unspecified whether subsequent arguments will be
treated with an assignment context during expansion. For example, it is
unspecified whether 'var=export; $var a=~' sets the variable a to a
literal <tilde> or to the value of $HOME, since lexical analysis sees
'$var' rather than 'export' as the command name.
(0000946)
ajosey (manager)
2011-08-26 08:54

After comments were raised on the reflector in mail seq 16402,
the bug was discussed at the 25 August 2011 teleconference,
and the proposed interpretation was revised.

The review time for the interpretation is being restarted.

This revised Interpretation was proposed 26 August 2011 for final 30 day review
(0000980)
ajosey (manager)
2011-10-04 07:36

The interpretation has now been approved.
(0003026)
mirabilos (reporter)
2016-01-19 23:15

| Here, mksh recognizes the first command word as 'export', and performs
| assignment-style tilde expansion; but ksh performs textual recognition

This is actually wrong… mksh in default mode always expands tildes
after equals signs. What you want is mksh -o posix -c …

- Issue History
Date Modified Username Field Change
2010-11-09 18:42 eblake New Issue
2010-11-09 18:42 eblake Status New => Under Review
2010-11-09 18:42 eblake Assigned To => ajosey
2010-11-09 18:42 eblake Name => Eric Blake
2010-11-09 18:42 eblake Organization => Red Hat
2010-11-09 18:42 eblake User Reference => ebb.export
2010-11-09 18:42 eblake Section => 2.9.1 Simple Commands
2010-11-09 18:42 eblake Page Number => 2316
2010-11-09 18:42 eblake Line Number => 73089
2010-11-09 18:42 eblake Interp Status => ---
2010-11-09 18:46 eblake Note Added: 0000613
2010-11-18 16:43 eblake Note Added: 0000618
2011-06-23 15:28 Don Cragun Interp Status --- => Pending
2011-06-23 15:28 Don Cragun Note Added: 0000865
2011-06-23 15:28 Don Cragun Status Under Review => Interpretation Required
2011-06-23 15:28 Don Cragun Resolution Open => Accepted As Marked
2011-06-23 15:29 Don Cragun Final Accepted Text => See Note: 0000865
2011-06-23 15:30 Don Cragun Tag Attached: issue8
2011-08-10 15:24 ajosey Interp Status Pending => Proposed
2011-08-10 15:24 ajosey Note Added: 0000933
2011-08-25 15:23 geoffclare Note Added: 0000942
2011-08-25 15:24 geoffclare Note Added: 0000943
2011-08-25 15:25 Don Cragun Note Edited: 0000865
2011-08-26 08:54 ajosey Note Added: 0000946
2011-08-26 08:55 ajosey Note Edited: 0000933
2011-10-04 07:36 ajosey Interp Status Proposed => Approved
2011-10-04 07:36 ajosey Note Added: 0000980
2014-04-10 14:44 eblake Relationship added related to 0000352
2016-01-19 23:15 mirabilos Note Added: 0003026
2019-02-14 16:43 eblake Relationship added related to 0001172
2020-02-10 15:16 geoffclare Status Interpretation Required => Applied
2020-09-21 15:48 eblake Relationship added parent of 0001393
2021-11-19 09:45 geoffclare Relationship added related to 0001535
2023-10-28 05:10 Don Cragun Relationship added child of 0001784
2023-10-28 05:20 Don Cragun Relationship replaced related to 0001393
2023-10-28 06:30 Don Cragun Relationship added related to 0001785
2023-10-28 06:34 Don Cragun Relationship deleted child of 0001784


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker