Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0001393 [Issue 8 drafts] Shell and Utilities Objection Error 2020-08-24 19:15 2020-10-08 10:37
Reporter McDutchie View Status public  
Assigned To
Priority normal Resolution Accepted As Marked  
Status Applied   Product Version Draft 1
Name Martijn Dekker
Organization https://github.com/ksh93 [^]
User Reference
Section XCU 3
Page Number 2510
Line Number 82523-82525
Final Accepted Text Note: 0005003
Summary 0001393: 'command' should not be treated as a declaration utility
Description The current POSIX draft adds three lines to the specification of the 'command' utility, lines 82523-82525:

| The command utility shall be treated as a declaration utility if the
| first argument passed to the utility is recognized as a declaration
| utility. In this case, subsequent words of the form name=word shall
| be expanded in an assignment context. See Section 2.9.1.1 (on page
| 2292).

(As an aside, if I were to be pedantic, I could argue that the phrasing "if the first argument passed to the utility is recognized as a declaration utility" effectively makes this behaviour unspecified; a 'command' utility implementation could be written not to recognise any utility as such and this phrasing would allow that. I am assuming that this is not the intended meaning.)

This addition effectively makes the 'command' utility into a kind of "Schrödinger's declaration utility" that is simultaneously a declaration utility and not a declaration utility. I think that is a bad idea, for the following three reasons:


***1.*** In the currently published release version of POSIX, 'command' is a regular utility (grammatically no different from, say, 'ls' or 'printf'), and there is nothing that allows regular utilities to treat their arguments specially in a shell grammar sense. So it follows that declaration utility behaviour *must* be disabled along with all other special behaviour by the 'command' utility as specified in the release version.

Consequently, the behaviour specified in the draft is incompatible with the behaviour specified in the current release version, as it is not possible to be simultaneously compliant with both.

(A similar argument could be made for 'export' and 'readonly' themselves, but at least they are already special built-in utilities, so the inconsistently is not so serious.)


***2.*** It does not specify sufficiently widespread current practice. For the following test:

   $shell -c 'v="a y=b"; command export x=$v; echo "[$x] [$y]"'

bash, bosh (Schily), dash < 0.5.11, NetBSD sh, pdksh (including OpenBSD (k)sh), yash, and 'zsh --emulate sh' output:

   [a] [b]

meaning they do *not* treat 'command export' as a declaration utility.

Dash >= 0.5.11, its derivative gwsh, FreeBSD sh, ksh93, and mksh output

   [a y=b] []

meaning they do treat 'command export' as a declaration utility. But together they are not enough to establish a de facto standard behaviour.


***3.*** Since 'command' is specified as a regular utility, both of the following *must* work:

    c='command'; "$c" export foo=bar
    set -- command export; "$@" foo=bar

So, in the draft version, this effectively means that the shell must determine at runtime whether to treat 'command' as a declaration utility, and not at parse time. And that requirement is both innovative and unreasonable.

For the following test:

    $shell -c 'v="a y=b"; set -- command export; "$@" x=$v; echo "[$x] [$y]"'

only dash 0.5.11 and its derivative gwsh currently manage to output '[a y=b] []' meaning they recognise 'command export' as a declaration utility at runtime, and all other shells (except ksh93 which breaks completely[*]) output '[a] [b]' meaning they don't.

[*] ksh93 (which I currently maintain the only actively and openly developed fork of) has a bug that breaks this test case altogether -- the 'export' utility is not run at all; instead, the equivalent of 'command -v export' is run. It would be good if I could implement a clean fix, which would disassociate the 'command' utility from the parser altogether and treat it as a regular built-in utility as currently specified in the release version.
Desired Action Delete lines 82523-82525.

Alternatively, change these lines to the following:

"If the first argument passed to the command utility is the name of a declaration utility (see Section 2.9.1.1 on page 2292), it is unspecified whether subsequent words of the form name=word are expanded in an assignment context."
Tags issue8
Attached Files

- Relationships
related to 0000351Appliedajosey 1003.1(2008)/Issue 7 certain shell special built-ins should expand arguments in assignment context 
related to 0001785Resolved Issue 8 drafts Conflict in specification of processing of declaration utilities 

-  Notes
(0004943)
McDutchie (reporter)
2020-08-24 19:16

Argh. Could someone please edit the summary to say "'command' should NOT be treated as a declaration utility"? Thanks and apologies.
(0004944)
Don Cragun (manager)
2020-08-24 23:09

Re Note: 0004943: Done,
(0004945)
geoffclare (manager)
2020-08-25 08:25

I have moved this to the Issue 8 drafts project and set the product version to Draft 1.
(0004971)
McDutchie (reporter)
2020-09-04 03:53

I had written: "So, in the draft version, this effectively means that the shell must determine at runtime whether to treat 'command' as a declaration utility, and not at parse time. And that requirement is both innovative and unreasonable."

I'd just like to expound this point a little. On shells where shell assignments allow special grammar like "a=(foo bar)" (which are bash, ksh93, mksh and zsh), the 'command' utility must be integrated into the parser if it is to preserve a declaration command's assignment behaviour including grammatical properties. ksh93 does this; bash, mksh, and zsh do not. So

    command typeset a=(1 2 3)

works on ksh93, but throws a syntax error on the others (zsh attempts and fails to interpret it as some kind of zsh-style pattern).

Of course this can only work on ksh93 because its command builtin is integrated into the parser (in ways that also cause multiple bugs including one serious bug, see footnote above). The assignments are recognised at parse time, not at runtime. If assignments can use special grammar such as parentheses, then it is not feasible or even possible to do this at runtime. Dash manages it because it doesn't support assignments with special syntax.

So, I think ksh93 is a case study on the reasons why it is a bad idea for 'command' to be anything else than a plain old regular builtin utility.
(0004972)
geoffclare (manager)
2020-09-04 09:34

Re: Note: 0004971 There is no requirement for this to be determined at runtime. The APPLICATION USAGE on page 2513 line 82660 says that the shell need only perform lexical analysis of the next argument when deciding if command should be treated as a declaration utility.

(There is an editorial error here: "command" should be in italics. We should fix that if the text here is otherwise unchanged by this bug.)
(0004973)
joerg (reporter)
2020-09-04 11:43

Re: Note: 0004971

bosh only looks at the flags in it's builtin command data structures if a builtin is a special builtin and in this case does not make variable assignments a push operation that is popped after the builtin command terminated.

My impression is that this best fits to the previous shell behavior.

BTW: is there a list with all ksh93 related problems?
(0005002)
joerg (reporter)
2020-09-24 14:55

It seems that before 0000351 no shell had a special treatment for quoting for the arguments of "export".

So a requirement for this behavior looks like invention.
(0005003)
geoffclare (manager)
2020-09-24 15:55

On page 2332 line 75618 section 2.14 export (APPLICATION USAGE), add a new paragraph:
In shells that support extended assignment syntax, for example to allow an array to be populated with a single assignment, such extensions can typically only be used in assignments specified as arguments to export if the command word is literally export, and not if it is some other word that expands to export. For example:
# Shells that support array assignment as an extension generally support this:
export x=(1 2 3); echo ${x[0]}  # outputs 1
# But generally do not support this:
e=export; $e x=(1 2 3); echo ${x[0]}  # syntax error

On page 2332 line 75637 section 2.14 export (RATIONALE), add a new paragraph:
Some implementations extend the shell's assignment syntax, for example to allow an array to be populated with a single assignment, and in order for such an extension to be usable in assignments specified as arguments to export these shells have export as a separate token in their grammar. This standard only permits an extension of this nature when the input to the shell would contain a syntax error according to the standard grammar. Note that although export can be a separate token in the shell's grammar, it cannot be a reserved word since export is a candidate for alias substitution whereas reserved words are not (see [xref to 2.3.1]).

On page 2335 line 75721 section 2.14 readonly (APPLICATION USAGE), change:
None.
to:
In shells that support extended assignment syntax, for example to allow an array to be populated with a single assignment, such extensions can typically only be used in assignments specified as arguments to readonly if the command word is literally readonly, and not if it is some other word that expands to readonly. For example:
# Shells that support array assignment as an extension generally support this:
readonly x=(1 2 3); echo ${x[0]}  # outputs 1
# But generally do not support this:
r=readonly; $r x=(1 2 3); echo ${x[0]}  # syntax error

On page 2335 line 75739 section 2.14 readonly (RATIONALE), add a new paragraph:
Some implementations extend the shell's assignment syntax, for example to allow an array to be populated with a single assignment, and in order for such an extension to be usable in assignments specified as arguments to readonly these shells have readonly as a separate token in their grammar. This standard only permits an extension of this nature when the input to the shell would contain a syntax error according to the standard grammar. Note that although readonly can be a separate token in the shell's grammar, it cannot be a reserved word since readonly is a candidate for alias substitution whereas reserved words are not (see [xref to 2.3.1]).

On page 2514 line 82700 section command (RATIONALE), change:
... or that prevents function lookup on b or c.
to:
... or that prevents function lookup on b or c. However, some implementations extend the shell's assignment syntax, for example to allow an array to be populated with a single assignment, and in order for such an extension to be usable in assignments specified as arguments to export and readonly these shells have those utility names as separate tokens in their grammar. When command is used to execute these utilities it also needs to be a separate token in the grammar so that the same extended assignment syntax can still be recognized in this case. This standard only permits an extension of this nature when the input to the shell would contain a syntax error according to the standard grammar, and therefore it cannot affect how '|' and ';' are parsed in the example above. Note that although command can be a separate token in the shell's grammar, it cannot be a reserved word since command is a candidate for alias substitution whereas reserved words are not (see [xref to 2.3.1]).

- Issue History
Date Modified Username Field Change
2020-08-24 19:15 McDutchie New Issue
2020-08-24 19:15 McDutchie Name => Martijn Dekker
2020-08-24 19:15 McDutchie Organization => https://github.com/ksh93 [^]
2020-08-24 19:15 McDutchie Section => XCU 3
2020-08-24 19:15 McDutchie Page Number => 2510
2020-08-24 19:15 McDutchie Line Number => 82523-82525
2020-08-24 19:16 McDutchie Note Added: 0004943
2020-08-24 23:09 Don Cragun Interp Status => ---
2020-08-24 23:09 Don Cragun Note Added: 0004944
2020-08-24 23:09 Don Cragun Summary 'command' should be treated as a declaration utility => 'command' should not be treated as a declaration utility
2020-08-25 08:24 geoffclare Project 1003.1(2016/18)/Issue7+TC2 => Issue 8 drafts
2020-08-25 08:25 geoffclare Note Added: 0004945
2020-08-25 08:25 geoffclare version => Draft 1
2020-09-04 03:53 McDutchie Note Added: 0004971
2020-09-04 09:34 geoffclare Note Added: 0004972
2020-09-04 11:43 joerg Note Added: 0004973
2020-09-21 15:48 eblake Relationship added child of 0000351
2020-09-24 14:55 joerg Note Added: 0005002
2020-09-24 15:55 geoffclare Note Added: 0005003
2020-09-24 15:56 geoffclare Final Accepted Text => Note: 0005003
2020-09-24 15:56 geoffclare Status New => Resolved
2020-09-24 15:56 geoffclare Resolution Open => Accepted As Marked
2020-09-24 15:56 geoffclare Tag Attached: issue8
2020-10-08 10:37 geoffclare Status Resolved => Applied
2023-10-28 05:10 Don Cragun Relationship added related to 0001784
2023-10-28 05:20 Don Cragun Relationship replaced related to 0000351
2023-10-28 06:28 Don Cragun Relationship added related to 0001785
2023-10-28 06:28 Don Cragun Relationship deleted related to 0001784


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker