0001276: incorrect resolution in bugid:839 - Austin Group Defect Tracker

(0004501)
stephane (reporter)
2019-07-30 13:16

Note that rule 8 (used in the fname production) defers to 7, but doesn't really need to.

What matters there is that a NAME token be not returned when the token doesn't form a valid variable name, so whether rule 7 classifies it as WORD or ASSIGNMENT_WORD doesn't really matter as long as it's not the NAME token

(0004508)
stephane (reporter)
2019-08-05 14:23

We may also want to rename "cmd_name" to something else as it's potentially misleading.

In

cmd arg

cmd is the WORD token identified as "cmd_name"

In

var=value < file cmd arg

cmd is identified as "cmd_word".

In those two examples, "cmd" is the "name of the command" being executed, but neither cmd_word nor cmd_name have to be the command's name like in $(echo cmd arg1) arg2 where the cmd_name is $(echo cmd arg1) but the command's name is "cmd" (assuming the default value of $IFS) or dryrun=; $dryrun cmd arg where cmd_name is $dryrun but the command name is "cmd".

The distinction between cmd_name and cmd_word here is about the token having different constraints when it's preceded by redirections/assignments and not (namely whether keywords are allowed).

Maybe "cmd_word_no_keyword" would be a better wording for "cmd_name".

(0004509)
stephane (reporter)
2019-08-05 14:23

One could also argue that forcing shells to interpret keywords as WORDs when preceded by assignments/redirections is not particularly useful.

Nobody's going to write:

foo=bar for arg

And expect that "for" to be looked up in $PATH.

On the other hand, a shell implementation may want to allow:

2> /dev/null [[ $a -eq $b ]]

Or

TIMEFMT=3 time cmd
...

Which would help with consistency, but is currently not allowed by POSIX as POSIX requires those to be interpreted as simple commands.

(0004530)
geoffclare (manager)
2019-08-19 10:05
edited on: 2019-08-19 10:12

The new wording for rule 7 suggested in the desired action looks good to me (but noting it has further changes proposed in bug 0001279), although it is missing the word "token" in the third bullet item, which should be:

Otherwise, it is unspecified whether the token WORD or ASSIGNMENT_WORD is returned.

Re Note: 0004508 either I'm confused or you have the new names the wrong way round - cmd_word is the one that can't be a keyword. So a new name for cmd_name should imply that it can be a keyword, not that it can't.

Re Note: 0004509 how would the rule 7 wording in the desired action need to be changed if we want to allow either behaviour?

(0004532)
stephane (reporter)
2019-08-20 18:34

Re: Note: 0004530

About Note: 0004508, yes sorry my bad.

About Note: 0004509

That could be: change:

> simple_command : cmd_prefix cmd_word cmd_suffix
> | cmd_prefix cmd_word
> | cmd_prefix
> | cmd_name cmd_suffix
> | cmd_name
> ;
> cmd_name : WORD /* Apply rule 7a */
> ;
> cmd_word : WORD /* Apply rule 7b */
> ;
> cmd_prefix : io_redirect
> | cmd_prefix io_redirect
> | ASSIGNMENT_WORD
> | cmd_prefix ASSIGNMENT_WORD

to (including 0001094 resolution (also included in 0001279)):

> simple_command : cmd_prefix cmd_word cmd_suffix
> | cmd_prefix cmd_word
> | cmd_prefix
> | cmd_word cmd_suffix
> | cmd_word
> ;
> cmd_word : WORD /* Apply rule 7 */
> ;
> cmd_prefix : io_redirect
> | cmd_prefix io_redirect
> | ASSIGNMENT_WORD /* Apply rule 7 */
> | cmd_prefix ASSIGNMENT_WORD /* Apply rule 7 */
> cmd_suffix : io_redirect
> | cmd_suffix io_redirect
> | WORD
> | cmd_suffix WORD

and rule 7 to:

> 7. [Assignment preceding command name]
>
> * If the TOKEN is exactly a reserved word, the token
> identifier for that reserved word shall result.
>
> * Otherwise
>
> If the TOKEN contains an unquoted (as determined
> while applying rule 4 from Token Recognition)
> <equals-sign> character that is not part of an
> embedded parameter expansion, command substitution,
> or arithmetic expansion construct (as determined
> while applying rule 5 from Token Recognition):
>
> • If the TOKEN begins with '=', then the token
> WORD shall be returned.
>
> • If all the characters in the TOKEN preceding
> the first such <equals-sign> form a valid name
> (see XBD Name), the token ASSIGNMENT_WORD shall
> be returned.
>
> • Otherwise, it is unspecified whether the WORD
> or ASSIGNMENT_WORD is returned.
>
> * Otherwise, the token WORD shall be returned.
>
> Assignment to the name within a returned ASSIGNMENT_WORD
> token shall occur as specified in Simple Commands.

And remove the "Otherwise, rule 7 applies" from rule 8 which doesn't make much sense (rule 7 will never yield a NAME token). Or replace with "Otherwise, a UNSPECIFIED token is returned" (see 0001279).

That would specify that "foo=bar for arg" is not any more a valid POSIX sh "simple command" than "for arg".

And foo=bar() { blah; } would still not be a valid sh function declaration, as foo=bar would still not be seen as a NAME token.

(0004534)
stephane (reporter)
2019-08-22 06:37

Note that 0000351 (about [command [-p]] export/readonly treating what looks like ASSIGNMENT_WORD specially) is related, but it doesn't seem like the proposed resolutions here would affect it.

That bug should be kept in mind when touching rule 7.

(0005882)
geoffclare (manager)
2022-07-11 15:55

Line numbers are for Issue 8 draft 2.1

Change line 75820 from:

If the TOKEN does not contain the character '=', rule 1 is applied. Otherwise, 7b shall be applied.

to:

If the TOKEN is exactly a reserved word, the token identifier for that reserved word shall result. Otherwise, 7b shall be applied.

Change line 75827 from:

If the TOKEN begins with '=', then rule 1 shall be applied.

to:

If the TOKEN begins with '=', then the token WORD shall be returned.

After applying bug 1279, change line 75831 from:

Otherwise, it is implementation-defined whether rule 1 is applied, ASSIGNMENT_WORD is returned, or the TOKEN is processed in some other way.

to:

Otherwise, it is implementation-defined whether the token WORD or ASSIGNMENT_WORD is returned, or the TOKEN is processed in some other way.

Change line 75833 from:

Otherwise, rule 1 shall be applied.

to:

Otherwise, the token WORD shall be returned.

(0005883)
geoffclare (manager)
2022-07-11 16:02

Note: 0004509 was discussed in the July 11 teleconference. It was decided that for expediency just the minimum changes needed to fix the originally reported problem would be made for this bug. If additional changes for Note: 0004509 are still desired, these should be submitted as a separate bug.

Issue History
Date Modified	Username	Field	Change
2019-07-30 13:11	stephane	New Issue
2019-07-30 13:11	stephane	Name	=> Stephane Chazelas
2019-07-30 13:11	stephane	Section	=> 2.10.2 shell grammar rules
2019-07-30 13:16	stephane	Note Added: 0004501
2019-07-30 14:26	eblake	Relationship added	child of 0000839
2019-07-30 14:27	eblake	Relationship added	related to 0001100
2019-08-05 14:23	stephane	Note Added: 0004508
2019-08-05 14:23	stephane	Note Added: 0004509
2019-08-19 10:05	geoffclare	Note Added: 0004530
2019-08-19 10:12	geoffclare	Note Edited: 0004530
2019-08-20 18:34	stephane	Note Added: 0004532
2019-08-22 06:37	stephane	Note Added: 0004534
2019-12-31 11:09	dennisw	Issue Monitored: dennisw
2022-07-11 15:55	geoffclare	Note Added: 0005882
2022-07-11 15:56	geoffclare	Interp Status	=> ---
2022-07-11 15:56	geoffclare	Final Accepted Text	=> Note: 0005882
2022-07-11 15:56	geoffclare	Status	New => Resolved
2022-07-11 15:56	geoffclare	Resolution	Open => Accepted As Marked
2022-07-11 15:57	geoffclare	Tag Attached: tc3-2008
2022-07-11 16:02	geoffclare	Note Added: 0005883
2022-08-05 09:26	geoffclare	Status	Resolved => Applied
2024-06-11 08:54	agadmin	Status	Applied => Closed

Notes
(0004501) stephane (reporter) 2019-07-30 13:16	Note that rule 8 (used in the fname production) defers to 7, but doesn't really need to. What matters there is that a NAME token be not returned when the token doesn't form a valid variable name, so whether rule 7 classifies it as WORD or ASSIGNMENT_WORD doesn't really matter as long as it's not the NAME token

(0004508) stephane (reporter) 2019-08-05 14:23	We may also want to rename "cmd_name" to something else as it's potentially misleading. In cmd arg cmd is the WORD token identified as "cmd_name" In var=value < file cmd arg cmd is identified as "cmd_word". In those two examples, "cmd" is the "name of the command" being executed, but neither cmd_word nor cmd_name have to be the command's name like in $(echo cmd arg1) arg2 where the cmd_name is $(echo cmd arg1) but the command's name is "cmd" (assuming the default value of $IFS) or dryrun=; $dryrun cmd arg where cmd_name is $dryrun but the command name is "cmd". The distinction between cmd_name and cmd_word here is about the token having different constraints when it's preceded by redirections/assignments and not (namely whether keywords are allowed). Maybe "cmd_word_no_keyword" would be a better wording for "cmd_name".

(0004509) stephane (reporter) 2019-08-05 14:23	One could also argue that forcing shells to interpret keywords as WORDs when preceded by assignments/redirections is not particularly useful. Nobody's going to write: foo=bar for arg And expect that "for" to be looked up in $PATH. On the other hand, a shell implementation may want to allow: 2> /dev/null [[ $a -eq $b ]] Or TIMEFMT=3 time cmd ... Which would help with consistency, but is currently not allowed by POSIX as POSIX requires those to be interpreted as simple commands.

(0004530) geoffclare (manager) 2019-08-19 10:05 edited on: 2019-08-19 10:12	The new wording for rule 7 suggested in the desired action looks good to me (but noting it has further changes proposed in bug 0001279), although it is missing the word "token" in the third bullet item, which should be: Otherwise, it is unspecified whether the token WORD or ASSIGNMENT_WORD is returned. Re Note: 0004508 either I'm confused or you have the new names the wrong way round - cmd_word is the one that can't be a keyword. So a new name for cmd_name should imply that it can be a keyword, not that it can't. Re Note: 0004509 how would the rule 7 wording in the desired action need to be changed if we want to allow either behaviour?

(0004532) stephane (reporter) 2019-08-20 18:34	Re: Note: 0004530 About Note: 0004508, yes sorry my bad. About Note: 0004509 That could be: change: > simple_command : cmd_prefix cmd_word cmd_suffix > \| cmd_prefix cmd_word > \| cmd_prefix > \| cmd_name cmd_suffix > \| cmd_name > ; > cmd_name : WORD /* Apply rule 7a / > ; > cmd_word : WORD / Apply rule 7b / > ; > cmd_prefix : io_redirect > \| cmd_prefix io_redirect > \| ASSIGNMENT_WORD > \| cmd_prefix ASSIGNMENT_WORD to (including 0001094 resolution (also included in 0001279)): > simple_command : cmd_prefix cmd_word cmd_suffix > \| cmd_prefix cmd_word > \| cmd_prefix > \| cmd_word cmd_suffix > \| cmd_word > ; > cmd_word : WORD / Apply rule 7 / > ; > cmd_prefix : io_redirect > \| cmd_prefix io_redirect > \| ASSIGNMENT_WORD / Apply rule 7 / > \| cmd_prefix ASSIGNMENT_WORD / Apply rule 7 / > cmd_suffix : io_redirect > \| cmd_suffix io_redirect > \| WORD > \| cmd_suffix WORD and rule 7 to: > 7. [Assignment preceding command name] > > If the TOKEN is exactly a reserved word, the token > identifier for that reserved word shall result. > > * Otherwise > > If the TOKEN contains an unquoted (as determined > while applying rule 4 from Token Recognition) > <equals-sign> character that is not part of an > embedded parameter expansion, command substitution, > or arithmetic expansion construct (as determined > while applying rule 5 from Token Recognition): > > • If the TOKEN begins with '=', then the token > WORD shall be returned. > > • If all the characters in the TOKEN preceding > the first such <equals-sign> form a valid name > (see XBD Name), the token ASSIGNMENT_WORD shall > be returned. > > • Otherwise, it is unspecified whether the WORD > or ASSIGNMENT_WORD is returned. > > * Otherwise, the token WORD shall be returned. > > Assignment to the name within a returned ASSIGNMENT_WORD > token shall occur as specified in Simple Commands. And remove the "Otherwise, rule 7 applies" from rule 8 which doesn't make much sense (rule 7 will never yield a NAME token). Or replace with "Otherwise, a UNSPECIFIED token is returned" (see 0001279). That would specify that "foo=bar for arg" is not any more a valid POSIX sh "simple command" than "for arg". And foo=bar() { blah; } would still not be a valid sh function declaration, as foo=bar would still not be seen as a NAME token.

(0004534) stephane (reporter) 2019-08-22 06:37	Note that 0000351 (about [command [-p]] export/readonly treating what looks like ASSIGNMENT_WORD specially) is related, but it doesn't seem like the proposed resolutions here would affect it. That bug should be kept in mind when touching rule 7.

(0005882) geoffclare (manager) 2022-07-11 15:55	Line numbers are for Issue 8 draft 2.1 Change line 75820 from: If the TOKEN does not contain the character '=', rule 1 is applied. Otherwise, 7b shall be applied. to: If the TOKEN is exactly a reserved word, the token identifier for that reserved word shall result. Otherwise, 7b shall be applied. Change line 75827 from: If the TOKEN begins with '=', then rule 1 shall be applied. to: If the TOKEN begins with '=', then the token WORD shall be returned. After applying bug 1279, change line 75831 from: Otherwise, it is implementation-defined whether rule 1 is applied, ASSIGNMENT_WORD is returned, or the TOKEN is processed in some other way. to: Otherwise, it is implementation-defined whether the token WORD or ASSIGNMENT_WORD is returned, or the TOKEN is processed in some other way. Change line 75833 from: Otherwise, rule 1 shall be applied. to: Otherwise, the token WORD shall be returned.

(0005883) geoffclare (manager) 2022-07-11 16:02	Note: 0004509 was discussed in the July 11 teleconference. It was decided that for expediency just the minimum changes needed to fix the originally reported problem would be made for this bug. If additional changes for Note: 0004509 are still desired, these should be submitted as a separate bug.

Aardvark Mark IV