Austin Group Defect Tracker

Aardvark Mark IV

Main | My View | View Issues | Change Log | Docs
  

Viewing Issue Simple Details [ Jump to Notes ] [ Issue History ] [ Print ]
ID Category Severity Type Date Submitted Last Update
0001191 [1003.1(2016/18)/Issue7+TC2] Shell and Utilities Editorial Omission 2018-04-13 14:33 2024-06-11 09:08
Reporter geoffclare View Status public  
Assigned To
Priority normal Resolution Accepted As Marked  
Status Closed  
Name Geoff Clare
Organization The Open Group
User Reference
Section 2.2 Quoting
Page Number 2346
Line Number 74697
Interp Status ---
Final Accepted Text See Note: 0004278.
Summary 0001191: The list of special characters in 2.2 is missing '-' and '!'
Description Section 2.2 lists special characters that either always or sometimes
need quoting if they are to represent themselves. It is missing the
characters '-' and '!' which are sometimes special (in bracket
expressions).

This appears to be an editorial oversight, since the purpose of this
introductory text seems to be to warn shell script writers about which
characters they need to think about quoting if they want them to be
treated literally, rather than stating a normative requirement about
which characters can be special. Luckily the lists here are not
referenced by places such as 2.2.3; had that been the case, this would
be a non-editorial matter.

The text here also misuses the term "may", which is supposed to indicate
"a feature or behavior that is optional for an implementation" (XBD 1.5).
Desired Action Change:
and the following may need to be quoted under certain circumstances. That is, these characters may be special depending on conditions described elsewhere in this volume of POSIX.1-2008:
*   ?   [   #   ~   =   %
to:
and the following might need to be quoted under certain circumstances. That is, these characters are sometimes special depending on conditions described elsewhere in this volume of POSIX.1-2008:
*   ?   [   -   !   #   ~   =   %
Tags tc3-2008
Attached Files

- Relationships
related to 0001193Closed Brace expansion and {var}>file redirects in the shell 

-  Notes
(0003956)
geoffclare (manager)
2018-04-13 14:44

 I suppose '^' ought to be included as well, since leaving it unquoted when it is the first character in a bracket expression produces unspecified results.

Not sure whether it would be okay just to include it in the second list, or if it needs a separate statement (because of the unspecified results thing).
(0003971)
shware_systems (reporter)
2018-04-19 05:54
edited on: 2018-04-19 05:55

 As that is only listing what chars require it sometimes, not why's, to me just including '^' ok... Xrefs in XRAT for this section to the why's for each char may be desirable.
(0003973)
stephane (reporter)
2018-04-19 15:32

 If you include ^, !, -, then you should also include ] as I take it it's linked to the recent discussion whereby when used in wildcards in shells (and possibly also in fnmatch with backslash), quoting ^, !, -, ] removes their special meaning within bracket expressions.


Also { and } should be quoted under some circumstance in many shells (bash (also in sh mode, I don't know if that applies to certified ones like on macOS), ksh93, zsh, pdksh, mksh (except with -o posix)) for brace expansion at least, it may be time to acknowledge it though it may warrant a separate ticket.

I would say it's also time to *require* ^ be understood as equivalent to !. That's what most people (who haven't been exposed to ancient Bourne or ksh88 shells) expect nowadays. It's silly to deviate from regexps here when ^ is no longer a pipe operator in POSIX shells.
(0003975)
geoffclare (manager)
2018-04-20 09:52

 Re Note: 0003973 I agree ] should be in the list. I think { and } should be raised separately as it's questionable whether brace expansion is allowed by the standard. Requiring ^ to be equivalent to ! would also best be raised as a separate bug.
(0003976)
chet_ramey (reporter)
2018-04-20 13:30

 Why would brace expansion not be allowed by the standard? `{' is a reserved word, not an operator, and does not qualify as a reserved word when used in a brace expansion context.
(0003977)
stephane (reporter)
2018-04-20 13:39
edited on: 2018-04-20 15:06

 Re: Note: 0003976

Currently, AFAICT, POSIX requires 
echo {a,b}
to output "{a,b}" and 
echo {fd}>&2
to output "{fd}" on stderr, making pdksh/ksh93/bash non-conformant (zsh disables those in sh mode).

IIRC, there's some wording that says the "{" reserved word is only recognised as such in command position. But even then "{echo foo}" where we have a "{echo" word and not "{" is still required to run the "{echo" command with "foo}" as argument.
(0003978)
chet_ramey (reporter)
2018-04-20 13:52

 (Sorry, I sent this only to Geoff first.)

If it's an ordinary character, any additional interpretation falls outside
the standard. A brace expansion is a WORD, and as long as the shell obeys
the Posix word expansion and quoting rules, it's simply an extension.
(0003979)
geoffclare (manager)
2018-04-20 14:09

 Re: Note: 0003978 The shell cannot have carte blanche to make any character it likes have a special meaning as an extension. Otherwise how are shell script writers supposed to know what characters they need to quote? Are you saying they need to quote everything they don't want treated as special? (Which would be contrary to what's obviously intended by the text that is the subject of this bug.)
(0003980)
chet_ramey (reporter)
2018-04-20 14:35

 Re: 3979: That is the situation we find ourselves in today. A script writer who expects portability should be, and usually is, aware of extensions in the various existing shells. Quoting is an easy remedy.

Look at the other parallel discussions: all shells have extensions, and different ones to boot. Script writers are already aware of them.

If you like, we can have a list of characters that script writers should be aware of, including `{', in the same spirit as the standard leaving the effects of common builtin extensions or characters following a $ that the standard doesn't cover as explicitly unspecified.
(0003981)
geoffclare (manager)
2018-04-20 14:56
edited on: 2018-04-23 08:22

 Re: Note: 0003980 valid extensions of shell syntax use the characters that the standard says are always special in ways that do not fit the POSIX syntax. For example the <<< redirection operator is a valid extension because < is an operator. Likewise process substitution using <(some_command) because < and ( are operators, and !(foo) pathname expansion because ( is an operator.

Or they extend a construct that is specified by POSIX in a way that POSIX doesn't specify a behaviour for, such as ${var/foo/bar} which is valid because / can't be a character in a variable name.

Brace expansion is the first time I have encountered an extension (of shell syntax) that does not appear to be allowed by POSIX.
(0003982)
chet_ramey (reporter)
2018-04-20 15:25

 There's the long-deprecated $[expr] (explicitly unspecified in 2.6), csh-style history expansion (not a special character), and possibly some single-character parameter expansions that other shells implement (also explicitly unspecified).
(0003983)
stephane (reporter)
2018-04-20 15:40
edited on: 2018-04-20 15:43

 Re: Note: 0003981

Other extensions that clash with POSIX that come to mind:

- extra keywords that can't be used as function names (POSIX reserves some of ksh's)
- special variables. POSIX lists a few like SECONDS, but not others like TMOUT, LD_PRELOAD and can't possibly list them all.
- zsh's %?foo that is not taken as a glob (won't expand to %xfoo (to allow kill %?job unquoted)
- csh-style history expansion (!, ^), still enabled by bash when running as sh, though that's only for interactive shells
- echo options (-e from Unix V8 1981, -E from bash 1992, combinations like -nn, -ne) supported by many shells, a problem because there's no "--" to end the options with echo (though zsh supports "-" as a (clashing) extension). Here I'd say POSIX should be amended to allow at least echo -[eEn]* (note the * instead of +) to be unspecified, at it's clearly a case where POSIX fails to describe the current reality (or even the reality from 25 years ago).
- &> redirection operator (foo &> bar per POSIX is "foo & > bar [nocommand]") (unlikely to be a problem in practice).
- echo **/* (zsh)
- .* expansion not including . and .. (pdksh/zsh) (though it's actually the behaviour most people would like)
- commands defined as alias in ksh that can't be used as function name.
(0003984)
geoffclare (manager)
2018-04-20 15:43
edited on: 2018-04-20 15:44

 Re: Note: 0003982 Ah yes, history expansion. I'd forgotten that one. It's also not a valid POSIX extension, but since it only affects interactive shells and is easy to turn off, it's not nearly as much of an issue.
(0003985)
geoffclare (manager)
2018-04-20 15:46

 Re: Note: 0003983 the context is extensions that cause scripts to need extra quoting, so I think most of those are not relevant.
(0003986)
chet_ramey (reporter)
2018-04-20 15:46

 Brace expansion is also easy to turn off, at least in bash (set +B), and it's easy to enable history expansion in non-interactive shells (set -H). It's a question of defaults.
(0003987)
geoffclare (manager)
2018-04-20 16:05

 Re: Note: 0003986 just to be clear, although I believe brace expansion is not currently allowed by POSIX, I am in favour of changing the standard to allow it. I am working on a new bug with the necessary changes.
(0004278)
Don Cragun (manager)
2019-03-07 16:26
edited on: 2019-03-07 16:28

 Change on Page 2346 Lines 74697-74699:
    
    and the following may need to be quoted under certain circumstances. That is, these characters may be special depending on conditions described elsewhere in this volume of POSIX.1-2008:

    
*   ?   [   #   ~   =   %


to:

    and the following might need to be quoted under certain circumstances. That is, these characters are sometimes special depending on conditions described elsewhere in this volume of POSIX.1-2008:

    
*   ?   [   ]   ^   -   !   #   ~   =   %

- Issue History
Date Modified Username Field Change
2018-04-13 14:33 geoffclare New Issue
2018-04-13 14:33 geoffclare Name => Geoff Clare
2018-04-13 14:33 geoffclare Organization => The Open Group
2018-04-13 14:33 geoffclare Section => 2.2 Quoting
2018-04-13 14:33 geoffclare Page Number => 2346
2018-04-13 14:33 geoffclare Line Number => 74697
2018-04-13 14:33 geoffclare Interp Status => ---
2018-04-13 14:44 geoffclare Note Added: 0003956
2018-04-19 05:54 shware_systems Note Added: 0003971
2018-04-19 05:55 shware_systems Note Edited: 0003971
2018-04-19 15:32 stephane Note Added: 0003973
2018-04-20 09:52 geoffclare Note Added: 0003975
2018-04-20 13:30 chet_ramey Note Added: 0003976
2018-04-20 13:39 stephane Note Added: 0003977
2018-04-20 13:52 chet_ramey Note Added: 0003978
2018-04-20 14:09 geoffclare Note Added: 0003979
2018-04-20 14:35 chet_ramey Note Added: 0003980
2018-04-20 14:56 geoffclare Note Added: 0003981
2018-04-20 15:06 stephane Note Edited: 0003977
2018-04-20 15:22 geoffclare Note Edited: 0003981
2018-04-20 15:25 chet_ramey Note Added: 0003982
2018-04-20 15:40 stephane Note Added: 0003983
2018-04-20 15:43 geoffclare Note Added: 0003984
2018-04-20 15:43 stephane Note Edited: 0003983
2018-04-20 15:44 geoffclare Note Edited: 0003984
2018-04-20 15:46 geoffclare Note Added: 0003985
2018-04-20 15:46 chet_ramey Note Added: 0003986
2018-04-20 16:05 geoffclare Note Added: 0003987
2018-04-23 08:19 geoffclare Note Edited: 0003981
2018-04-23 08:22 geoffclare Note Edited: 0003981
2019-03-07 16:17 geoffclare Relationship added related to 0001193
2019-03-07 16:26 Don Cragun Note Added: 0004278
2019-03-07 16:27 Don Cragun Final Accepted Text => See Note: 0004278.
2019-03-07 16:27 Don Cragun Status New => Resolved
2019-03-07 16:27 Don Cragun Resolution Open => Accepted As Marked
2019-03-07 16:28 Don Cragun Note Edited: 0004278
2019-03-07 16:29 Don Cragun Tag Attached: tc3-2008
2019-11-08 11:42 geoffclare Status Resolved => Applied
2024-06-11 09:08 agadmin Status Applied => Closed

Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker