View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0001389 | 1003.1(2016/18)/Issue7+TC2 | Shell and Utilities | public | 2020-08-13 19:52 | 2021-01-28 16:57 |
| Reporter | kre | Assigned To | |||
| Priority | normal | Severity | Objection | Type | Error |
| Status | Closed | Resolution | Duplicate | ||
| Name | Robert Elz | ||||
| Organization | |||||
| User Reference | |||||
| Section | 2.9.1.1 1.c (2.9.1.4 1.c in Issue8 Draft 1), 2.9.1.1 1.d, 2.9.1.1 1.e.i.a | ||||
| Page Number | 2367 (2335 in Issue8 draft 1) | ||||
| Line Number | 75564-73 75577-78 (74159-64 74168-71 in Issue8 draft 1) | ||||
| Interp Status | --- | ||||
| Final Accepted Text | |||||
| Summary | 0001389: Shell command search procedure is incorrect | ||||
| Description | POSIX currently specifies that normal built-in standard utilities be invoked only when the PATH search locates that utility. There are all kinds of problems with this, but the most important is that it does not document the standard behaviour (rather it documents what it seems some people would like the standard behaviour to be). I set PATH to a list of directories that does not include /bin (which, on my system, is where the standard utilities ls and test are to be found - for completeness here, printf is in /usr/bin which remained in $PATH, though the results below do not show that). (I am not showing the value of my PATH with /bin removed, as what's there isn't actually relevant for anything here). Then I ran the following script, which tests every (posixish) shell I have access to that I could remember I have... for SHELL in /bin/sh /bin/ksh bash 'bash -o posix' fbsh dash bosh yash \ pdksh mksh ksh93 zsh 'zsh --emulate sh' do $SHELL -c ' printf "\n\nTesting %s : \n" '"'$SHELL'"' ls /bin/test /bin/ls /bin/test IFS=: for p in $PATH do [ -x $p/test ] && echo found test in $p done test x = x || echo weird ' done >/tmp/results 2>&1 It needs full pathnames for the standard NetBSD shells /bin/sh and /bin/ksh as /bin isn't in PATH any more - /bin/ksh and pdksh are more or less the same thing, though probably have slightly different sets of bugs fixed. fbsh is the FreeBSD shell, the rest I think are well known. I don't have a copy of ksh88. In that the initial printf is just for documentation, the two ls lines that follow are to show first that "ls" is not found (as /bin is not in $PATH, and second that /bin/test exists - that is, that is where the test utility would normally be found). From those two lines I would expect (in every shell) to see some kind of error about "ls" not being found, and then "/bin/test" as the output from the 2nd /bin/ls command (and indeed, that is what happens, as shown below). IFS is set to break apart $PATH, and the for loop simply verifies that there is no other "test" command anywhere else that might be invoked. If there is another test, the printf would tell us where it was located, if there isn't, then there will be no output. Obviously all shells should do the same thing here, and they do, none finds a "test" anywhere else. Note that there is actually a possible issue here with the '[' command (which also lives in /bin) so a shell might complain that '[' is not found. None do. The final line is the important one, if a shell implemented the specification in 2.9.1.1 1.c and 2.9.1.1 1.e.i.a "correctly", there should be a "test not found" error (in some format or other, just like the "ls not found" error) here, as the built-in version of the standard utility test is only supposed to be located if test is found in $PATH (which we have just seen it will not be). The "|| echo weird" is just in case some non-standard test were to be invoked, and there "echo" (which also lives in /bin) could also result in "command not found" - this we don't know, as there is no attempt made to execute that "echo" command, the test command (in every shell I tested - I had hoped that in one or two might not have test built in) is a built-in command, and is executed (IMO correctly) here (and operates correctly, returning a status of 0, so the || part is never attempted). The results showing all of this appear bust below. To me this is clear evidence that the current specification is not specifying the standard behaviour, and should be fixed. Testing /bin/sh : ls: not found /bin/test Testing /bin/ksh : /bin/ksh: ls: not found /bin/test Testing bash : bash: ls: command not found /bin/test Testing bash -o posix : bash: ls: command not found /bin/test Testing fbsh : fbsh: ls: not found /bin/test Testing dash : dash: 1: ls: not found /bin/test Testing bosh : bosh: ls: not found /bin/test Testing yash : yash: no such command `ls' /bin/test Testing pdksh : pdksh: ls: not found /bin/test Testing mksh : mksh: ls: not found /bin/test Testing ksh93 : ksh93: ls: not found [No such file or directory] /bin/test Testing zsh : zsh:1: command not found: ls /bin/test Testing zsh --emulate sh : zsh:1: command not found: ls /bin/test | ||||
| Desired Action | First, if considering Issue8 draft 1, all references below to section 2.9.1.1 should be interpreted as references to section 2.9.1.4 In section 2.9.1.1 1.c (page 2367) delete the sentences: If the implementation has provided a standard utility Replace the entire text of section 2.9.1.1 1.d (which differs dramatically between Issue 7 and Issue 8 draft 1, so the old text is not quoted here), (page 2367) with: If the command name matches the name of a utility built In section 2.9.1.1 1.e replace the entire first part of the text of sub-section i (including the entire contents of sub-sub-sections 'a' and 'b') down to, but not including the paragraph that begins Once a utility has been searched for and found The text to be removed is not included here, as it differs between Issue 7 and Issue 8 draft 1. The replacement text should be: If the search is successful the shell shall execute the [This is using the section numbering for the reference as it appears in Issue 8 draft 1, where 2.9.1.6 is the section entitled: Non-built-in Utility Execution.] These changes will make the standard match what shells actually implement, rather than some pipe dream about what they should implement. | ||||
| Tags | No tags attached. | ||||
| duplicate of | 0000854 | Closed | 1003.1(2013)/Issue7+TC1 | requirement for additional built-in utilities to be searched for via $PATH was not and is not existing practice | |
| related to | 0001390 | Closed | 1003.1(2016/18)/Issue7+TC2 | No method to invoke a non-standard built-in utility | |
| related to | 0001391 | Closed | 1003.1(2016/18)/Issue7+TC2 | Unclear language in specification of utilities implemented as functions |
|
|
Although it goes into more detail, this is effectively a repeat of bug 0000854 and should be closed as a duplicate. |
|
|
Just in case this is not known, ksh93 implements PATH search for builtin commands and this was a requirement for ksh93 integration into OpenSolaris in 2008 after a long and heaty discussion. The reason for that requirement was to retain POSIX compliance even though basic standard utilities are built into the standard shell. If a shell does not implement PATH search for additional builtins, we would need to forbid shells to add more builtins than the minimal set. ksh93 added many additional builtins like cat, chown, cmp, ... and POSIX grants that a user that if he has private version of system utilities, the shell needs to use them if they are first in PATH. If a shell adds more builtins than currently documented, this would no longer work as historically builtin are always found first. As a consequence, we created a list of tainted utilities (see 0000854) that already may be builtins. If you like to add more builtins from the list of standard utilities, you either need to implement PATH search for these builtins, or you are forbidden to implement these utilities as builtins. In ksh93, builtin commands are managed with the builtin named "builtin". Call builtin for a list....e.g. builtin : . [ /bin/cat /bin/chown /bin/cksum /bin/cmp /bin/comm /bin/cut /bin/dirname /bin/fold /bin/getconf /bin/head /bin/join /bin/logname /bin/mkdir /bin/mkfifo /bin/mktemp /bin/paste /bin/pathchk /bin/rev /bin/rmdir /bin/sum /bin/sync /bin/tail /bin/tee /bin/tty /bin/uniq /bin/wc /bin/xgrep /sbin/sync /usr/ast/bin/basename /usr/ast/bin/cat /usr/ast/bin/chgrp /usr/ast/bin/chmod /usr/ast/bin/chown /usr/ast/bin/cksum /usr/ast/bin/cmp /usr/ast/bin/comm /usr/ast/bin/cp /usr/ast/bin/cut /usr/ast/bin/date /usr/ast/bin/dirname /usr/ast/bin/egrep /usr/ast/bin/expr /usr/ast/bin/fds /usr/ast/bin/fgrep /usr/ast/bin/fmt /usr/ast/bin/fold /usr/ast/bin/grep /usr/ast/bin/head /usr/ast/bin/id /usr/ast/bin/join /usr/ast/bin/ln /usr/ast/bin/logname /usr/ast/bin/md5sum /usr/ast/bin/mkdir /usr/ast/bin/mkfifo /usr/ast/bin/mktemp /usr/ast/bin/mv /usr/ast/bin/paste /usr/ast/bin/pathchk /usr/ast/bin/readlink /usr/ast/bin/rev /usr/ast/bin/rm /usr/ast/bin/rmdir /usr/ast/bin/stty /usr/ast/bin/sum /usr/ast/bin/sync /usr/ast/bin/tail /usr/ast/bin/tee /usr/ast/bin/tty /usr/ast/bin/uname /usr/ast/bin/uniq /usr/ast/bin/wc /usr/ast/bin/xgrep /usr/sbin/sync /usr/xpg4/bin/basename /usr/xpg4/bin/egrep /usr/xpg4/bin/fgrep /usr/xpg4/bin/grep /usr/xpg4/bin/id /usr/xpg4/bin/rm /usr/xpg4/bin/tail /usr/xpg6/bin/expr alarm alias bg break builtin cd command .... Note that the builtins bound to /usr/ast/bin are not fully POSIX compliant and there fore are not bound to a path that appears in the default PATH. If you do not care about POSIX compliance or if you do not like to add more builtins to your shell, you may do what ever you like. If you however like to add more builtins, you need to follow long existing rules about PATH search. |
|
|
ksh93 does not follow the POSIX rules either, and never has done: (1) ksh93 has several builtins (including builtin aliases, something POSIX doesn't allow either), not bound to any path search, that are "not allowed" by this rule, i.e.: which are neither a standard utility, nor in the list of names reserved as "unspecified" in XCU 2.9.1.4(b). They are: alarm, compound, enum, float, functions, integer, nameref, r, redirect. (2) Several POSIX standard regular builtins are *never* subject to a path search in ksh93, although POSIX specifies that they must be. These are: [, echo, printf, sleep, test, type, ulimit. What's more, path-bound builtins in ksh93 behave exactly like external commands. The only difference is their path doesn't actually exist in the file system. But that is a mere implementation detail. POSIX has no business specifying or regulating implementation details. So, if this path search rule exists because of ksh93, then it is based on a fundamental misunderstanding of what ksh93 does. In the real world, there are zero shells that actually manage to follow this rule. Not even the deliberately pedantic 'yash -o posix' fully manages, though it tries hard. Bash, Busybox ash, dash, FreeBSD sh, NetBSD sh, mksh, pdksh (OpenBSD sh), and zsh all ignore this rule completely. So does yash in its sane default mode. This entire path search rule for builtins should quite simply be removed. I suppose it won't do much damage if POSIX decides to keep it; I'm confident the overwhelming majority of shells will continue to ignore it completely. The real damage is to POSIX's reputation as a standard to be taken seriously. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2020-08-13 19:52 | kre | New Issue | |
| 2020-08-13 19:52 | kre | Name | => Your Name Here |
| 2020-08-13 19:52 | kre | Section | => (section number or name, can be interface name) |
| 2020-08-13 19:52 | kre | Page Number | => (page or range of pages) |
| 2020-08-13 19:52 | kre | Line Number | => 75564-73 75577-78 (74159-64 74168-71 in Issue8 draft 1) |
| 2020-08-13 21:17 | Don Cragun | Name | Your Name Here => Robert Elz |
| 2020-08-13 21:17 | Don Cragun | Interp Status | => --- |
| 2020-08-13 21:17 | Don Cragun | Relationship added | related to 0001390 |
| 2020-08-13 21:18 | Don Cragun | Relationship added | related to 0001391 |
| 2020-08-13 21:24 | Don Cragun | Severity | Editorial => Objection |
| 2020-08-13 21:24 | Don Cragun | Type | Clarification Requested => Error |
| 2020-08-13 21:30 | Don Cragun | Section | (section number or name, can be interface name) => 2.9.1.1 1.c (2.9.1.4 1.c in Issue8 Draft 1), 2.9.1.1 1.d, 2.9.1.1 1.e.i.a |
| 2020-08-13 21:30 | Don Cragun | Page Number | (page or range of pages) => 2367 (2335 in Issue8 draft 1) |
| 2020-08-14 08:47 | geoffclare | Note Added: 0004920 | |
| 2020-08-14 08:47 | geoffclare | Relationship added | duplicate of 0000854 |
| 2020-08-14 10:39 | joerg | Note Added: 0004924 | |
| 2020-08-14 10:40 | joerg | Note Edited: 0004924 | |
| 2020-08-14 10:40 | joerg | Note Edited: 0004924 | |
| 2020-08-14 10:41 | joerg | Note Edited: 0004924 | |
| 2020-08-19 10:36 | joerg | Note Edited: 0004924 | |
| 2020-08-20 15:31 | McDutchie | Note Added: 0004934 | |
| 2021-01-28 16:57 | Don Cragun | Status | New => Closed |
| 2021-01-28 16:57 | Don Cragun | Resolution | Open => Duplicate |
