Austin Group Defect Tracker

Aardvark Mark III


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0001150 [1003.1(2016)/Issue7+TC2] Shell and Utilities Editorial Omission 2017-06-16 01:33 2017-06-17 08:31
Reporter kre View Status public  
Assigned To
Priority normal Resolution Open  
Status New  
Name Robert Elz
Organization
User Reference
Section 2.6.3
Page Number 2357 - 2358
Line Number 75182 - 75224
Interp Status ---
Final Accepted Text
Summary 0001150: exit status of command substitution not properly specified
Description The description of Command Substitution (XCU 2.6.3) does not say
what is intended to be done (if anything) with the exit status
of the commands that are performed while evaluating the command
substitution..

There is what is said in 2.9.1 about
        a=$(command)
which uses the status of command as the status of the assignment, when
there is no command word, but that is the only case that is explicit.

Normally, it is not going to matter, as when there is a command
(rather than simply ASSIGNMENT_WORDs) the exit status of that command
is explicit in the specification, and does not take any notice of
the exit status of any command substitutions that needed to be
evaluated while preparing to run the command.

For example
        : $(exit 1)
has an exit status of 0, because the specification for ':' requires it.
Hence any status from the $() is irrelevant and need not even be considered.

However there are two issues to consider which are not explicit.

First, does -e apply to the exit status of a command substitution?
I have no doubt that it applies inside the command substitution, so

        set -e; printf %s $( exit 1; printf foo )

should output nothing, the command substitution exits when exit returns
status 1, because of the -e (errexit) option being set.

But what about

        set -e; printf %s $(exit 1)foo

Everything I have tested prints "foo" here, which I believe is the
intended result, but nothing I can find actually says so.

Second
        return $( exit 1 )

After word expansion, we have just

        return

as $(exit 1) produces nothing, and is not quoted, so is simply removed.

The specification for "return" says ...
        The value of the special parameter '?' shall be set to n, an unsigned
        decimal integer, or to the exit status of the last command executed
        if n is not specified.

Here n is not specified, but is that "exit 1" the last command executed?

The "exit" executes in a sub-shell environment, so cannot be itself, but the
exit status of the command substitution could be.

Here shells differ, given

        $SHELL -c 'f() { return $( exit 1); } ; f; echo $?'

some shells report 1, and some (well, most by number, if not popularity) 0.

My gut feeling is that 0 is correct, but again nothing actually says so.

Desired Action Add a new paragraph, after line 75224 in XCU 2.6.3 (immediately before
section 2.6.4) saying:

   Other than as specified in section 2.9.1 the final exit status of the
   commands executed to perform a command substitution shall not be used for
   any purpose. That is a shell shall not exit if the errexit option (-e) is
   set, and a command substitution fails, though it shall if the result of the
   command substitution becomes the result of the command as specified in
   section 2.9.1. Similarly, in any other situation where the exit status of
   the last or previous command is required, the results of any intervening
   command substitutions shall be ignored. Nothing in this paragraph shall
   affect the commands being executed, or the effect of the errexit option
   upon, the commands executed to perform the command substitution.
Tags No tags attached.
Attached Files

- Relationships

-  Notes
(0003766)
kre (reporter)
2017-06-16 01:46

Oops, I should have tested my example, when I wrote ...

         set -e; printf %s $( exit 1; printf foo )

I (of course) really meant ...

         set -e; printf %s $( (exit 1); printf foo )

If the exit in there was explicit, rather than just an exit
status, then the -e option setting would be irrelevant.

Alternatively ...

         set -e; printf %s $( false; printf foo )

would have worked as the example.
(0003767)
stephane (reporter)
2017-06-16 06:10

See also

   exit $(exit 1)
   exit < "file$(exit 1)"

in zsh and bash (I suppose the two "popular" shell implementations you're refering to).

Command-less redirections are also cases to consider:

   < file$(exit 4)
   < file$(exit 5) var=$(exit 6)

Which give different results depending on the shell.

There's also:

   echo < file$(exit 5) "$? $(exit 3)$? $(exit 4)$?"

And:

   var=$(exit 5) eval 'echo "$?"'

And:

   eval 'echo "$?"' < file$(exit 5)

And:

   { echo "$?"; } < file$(exit 5)

And:

   exec 3< file$(exit 5)
(0003768)
joerg (reporter)
2017-06-16 09:38

The standard is obvious here:

If return or exit are called without parameter, the return/exit status
is the status of the last command executed.

Given that with "return $(exit 1)", the last command executed
was "exit 1", it seems to be obvious that the expected return value is 1.

This is important as we like to have an orthogonal behavior and:

sh -c 'FOO=$(exit 99); echo $?'

is expected to print 99 while

sh -c 'FOO=$(exit 99) :; echo $?'

is expected to print 0
(0003769)
joerg (reporter)
2017-06-16 09:54

Re: Note: 0003767

It would be nice if you did post some expected exit status values....

Following the behavior of existing shells may be a bad idea, as these
shells may suffer from bugs that we don't like to standardize.
(0003770)
kre (reporter)
2017-06-16 10:06

Re note 3768 === It is not nearly as obvious as you think.

In "return $(exit 1)" the "exit 1" is executed in a subshell environment
(that is a requirement), and:

   Changes made to the subshell environment shall not affect the
   shell environment.

That's from XCU 2.12. The exit status change in the subshell environment
cannot directly affect the shell environment. The only way to make that
happen would be to export the exit status of the command substitution, and
whether or when that can happen was the subject of the bug report - it is
just not stated, except for the one case in 2.9.1

Most shells (incl ksh93) do the "obvious" thing and return the status of
the last command that occurred, before the return $(exit 1), leaving the
exit status of the command substitution unused, as it is in just about every
other situation. Making an exception for this case is exactly the opposite
to achieving orthogonal behavior.

Your exit 99 examples I agree with, but also remember that

sh -c ': $(exit 99); echo $?' is expected to print 0.

And re note 3769, I believe that in all of STephane's examples, the
$(exit N) should simply be ignored (unless I missed one somewhere).
They produce no output, and were not used in a context where the exit
status of the command substitution should be extracted, so they just turn
into meaningless syntactic noise.
(0003771)
joerg (reporter)
2017-06-16 10:16
edited on: 2017-06-16 10:21

Re: Note: 0003770

The main shell waits for the sub-shell and the exit status remembered
by the main shell is the exit status returned via the wait() or waitid()
call from the sub-shell.

This behavior is responsible for the documented and expected behavior
that resuts in

sh -c 'FOO=$(exit 99); echo $?'

is expected to print 99 while

sh -c 'FOO=$(exit 99) :; echo $?'

is expected to print 0

If you like to see a different behavior than the one I explained
in Note: 0003768, there would be a need to introduce extra code
to the shell in order to make it behave non-orthogonal.

BTW I wrote Note: 0003768 because I know that:
 
sh -c ': $(exit 99); echo $?'

is expected to print 0.

As I prefer orthogonal behavior, I believe that requiring:

$SHELL -c 'f() { return $( exit 1); } ; f; echo $?'

to print 0 would be a mistake

(0003772)
kre (reporter)
2017-06-16 11:07

Re note 3767, First, thanks for the additional cases to consider.

In this example

    < file$(exit 4)

(assuming "file" exists and is readable) I think the wording of 2.9.1
is clear enough, and the exit status from this is 4. And I think just
about all shells (that pretend to be posix, or close) implement that.

For this one

   < file$(exit 5) var=$(exit 6)

we have a similar situation (no command word) so the exit status of the
last command substitution applies. But here we are explicitly allowed to
perform steps 3 and 4 (redirections and var assignments, resp) in whichever
order we prefer, so the shell can set the exit status to 5 or 6 for this one.

In all the other cases, where there is a command word, I would expect the
command substitutions, which produce no output, to have no effect whatever.
Their exit status should simply be ignored.
(0003773)
stephane (reporter)
2017-06-16 15:38

Re: Note: 0003772

I wouldn't say it's that clear cut. Not many people would object that

a=$(exit 4) b=$?

may (if not should) assign 4 to $b (most do, dash and pdksh don't). With that in mind, it's hard to justify that

a=$(exit 4)$?

may not. Even if I agree it's not desirable. It's very unlikely anyone may want to do something like the above and expect $? to be the exit status of the previous cmdsubst. Things like this would be more likely:

   msg="$(translate 'last exit status'): $?"

where you don't want $? to be the exit status of that $(translate).

The question is if we mandate one way or the other, is it going to break existing scripts (which were probably not very portable anyway) which make the other assumption? Most of those are corner cases. We could leave it unspecified but raise awareness and give ways to avoid the problem. Like the above can be written portably:

   a=$(exit 4); b=$?
   # or b=$?; a=$(exit 4) # if you want $? to be the exit status of the last command

   ret=$?
   msg="$(translate 'last exit status'): $ret"
(0003774)
joerg (reporter)
2017-06-16 15:52
edited on: 2017-06-16 15:59

Re: Note: 0003773

it seems that many shells are broken here...

Does anybody disagree that:

   $SHELL -c 'f() { (exit 3); return $( exit 1); } ; f; echo $?'

should print "3"?

The reason for the diverging behavior is the places where
the shells implement checkpoints to save the current value
of "exitcode" for the next expansion of $?.

The Bourne Shell and it's childs have such a checkpoint at the
end of the interpreter function and this function is called
from the command substitution.....

ksh88 removed one such checkpoint after the wait() for the
$(cmd), but left the checkpoint in the interpreter function
"execute()"

I could make it print "3" in bosh after I removed the checkpoint
past the wait() from $(cmd) and after I restored the old saved
value of the exit code after calling "execute()" for $(cmd).

(0003775)
stephane (reporter)
2017-06-16 16:05

Re: Note: 0003774

Here, that's one case where it doesn't really matter. I agree 3 is the most sensible outcome. Like in

$SHELL -c 'f() { (exit 3); a=$(exit 1) return; } ; f; echo $?'

(ksh93 returns 0 which clearly is a bug)

But that doesn't really matter because nobody is going to pass a cmdsubst that expands to nothing to exit or return or call return with a preceding assignment.
(0003776)
kre (reporter)
2017-06-16 16:19

Re note 3773:

  | Not many people would object that
  | a=$(exit 4) b=$?
  | may (if not should) assign 4 to $b

I would not object to may, but would to should.

In
        a=a; a=x b=$a
it is explicitly unspecified whether b=a or b=x at
the conclusion. It would be astounding if there was
to be a stricter requirement for $?

ash based shells (not just dash) only set $? after
this whole simple command has completed, then set
it to 4 (the result of the last cmdsub in a command
with no command word, as required.)

Exactly when $? is required to be updated is perhaps
something else that needs better specification, this
relates to the "checkpoints" that Joerg mentions in
note 3774.

I doubt much, if anything, will be broken, whatever
ends up being specified here, until the issue appeared
during an academic discussion on a related topic a day
or two ago, I had no idea that shells behaved differently.
If any scripts were being affected by that, I think we
would have heard about it before now.
(0003777)
shware_systems (reporter)
2017-06-16 16:33

Re:
"There is what is said in 2.9.1 about
        a=$(command)
which uses the status of command as the status of the assignment, when
there is no command word, but that is the only case that is explicit."

This can be construed that assignments and redirections are not included, and command as used there is only the words that are what should be the command name and any arguments. Example:
a="" $(does-not-write-a-command-to-stdout-but-exits-5) argish1
would have the 5 as new value for $?.
If the command name does not name a function, built-in, or file on the path, the result from the command substitution would override the 127 return of "command not found". I'm not saying this is the intent, just that it can be read that way and lead to some of the differences discussed.
(0003778)
kre (reporter)
2017-06-16 16:35

And referring to 3773 again, I would say it is that clear
cut (what was in 3772) - you changed the goalposts.

That is, in
   a=$(exit 4) b=$?
what gets assigned to b may be debatable, what is not however
is what gets printed from
   a=$(exit 4) b=$?; echo $?
which must be 4 (at least if you believe section 2.9.1's requirement
on the exit status of a simple command with no command word.)

That was the point of note 3772. This is the one clear place where the
exit status of a command substitution is used. It might be the only one.
(0003779)
kre (reporter)
2017-06-16 16:40
edited on: 2017-06-16 16:43

Re 3777, no it can't be read that way - if there is a command word at all
then 2.9.1 does not apply, and the exit status is that that comes from
executing the command word - even if that was originally planned on being
an arg to a command which failed to be produced, and so ends up producing
a 127 exit code. If there was no "argish1" in your example, then the
exit code would be 5.

And when I say 2.9.1 here I am of course referring to just the one sentence
in that long section that relates to using the exit status of the last
command substitution performed when there is no command word. All the
rest of it still applies, obviously (I hope.)

(0003780)
joerg (reporter)
2017-06-16 16:45
edited on: 2017-06-16 16:45

What gets assigned with

a=$(exit 4) b=$?

may depend on whether the macro expansion is done for all assignments
first or step by step and whether/where is a checkpoint to save the
exit code for $? expansion.

You are right for a=$(exit 4) b=$?; echo $?

(0003781)
shware_systems (reporter)
2017-06-17 07:51

Re: 3779
I agree the intent is along those lines, but isn't explicit enough to rule out the case presented. It's little different from 'a=echo; $a "Output TexT";', which I'd expect to write to stdout. The substitution results get evaluated as the command name that affects $?, whether arguments present or not. An implementation might limit 127 to case where './echox "Text";' not found on the path, for the simpler usage not involving command substitution, to resolve the conflicting requirements.
(0003782)
kre (reporter)
2017-06-17 08:31

Re 3781...

It is actually quite explicit, re-read 2.9.1 (steps 1-4 in the prelude, and
what immediately follows) and you will see it (should see it.)

The shell takes the (unexpanded) command line, removes any words that are
var-assigns (start with unquoted name= and precede any non-redirect word
which does not start that way) and any redirect words (contain an unquoted
redirect operator), then applies the expansion rules to all the words that
are left. The result of this can have some words vanish (expand to nothing)
and new words appear (file name expansion, field splitting).

Once that is complete we look at what resulted, if they are any words it
is some kind of command (function, built in, whatever), and is processed
that way (and its exit status comes from the command - which can include
the 126 and 127 exit codes from the shell attempting to execute the command.)

If there were are no words remaining (either never were any, or they
all vanished) then the rules for processing commandless redirections and
variable expansions apply. In this case (and as stated at least, only in
this case) does the exit status of the last executed command substitution
(when the redirects and var-assigns are expanded, or possibly from the earlier
word expansions) get used to set $? (the exit status of the empty command.)

- Issue History
Date Modified Username Field Change
2017-06-16 01:33 kre New Issue
2017-06-16 01:33 kre Name => Robert Elz
2017-06-16 01:33 kre Section => 2.6.3
2017-06-16 01:33 kre Page Number => 2357 - 2358
2017-06-16 01:33 kre Line Number => 75182 - 75224
2017-06-16 01:46 kre Note Added: 0003766
2017-06-16 06:10 stephane Note Added: 0003767
2017-06-16 09:38 joerg Note Added: 0003768
2017-06-16 09:54 joerg Note Added: 0003769
2017-06-16 10:06 kre Note Added: 0003770
2017-06-16 10:16 joerg Note Added: 0003771
2017-06-16 10:21 joerg Note Edited: 0003771
2017-06-16 11:07 kre Note Added: 0003772
2017-06-16 15:38 stephane Note Added: 0003773
2017-06-16 15:52 joerg Note Added: 0003774
2017-06-16 15:59 joerg Note Edited: 0003774
2017-06-16 16:05 stephane Note Added: 0003775
2017-06-16 16:19 kre Note Added: 0003776
2017-06-16 16:33 shware_systems Note Added: 0003777
2017-06-16 16:35 kre Note Added: 0003778
2017-06-16 16:40 kre Note Added: 0003779
2017-06-16 16:43 kre Note Edited: 0003779
2017-06-16 16:45 joerg Note Added: 0003780
2017-06-16 16:45 joerg Note Edited: 0003780
2017-06-17 07:51 shware_systems Note Added: 0003781
2017-06-17 08:31 kre Note Added: 0003782


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker