Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0001039 [1003.1(2013)/Issue7+TC1] Shell and Utilities Editorial Clarification Requested 2016-03-26 07:30 2019-10-23 10:17
Reporter kre View Status public  
Assigned To
Priority normal Resolution Accepted As Marked  
Status Applied  
Name Robert Elz
Organization
User Reference
Section 2.5.2
Page Number 2324
Line Number 73759-73761
Interp Status Approved
Final Accepted Text Note: 0003617
Summary 0001039: Which option flags should appear in $- ?
Description It is clear that the option flags documented with the set builtin are
(when set) to appear in $- - but the description of the '-' parameter
section (2.5.2) says ...

     Expands to the current option flags (...) as specified on invocation,
     by the set special built-in command, or implicitly by the shell.

The second part of that (second life of it the way I formatted the quote)
is fine, but the "as specified on invocation" makes it unclear whether the
3 options supported by sh(1) ...
     http://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html [^]
that is, -c, -i, and -s, are to be included in $- or not.

While -i can be useful (it can be used to allow switching a shell between
interactive mode and non-interactive mode dynamically) -c and -s are simply
useless in $- (changing their values, if permitted, cannot usefully do
anything at all.)

Further, it is useful to allow idioms like "set +$-" (to turn off all options)
that tend to fail if $- contains options that set does not allow to be modified.

Desired Action Change the wording of the definition of the - (hyphen) special parameter in
section 2.5.2 from

      Expands to the current option flags (the single-letter option names
      concatenated into a string) as specified on invocation, by the set
      special built-in command, or implicitly by the shell.

to

      Expands to the currently set option flags (the single-letter option
      names of options that are set, concatenated into a string) of options
      defined for the set special built-in command, which have been set by
      use of the set command, or as specified on invocation, or implicitly
      by the shell.

      Implementations may add options to those specified for the set special
      built-in command, but only options that may be set or reset by that
      command shall ever appear in "$-".

Or something to similar effect. Note, "may be set or reset" doesn't
necessarily mean "do something useful", if an implementation allows "set -c"
then 'c' can appear in $- even though doing "set -c" is most probably a no-op.
On the other hand, if "set -c" generates an error, then 'c' should not appear
in $- even if the shell was invoked as "sh -c string".
Tags tc3-2008
Attached Files

- Relationships

-  Notes
(0003617)
geoffclare (manager)
2017-03-16 16:22

Interpretation response
------------------------
The standard states that $- includes the options set on the sh command line, and conforming implementations must conform to this. However, concerns have been raised about this which are being referred to the sponsor.

Rationale:
-------------
Some existing shells do not include c or s in $-.

Notes to the Editor (not part of this interpretation):
-------------------------------------------------------
On 2016 edition page 2350 line 74880 section 2.5.2, after:
(Hyphen.) Expands to the current option flags (the single-letter option names concatenated into a string) as specified on invocation, by the set special built-in command, or implicitly by the shell.
add:
It is unspecified whether the -c and -s options are included in the expansion of <tt>$-</tt>. The -i option shall be included in <tt>$-</tt> if the shell is interactive, regardless of whether it was specified on invocation.
(0003618)
kre (reporter)
2017-03-16 17:16

I agree to what is being proposed for the substantive part of the standard,
but it appears (from the etherpad, and the resolution) that the point of
this issue was missed.

That is, once upon a time, it used to be possible to do

      set -$-

and expect it would work (while being a no-op), or

      set +$-

to clear all flags set (whether this clears the i or m flags is not
important for current purposes - what matters is whether it generates a
syntax error from the shell).

The problem is that shells that include 'c' in particular in $- (or some
of them) later object to "set -c" (which is in itself, not surprising).

That makes
     set -$-
a syntax error if the shell was started as sh -c ...

Would it be possible to add some test to the rationale, to suggest to shell
implementors that anything that is in $- should be acceptable input to "set" ?

Whether implementors choose to leave 'c' (in particular) out of $-, and
then object to set -c, or whether they include 'c' in $- (when appropriate)
and then simply ignore "set -c" (make it a no-op, same with set +c of course)
doesn't matter, just so that set +$- and set -$- never produce syntax errors.

kre
(0003619)
joerg (reporter)
2017-03-16 17:31
edited on: 2017-03-16 17:38

What you like to see here was never possible with ksh88 - the role model for POSIX

It does not even work with the historic Bourne Shell, if you called "sh -i".

You may believe what your memory tells you because the historic Bourne
Shell did not automatically include -i for interactive shells.

(0003620)
kre (reporter)
2017-03-16 18:07

You are almost certainly correct about the original Bourne shell and -i,
but it was hardly relevant. In normal use, the issue only arose in scripts
(there were no functions), and your typical script is not run with sh -i,
nor does it (attempt to) "set -i", so even if there had been a problem,
I would never have noticed it.

The effect of the way that it is now, is that with many shells, scripts that
I have had for 30 years or more, no longer work (with some shells - fortunately,
with the shell I maintain, and use almost exclusively for scripts, it all works,
which you might think is a happy accident...)
(0003622)
geoffclare (manager)
2017-03-17 09:09

Executing set +$- is not something we should encourage applications to do. There can be option letters relating to extensions in $-, so by executing set +$- (if it succeeds) the application has made use of an extension and can no longer rely on the shell behaving as described in the standard.

Applications should use set +... only to turn off specific options that they want to turn off. They cannot expect "turn off all options" to achieve something useful or sensible in all implementations.
(0003624)
kre (reporter)
2017-03-17 10:46

Re:
    so by executing set +$- (if it succeeds) the application has made
    use of an extension

I would certainly hope (and the standard should encourage) that if extensions
are added they are enabled by turning *on* flags (if that is the mechanism
selected) rather than by turning flags off.

The default state for executing a script called as "sh script" really should
be for all flags to be clear, and in that state the shell should be operating
in its normal operating mode (which might or might not be posix compliant,
but it will usually be close.)

So, turning flags off really should be safe, and any shell that requires a
flag to be set (as distinct from perhaps a -o option) to operate as a standard
shell should be regarded as broken.

All that said, my main aim is not "set +$-" as useful as that might be, but
"set -$-" because that is what many scripts I own do. That is, a no-op,
just set the flags that are already set.

It is an idiom that traces back to (probably) bugs (or design issues) in the
earliest Bourne shells, where "set a b c" would set $1 $2 $3 as expected, but
would also clear all the flags, in some versions "set - a b c" would work, but
not in others, but in all (at the time) "set -$- a b c" would leave the flags
in their current state, and also set the arg list as requested (provided there
was at least one arg, hence the usual idiom is "set -$- X <whatever>; shift")

So my scripts (many of which were written back in the early 80's) all use
that idiom, and personally, I think it is very important to keep working.
Until shells started adding 'c' to $- and then refusing set -c this was the
one constant about the set command that was consistently implemented,
everywhere.

kre
(0003625)
joerg (reporter)
2017-03-17 10:54

There are several shell implementations that use something like

  set -o posix

to make it POSIX compliant and this is done automatically under certain
conditions. If such a shell implements a one-letter equivalent for
the "posix" option, set +$- would turn POSIX compliance off.
(0003630)
kre (reporter)
2017-03-17 12:07

Re:
     If such a shell implements a one-letter equivalent for
     the "posix" option, set +$- would turn POSIX compliance off.

If the shell implementer was half way rational, the one letter flag
equivalent of such an option would enable "extended mode" and turning
it off would be equivalent to "set -o posix". So, if P is the flag
for posix mode, P in $- would mean not posix, and set +P would be the
same as "set -o posix".
(0003634)
joerg (reporter)
2017-03-17 12:51

In the recent Bourne Shell, P is the mode to switch on fine grained
privileges behavior for the Solaris kernel mode privs. If you turn
this off, you may no longer be able to e.g. run privileged backup
programs if you do what you propose inside the backup script.
(0003636)
kre (reporter)
2017-03-17 13:05

Re: 3634 - of course, disabling options disables the functionality.
For a more mundane example, which applies everywhere, if a script
does set +x (either explicitly, or because of set +$-) then a user
attempting to debug the script with "sh -x script" is going to be
defeated (or at least annoyed.)

Actions have consequences, and everything that is done has the potential
to cause some side effect elsewhere.

But once again, I do not need a tutorial on why I should not be doing set +$-
in my scripts, what I would like, is for some kind of guarantee that
"set -$-" will remain a no-op.
(0004002)
McDutchie (reporter)
2018-04-26 11:44

kre, I don't think you're going to get a guarantee that the 'set -$-' workaround is a no-op, because in fact it fails on most current and historical shells.

Historical:

$ sh -c 'set -$-' # Bourne shell, Xenix (1988)
sh: -s: bad option(s)

$ sh -c 'set -$-' # Bourne shell, Solaris 10.3 (2010)
sh: -s: bad option(s)

$ ksh -c 'set -$-' # ksh88, Solaris 10.3 (2010)
ksh: -shc: bad option(s)

Presumably it works on the historical ash (Almquist shell) as it does on all current derivatives, but I don't have any historical ash I can test.

Current:

$ ksh -c 'set -$-' # ksh93
ksh: set: -c: unknown option
Usage: set [-sabefhkmnprtuvxBCGH] [-A name] [-o[option]] [arg ...]

$ ksh -c 'set -$-' # NetBSD ksh (pdksh)
ksh: set: -c: unknown option
ksh: set:

$ ksh -c 'set -$-' # OpenBSD sh and ksh (pdksh)
ksh: set: -c: unknown option

$ mksh -c 'set -$-'
mksh: set: -c: unknown option

$ bash -c 'set -$-'
bash: line 0: set: -c: invalid option
set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]

$ yash -c 'set -$-'
set: the cmdline option cannot be changed once the shell has been initialized

It only works on:

- NetBSD sh (of course)

- FreeBSD sh

- dash

- Busybox ash

- zsh

...so basically on Almquist derivatives plus zsh. That minority is nowhere close to establishing de facto standard behaviour.
(0004003)
joerg (reporter)
2018-04-26 12:12
edited on: 2018-04-26 12:59

Interesting... this new requirement is in conflict with:

-
    (Hyphen.) Expands to the current option flags (the single-letter option names concatenated into a string) as specified on invocation, by the set special built-in command, or implicitly by the shell.

and thus should be removed again since it would make existing implementations non-conforming.

The purpose of the standard is to document existing behavior, not to invent new behavior that is in conflict with existing behavior.

BTW: the documented method to save the current settings in a POSIX shell is to call "set +o", keep the output and run that result if needed.

(0004004)
kre (reporter)
2018-04-26 12:47

Re note 4002 ... the resolution of this (note 3617) is fine.

I have more or less given up on -$- ... it was useful back in the
original Bourne shell days, when there were all kinds of bugs in
the "set" command (way before -- existed - and long before ash was
even a dream, or the need for it was anywhere on the horizon - very
early 80's or late 70's). It certainly did work on old Bourne
shells (and that will be why it worked in ash and its descendants,
I would assume.)

The comments that followed note 3617 related to comments I saw on the
etherpad from when this bug was discussed.
(0004005)
kre (reporter)
2018-04-26 12:55

Re note 4003.

Joerg, there is no conflict. -i is an option to sh (see the utilities
page for sh) note "or implicitly by the shell" in the text you quoted.

If the shell implictly sets the 'i' flag (because it is started in a
way that meets the requirements of being interactive) so should always
have been included in $-

It was just that most shells concentrated only on the flags defined to
be able to be used with the "set" builtin, and not necessarily on those
which are only relevant on the command line. It was to clarify that that
the original bug report was submitted.

If your sh doesn't set 'i' - fix it. It was always broken.
(0004006)
joerg (reporter)
2018-04-26 13:06
edited on: 2018-04-26 14:34

I added "i" to $- for an interactive bosh in June 2012 because it seemed to be useful.

I however do't understand why "set -c" should be allowed or why $- should not include "r" for sh -r.

Note: change the last part of the last sentence since the Bourne Shell does not include "c" in $- for sh -c.

(0004007)
McDutchie (reporter)
2018-04-26 13:58

kre wrote:
> It certainly did work on old Bourne
> shells (and that will be why it worked in ash and its descendants,
> I would assume.)

Did you miss that I just tested it on a 1988 Bourne shell (Xenix)? It doesn't work. To me this looks like an old ash-specific workaround for an ash-specific bug.
(0004008)
kre (reporter)
2018-04-26 13:59

Re 4006:

  | I however do't understand why "set -c" should be allowed

It isn't. What makes you think that changed?

  | or why $- should not include "c" for sh -c

It can (it will be unspecified, as many shells do not). This
one I would have preferred not to see (it is the one that
broke -$-) but shells do it. I don't see any point (why
would a script ever care whether it was on the command line
or in a file) in it, but given that we're not allowing in $-
only the options that can be altered (or at least, preserved)
with "set" allowing it does no harm either.
(0004009)
kre (reporter)
2018-04-26 14:20

Re note: 4007.

No, I did not miss the reference to the 1988 vintage sh test.

That's much too late, I cannot even imagine what might have been
done to that shell (that was probably a SysVR3 shell, which had
lots of changes).

The one I am referring to was the original 7th edition (and 32V)
shell from Bell Labs.

In that, there was no --, to set the args, "set a b c" would fail if
"a" happened to start with a '-' (it would be interpreted as option
flags). The workaround was "set - a b c" but that would reset all
the flags (there was no "set +..." in that shell, the when a '-' option
was given to "set" it cleared the flags, and then set those given.)
Hence, "set -$- a b c" - set the flags back to what they were, and
then set the positional parameters.

I used this long before ash was needed (which was between 4.3BSD and 4.4
at the time of the lawsuit - sometime early/mid 90's).

Get yourself an original 4.2BSD /bin/sh and test that one - Berkeley did
almost nothing to sh (everyone used csh) so that should be quite close to
the v7 version.
(0004010)
joerg (reporter)
2018-04-26 14:39
edited on: 2018-04-26 14:40

Re: Note: 0004009

It is a bit more complex:

The original Bourne Shell did not include (and still does not) "c" in $- for sh -c.

The original Bourne Shell did allow set -r but did not yet support set +x.

Later set +o was added and since this was a security problem with the -r flag, some flags (besides "c") have been forbidden with set +-<X>.

The forbidden flags today are "sicrp"

(0004012)
kre (reporter)
2018-04-26 16:03
edited on: 2018-04-26 16:22

I have no idea why this issue is still under discussion, can we not just
stop?

It doesn't really matter why I used to use -$- (it was in my sh code
all over the place which is why I would have preferred to keep the
idiom, but I think I have removed it all now).

I know that the original sh did not include 'c' in $- - that's why -$-
was able to work (though the one liners that tend to be used with sh -c
would rarely have needed it anyway.) And I know there was no "set +".
I wasn't really aware of -r (that was just done when the name was rsh,
I don't believe that was ever in $- in the early shell, not that I would
have cared, or noticed, since (that) rsh was never of any interest).

I believe "set +o" is much more recent.

I also don't believe there are any "forbidden" flags, there are some that
cannot be set in some shells -- but can be in others - though not always with
any effect,. The 's' flag in the NetBSD shell is essentially a 1 bit
variable - scripts (if they're insane) can use it however they like, it
can be set/reset but that does nothing (except reveal the state of the
shell at startup, if it has not been changed). If the shell is reading stdin,
it is reading stdin, and turning off a flag does not change that (where
otherwise would it read the next command from?) Nor does turning it on
cause the shell to suddenly leave its script and start reading stdin.
I suppose it could just be left out of 'set' but then it would need to
be special cased in command line processing (as -c requires) and that
would take extra code (and so introduce more bugs) - easier to just
set it at startup, then leave it and ignore it.

Unless there is really some problem with the accepted text, which I do
not believe there is, can't we just concentrate on issues that need actual
resolution, rather than just making noise?

(0004147)
ajosey (manager)
2018-10-11 15:18

Interpretation Proposed: 11 Oct 2018
(0004165)
ajosey (manager)
2018-11-12 19:48

Interpretation approved: 12 November 2018

- Issue History
Date Modified Username Field Change
2016-03-26 07:30 kre New Issue
2016-03-26 07:30 kre Name => Robert Elz
2016-03-26 07:30 kre Section => 2.5.2
2016-03-26 07:30 kre Page Number => unknown
2016-03-26 07:30 kre Line Number => unknown
2016-03-26 08:12 Don Cragun Page Number unknown => 2324
2016-03-26 08:12 Don Cragun Line Number unknown => 73759-73761
2016-03-26 08:12 Don Cragun Interp Status => ---
2017-03-16 16:22 geoffclare Note Added: 0003617
2017-03-16 16:23 geoffclare Interp Status --- => Pending
2017-03-16 16:23 geoffclare Final Accepted Text => Note: 0003617
2017-03-16 16:23 geoffclare Status New => Interpretation Required
2017-03-16 16:23 geoffclare Resolution Open => Accepted As Marked
2017-03-16 16:23 geoffclare Tag Attached: tc3-2008
2017-03-16 17:16 kre Note Added: 0003618
2017-03-16 17:31 joerg Note Added: 0003619
2017-03-16 17:38 joerg Note Edited: 0003619
2017-03-16 18:07 kre Note Added: 0003620
2017-03-17 09:09 geoffclare Note Added: 0003622
2017-03-17 10:46 kre Note Added: 0003624
2017-03-17 10:54 joerg Note Added: 0003625
2017-03-17 12:07 kre Note Added: 0003630
2017-03-17 12:51 joerg Note Added: 0003634
2017-03-17 13:05 kre Note Added: 0003636
2018-04-26 11:44 McDutchie Note Added: 0004002
2018-04-26 12:12 joerg Note Added: 0004003
2018-04-26 12:23 joerg Note Edited: 0004003
2018-04-26 12:47 kre Note Added: 0004004
2018-04-26 12:55 kre Note Added: 0004005
2018-04-26 12:59 joerg Note Edited: 0004003
2018-04-26 13:06 joerg Note Added: 0004006
2018-04-26 13:58 McDutchie Note Added: 0004007
2018-04-26 13:59 kre Note Added: 0004008
2018-04-26 14:20 kre Note Added: 0004009
2018-04-26 14:34 joerg Note Edited: 0004006
2018-04-26 14:39 joerg Note Added: 0004010
2018-04-26 14:40 joerg Note Edited: 0004010
2018-04-26 16:03 kre Note Added: 0004012
2018-04-26 16:22 kre Note Edited: 0004012
2018-04-26 19:03 McDutchie Note Added: 0004013
2018-04-26 19:08 McDutchie Note Deleted: 0004013
2018-10-11 15:18 ajosey Interp Status Pending => Proposed
2018-10-11 15:18 ajosey Note Added: 0004147
2018-11-12 19:48 ajosey Interp Status Proposed => Approved
2018-11-12 19:48 ajosey Note Added: 0004165
2019-10-23 10:17 geoffclare Status Interpretation Required => Applied


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker