0001039: Which option flags should appear in $- ?

Notes
(0003617) geoffclare (manager) 2017-03-16 16:22	Interpretation response ------------------------ The standard states that $- includes the options set on the sh command line, and conforming implementations must conform to this. However, concerns have been raised about this which are being referred to the sponsor. Rationale: ------------- Some existing shells do not include c or s in $-. Notes to the Editor (not part of this interpretation): ------------------------------------------------------- On 2016 edition page 2350 line 74880 section 2.5.2, after: (Hyphen.) Expands to the current option flags (the single-letter option names concatenated into a string) as specified on invocation, by the set special built-in command, or implicitly by the shell. add: It is unspecified whether the -c and -s options are included in the expansion of <tt>$-</tt>. The -i option shall be included in <tt>$-</tt> if the shell is interactive, regardless of whether it was specified on invocation.

(0003618) kre (reporter) 2017-03-16 17:16	I agree to what is being proposed for the substantive part of the standard, but it appears (from the etherpad, and the resolution) that the point of this issue was missed. That is, once upon a time, it used to be possible to do set -$- and expect it would work (while being a no-op), or set +$- to clear all flags set (whether this clears the i or m flags is not important for current purposes - what matters is whether it generates a syntax error from the shell). The problem is that shells that include 'c' in particular in $- (or some of them) later object to "set -c" (which is in itself, not surprising). That makes set -$- a syntax error if the shell was started as sh -c ... Would it be possible to add some test to the rationale, to suggest to shell implementors that anything that is in $- should be acceptable input to "set" ? Whether implementors choose to leave 'c' (in particular) out of $-, and then object to set -c, or whether they include 'c' in $- (when appropriate) and then simply ignore "set -c" (make it a no-op, same with set +c of course) doesn't matter, just so that set +$- and set -$- never produce syntax errors. kre

(0003619) joerg (reporter) 2017-03-16 17:31 edited on: 2017-03-16 17:38	What you like to see here was never possible with ksh88 - the role model for POSIX It does not even work with the historic Bourne Shell, if you called "sh -i". You may believe what your memory tells you because the historic Bourne Shell did not automatically include -i for interactive shells.

(0003620) kre (reporter) 2017-03-16 18:07	You are almost certainly correct about the original Bourne shell and -i, but it was hardly relevant. In normal use, the issue only arose in scripts (there were no functions), and your typical script is not run with sh -i, nor does it (attempt to) "set -i", so even if there had been a problem, I would never have noticed it. The effect of the way that it is now, is that with many shells, scripts that I have had for 30 years or more, no longer work (with some shells - fortunately, with the shell I maintain, and use almost exclusively for scripts, it all works, which you might think is a happy accident...)

(0003622) geoffclare (manager) 2017-03-17 09:09	Executing set +$- is not something we should encourage applications to do. There can be option letters relating to extensions in $-, so by executing set +$- (if it succeeds) the application has made use of an extension and can no longer rely on the shell behaving as described in the standard. Applications should use set +... only to turn off specific options that they want to turn off. They cannot expect "turn off all options" to achieve something useful or sensible in all implementations.

(0003624) kre (reporter) 2017-03-17 10:46	Re: so by executing set +$- (if it succeeds) the application has made use of an extension I would certainly hope (and the standard should encourage) that if extensions are added they are enabled by turning on flags (if that is the mechanism selected) rather than by turning flags off. The default state for executing a script called as "sh script" really should be for all flags to be clear, and in that state the shell should be operating in its normal operating mode (which might or might not be posix compliant, but it will usually be close.) So, turning flags off really should be safe, and any shell that requires a flag to be set (as distinct from perhaps a -o option) to operate as a standard shell should be regarded as broken. All that said, my main aim is not "set +$-" as useful as that might be, but "set -$-" because that is what many scripts I own do. That is, a no-op, just set the flags that are already set. It is an idiom that traces back to (probably) bugs (or design issues) in the earliest Bourne shells, where "set a b c" would set $1 $2 $3 as expected, but would also clear all the flags, in some versions "set - a b c" would work, but not in others, but in all (at the time) "set -$- a b c" would leave the flags in their current state, and also set the arg list as requested (provided there was at least one arg, hence the usual idiom is "set -$- X <whatever>; shift") So my scripts (many of which were written back in the early 80's) all use that idiom, and personally, I think it is very important to keep working. Until shells started adding 'c' to $- and then refusing set -c this was the one constant about the set command that was consistently implemented, everywhere. kre

(0003625) joerg (reporter) 2017-03-17 10:54	There are several shell implementations that use something like set -o posix to make it POSIX compliant and this is done automatically under certain conditions. If such a shell implements a one-letter equivalent for the "posix" option, set +$- would turn POSIX compliance off.

(0003630) kre (reporter) 2017-03-17 12:07	Re: If such a shell implements a one-letter equivalent for the "posix" option, set +$- would turn POSIX compliance off. If the shell implementer was half way rational, the one letter flag equivalent of such an option would enable "extended mode" and turning it off would be equivalent to "set -o posix". So, if P is the flag for posix mode, P in $- would mean not posix, and set +P would be the same as "set -o posix".

(0003634) joerg (reporter) 2017-03-17 12:51	In the recent Bourne Shell, P is the mode to switch on fine grained privileges behavior for the Solaris kernel mode privs. If you turn this off, you may no longer be able to e.g. run privileged backup programs if you do what you propose inside the backup script.

(0003636) kre (reporter) 2017-03-17 13:05	Re: 3634 - of course, disabling options disables the functionality. For a more mundane example, which applies everywhere, if a script does set +x (either explicitly, or because of set +$-) then a user attempting to debug the script with "sh -x script" is going to be defeated (or at least annoyed.) Actions have consequences, and everything that is done has the potential to cause some side effect elsewhere. But once again, I do not need a tutorial on why I should not be doing set +$- in my scripts, what I would like, is for some kind of guarantee that "set -$-" will remain a no-op.

(0004002) McDutchie (reporter) 2018-04-26 11:44	kre, I don't think you're going to get a guarantee that the 'set -$-' workaround is a no-op, because in fact it fails on most current and historical shells. Historical: $ sh -c 'set -$-' # Bourne shell, Xenix (1988) sh: -s: bad option(s) $ sh -c 'set -$-' # Bourne shell, Solaris 10.3 (2010) sh: -s: bad option(s) $ ksh -c 'set -$-' # ksh88, Solaris 10.3 (2010) ksh: -shc: bad option(s) Presumably it works on the historical ash (Almquist shell) as it does on all current derivatives, but I don't have any historical ash I can test. Current: $ ksh -c 'set -$-' # ksh93 ksh: set: -c: unknown option Usage: set [-sabefhkmnprtuvxBCGH] [-A name] [-o[option]] [arg ...] $ ksh -c 'set -$-' # NetBSD ksh (pdksh) ksh: set: -c: unknown option ksh: set: $ ksh -c 'set -$-' # OpenBSD sh and ksh (pdksh) ksh: set: -c: unknown option $ mksh -c 'set -$-' mksh: set: -c: unknown option $ bash -c 'set -$-' bash: line 0: set: -c: invalid option set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...] $ yash -c 'set -$-' set: the cmdline option cannot be changed once the shell has been initialized It only works on: - NetBSD sh (of course) - FreeBSD sh - dash - Busybox ash - zsh ...so basically on Almquist derivatives plus zsh. That minority is nowhere close to establishing de facto standard behaviour.

(0004003) joerg (reporter) 2018-04-26 12:12 edited on: 2018-04-26 12:59	Interesting... this new requirement is in conflict with: - (Hyphen.) Expands to the current option flags (the single-letter option names concatenated into a string) as specified on invocation, by the set special built-in command, or implicitly by the shell. and thus should be removed again since it would make existing implementations non-conforming. The purpose of the standard is to document existing behavior, not to invent new behavior that is in conflict with existing behavior. BTW: the documented method to save the current settings in a POSIX shell is to call "set +o", keep the output and run that result if needed.

(0004004) kre (reporter) 2018-04-26 12:47	Re note 4002 ... the resolution of this (note 3617) is fine. I have more or less given up on -$- ... it was useful back in the original Bourne shell days, when there were all kinds of bugs in the "set" command (way before -- existed - and long before ash was even a dream, or the need for it was anywhere on the horizon - very early 80's or late 70's). It certainly did work on old Bourne shells (and that will be why it worked in ash and its descendants, I would assume.) The comments that followed note 3617 related to comments I saw on the etherpad from when this bug was discussed.

(0004005) kre (reporter) 2018-04-26 12:55	Re note 4003. Joerg, there is no conflict. -i is an option to sh (see the utilities page for sh) note "or implicitly by the shell" in the text you quoted. If the shell implictly sets the 'i' flag (because it is started in a way that meets the requirements of being interactive) so should always have been included in $- It was just that most shells concentrated only on the flags defined to be able to be used with the "set" builtin, and not necessarily on those which are only relevant on the command line. It was to clarify that that the original bug report was submitted. If your sh doesn't set 'i' - fix it. It was always broken.

(0004006) joerg (reporter) 2018-04-26 13:06 edited on: 2018-04-26 14:34	I added "i" to $- for an interactive bosh in June 2012 because it seemed to be useful. I however do't understand why "set -c" should be allowed or why $- should not include "r" for sh -r. Note: change the last part of the last sentence since the Bourne Shell does not include "c" in $- for sh -c.

(0004007) McDutchie (reporter) 2018-04-26 13:58	kre wrote: > It certainly did work on old Bourne > shells (and that will be why it worked in ash and its descendants, > I would assume.) Did you miss that I just tested it on a 1988 Bourne shell (Xenix)? It doesn't work. To me this looks like an old ash-specific workaround for an ash-specific bug.

(0004008) kre (reporter) 2018-04-26 13:59	Re 4006: \| I however do't understand why "set -c" should be allowed It isn't. What makes you think that changed? \| or why $- should not include "c" for sh -c It can (it will be unspecified, as many shells do not). This one I would have preferred not to see (it is the one that broke -$-) but shells do it. I don't see any point (why would a script ever care whether it was on the command line or in a file) in it, but given that we're not allowing in $- only the options that can be altered (or at least, preserved) with "set" allowing it does no harm either.

(0004009) kre (reporter) 2018-04-26 14:20	Re note: 4007. No, I did not miss the reference to the 1988 vintage sh test. That's much too late, I cannot even imagine what might have been done to that shell (that was probably a SysVR3 shell, which had lots of changes). The one I am referring to was the original 7th edition (and 32V) shell from Bell Labs. In that, there was no --, to set the args, "set a b c" would fail if "a" happened to start with a '-' (it would be interpreted as option flags). The workaround was "set - a b c" but that would reset all the flags (there was no "set +..." in that shell, the when a '-' option was given to "set" it cleared the flags, and then set those given.) Hence, "set -$- a b c" - set the flags back to what they were, and then set the positional parameters. I used this long before ash was needed (which was between 4.3BSD and 4.4 at the time of the lawsuit - sometime early/mid 90's). Get yourself an original 4.2BSD /bin/sh and test that one - Berkeley did almost nothing to sh (everyone used csh) so that should be quite close to the v7 version.

(0004010) joerg (reporter) 2018-04-26 14:39 edited on: 2018-04-26 14:40	Re: Note: 0004009 It is a bit more complex: The original Bourne Shell did not include (and still does not) "c" in $- for sh -c. The original Bourne Shell did allow set -r but did not yet support set +x. Later set +o was added and since this was a security problem with the -r flag, some flags (besides "c") have been forbidden with set +-<X>. The forbidden flags today are "sicrp"

(0004012) kre (reporter) 2018-04-26 16:03 edited on: 2018-04-26 16:22	I have no idea why this issue is still under discussion, can we not just stop? It doesn't really matter why I used to use -$- (it was in my sh code all over the place which is why I would have preferred to keep the idiom, but I think I have removed it all now). I know that the original sh did not include 'c' in $- - that's why -$- was able to work (though the one liners that tend to be used with sh -c would rarely have needed it anyway.) And I know there was no "set +". I wasn't really aware of -r (that was just done when the name was rsh, I don't believe that was ever in $- in the early shell, not that I would have cared, or noticed, since (that) rsh was never of any interest). I believe "set +o" is much more recent. I also don't believe there are any "forbidden" flags, there are some that cannot be set in some shells -- but can be in others - though not always with any effect,. The 's' flag in the NetBSD shell is essentially a 1 bit variable - scripts (if they're insane) can use it however they like, it can be set/reset but that does nothing (except reveal the state of the shell at startup, if it has not been changed). If the shell is reading stdin, it is reading stdin, and turning off a flag does not change that (where otherwise would it read the next command from?) Nor does turning it on cause the shell to suddenly leave its script and start reading stdin. I suppose it could just be left out of 'set' but then it would need to be special cased in command line processing (as -c requires) and that would take extra code (and so introduce more bugs) - easier to just set it at startup, then leave it and ignore it. Unless there is really some problem with the accepted text, which I do not believe there is, can't we just concentrate on issues that need actual resolution, rather than just making noise?

(0004147) ajosey (manager) 2018-10-11 15:18	Interpretation Proposed: 11 Oct 2018

(0004165) ajosey (manager) 2018-11-12 19:48	Interpretation approved: 12 November 2018

Issue History
Date Modified	Username	Field	Change
2016-03-26 07:30	kre	New Issue
2016-03-26 07:30	kre	Name	=> Robert Elz
2016-03-26 07:30	kre	Section	=> 2.5.2
2016-03-26 07:30	kre	Page Number	=> unknown
2016-03-26 07:30	kre	Line Number	=> unknown
2016-03-26 08:12	Don Cragun	Page Number	unknown => 2324
2016-03-26 08:12	Don Cragun	Line Number	unknown => 73759-73761
2016-03-26 08:12	Don Cragun	Interp Status	=> ---
2017-03-16 16:22	geoffclare	Note Added: 0003617
2017-03-16 16:23	geoffclare	Interp Status	--- => Pending
2017-03-16 16:23	geoffclare	Final Accepted Text	=> Note: 0003617
2017-03-16 16:23	geoffclare	Status	New => Interpretation Required
2017-03-16 16:23	geoffclare	Resolution	Open => Accepted As Marked
2017-03-16 16:23	geoffclare	Tag Attached: tc3-2008
2017-03-16 17:16	kre	Note Added: 0003618
2017-03-16 17:31	joerg	Note Added: 0003619
2017-03-16 17:38	joerg	Note Edited: 0003619
2017-03-16 18:07	kre	Note Added: 0003620
2017-03-17 09:09	geoffclare	Note Added: 0003622
2017-03-17 10:46	kre	Note Added: 0003624
2017-03-17 10:54	joerg	Note Added: 0003625
2017-03-17 12:07	kre	Note Added: 0003630
2017-03-17 12:51	joerg	Note Added: 0003634
2017-03-17 13:05	kre	Note Added: 0003636
2018-04-26 11:44	McDutchie	Note Added: 0004002
2018-04-26 12:12	joerg	Note Added: 0004003
2018-04-26 12:23	joerg	Note Edited: 0004003
2018-04-26 12:47	kre	Note Added: 0004004
2018-04-26 12:55	kre	Note Added: 0004005
2018-04-26 12:59	joerg	Note Edited: 0004003
2018-04-26 13:06	joerg	Note Added: 0004006
2018-04-26 13:58	McDutchie	Note Added: 0004007
2018-04-26 13:59	kre	Note Added: 0004008
2018-04-26 14:20	kre	Note Added: 0004009
2018-04-26 14:34	joerg	Note Edited: 0004006
2018-04-26 14:39	joerg	Note Added: 0004010
2018-04-26 14:40	joerg	Note Edited: 0004010
2018-04-26 16:03	kre	Note Added: 0004012
2018-04-26 16:22	kre	Note Edited: 0004012
2018-04-26 19:03	McDutchie	Note Added: 0004013
2018-04-26 19:08	McDutchie	Note Deleted: 0004013
2018-10-11 15:18	ajosey	Interp Status	Pending => Proposed
2018-10-11 15:18	ajosey	Note Added: 0004147
2018-11-12 19:48	ajosey	Interp Status	Proposed => Approved
2018-11-12 19:48	ajosey	Note Added: 0004165
2019-10-23 10:17	geoffclare	Status	Interpretation Required => Applied

Relationships

Aardvark Mark IV