Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0001544 [1003.1(2016/18)/Issue7+TC2] Shell and Utilities Editorial Clarification Requested 2022-01-08 03:21 2022-05-19 09:05
Reporter calestyo View Status public  
Assigned To
Priority normal Resolution Accepted As Marked  
Status Applied  
Name Christoph Anton Mitterer
Organization
User Reference
Section uudecode
Page Number 3357
Line Number 113061-113063
Interp Status ---
Final Accepted Text Note: 0005799
Summary 0001544: uudecode: standardise or at least reserve - as another special symbol for decoding to stdout
Description As of now:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/uudecode.html [^]
doesn’t specify "-" as a special symbol for the -o option in order to print the decoded output to stdout, but only the special symbol '/dev/stdout'

The rationale even explains:
In early drafts, the [ -o outfile] option-argument allowed the use of - to mean standard output. The symbol - has only been used previously in POSIX.1-2017 as a standard input indicator. The standard developers did not wish to overload the meaning of - in this manner. The /dev/stdout concept exists on most modern systems. The /dev/stdout syntax does not refer to a new special file. It is just a magic cookie to specify standard output.
Desired Action The problem with that is that there is no strict guarantee that this file really exists or is available to the process.

Imagine a chrooted process where /dev/ and /proc are not bind-mounted or take e.g. Debian's initramfs, where at least as of now, /dev/stdout doesn't exist as a file.

IMO, the decision to use a file name as special symbol was rather unfortunate if not short sighted.
I've seen at least one implementation which did not support '/dev/stdout' (as a special symbol until recently (busybox, and even now it depends on build configuration so it's not guaranteed to work).


This can of course have the effect that output that should go to stdout, goes actually to a file /dev/stdout, if the implementation "forgot" about that speciality of POSIX, and the file doesn't exist as "special file" that also causes output to stdout.
The problem is that this could of course cause leakage of sensitive data.


At least GNU's uudecode and busybox' unconditionally support "-" as special symbol for -o.

It may make sense to reserve "-" as well for that purpose (i.e. that conforming applications must not use it as literal filename).... or even directly standardise it as such (i.e. that conforming implementations need to support it)).

Of course this may break compatibility if anyone really used it in the sense of a actual filename.

Alternatively a new option could be standardised, overriding -o and meaning write to stdout.


I think POSIX should at least reserve
Tags issue8
Attached Files

- Relationships

-  Notes
(0005584)
geoffclare (manager)
2022-01-10 10:08

It's odd that uudecode is required to treat a pathname of "-" encoded in the input file as meaning standard output (see STDOUT) but is not allowed to do so when the same pathname is specified with "-o -".

Solaris and macOS both create a file named "-" when "-o -" is specified, so the suggestion of disallowing this is unlikely to achieve consensus.

I think the best we can do for now is to make it unspecified whether "-o -" means standard output or a file named "-", perhaps with an entry under FUTURE DIRECTIONS saying a future version may require it to mean standard output.

Incidentally, the rationale is no longer correct when it says "The symbol - has only been used previously in POSIX.1-2017 as a standard input indicator", since the STDOUT section for uniq was changed in Issue 7 to say it is allowed to treat an output_file operand of "-" as meaning standard output.
(0005586)
calestyo (reporter)
2022-01-10 15:16

It would be a very very tiny step forward... but at least something.


OTOH, also Solaris/macOS would likely benefit from a more strict standardisation:

I guess it's more likely that code from a platform (typically Linux / GNU) that blindly assumes "-" to be stdout is used on their side (and thus produce unexpected results) than the other way round.

Are any people from Solaris / macOS involved here? Maybe one could just ask if these guys would be willing to adapt.
(0005592)
alanc (reporter)
2022-01-11 00:19

Oracle Solaris no longer formally participates in the standard committee, though a few developers like myself maintain an informal interest.

I cannot officially speak on behalf of Oracle, but we are generally willing to consider improving Solaris compatibility with Linux & BSD systems where it makes sense and doesn't break existing customer usage, but that would only help those who update their systems, not those using older releases.
(0005707)
shware_systems (reporter)
2022-02-24 15:51

When the decision was made to use /dev/stdout, XBD 10.1 should have been updated also as an additional entry to /dev/null/ and /dev/tty/, it appears. Otherwise its use is not portable. Entries for stdin and stderr probably useful also, as a Future Direction.
(0005708)
calestyo (reporter)
2022-02-24 16:01

True, though I'd consider that a separate issue, and even if that was changed, '-' or some *guaranteed* alternative mechanism like a new command line option should be standardised.

'/dev/stdout' as a special symbol alone would have been in principle already enough (despite it being a bit awkward)... but one simply sees that it happens far too easy, that implementations consider this a file, which - even if added to XBD 10.11 - might still simply be missing.
(0005709)
geoffclare (manager)
2022-02-24 16:26

Re Note: 0005707 The RATIONALE states "The /dev/stdout syntax does not refer to a new special file. It is just a magic cookie to specify standard output."
(0005710)
calestyo (reporter)
2022-02-24 16:39

I guess he meant that in the sense, that even if an implementation would not support that as magic cookie,... there would still be the safeguard whether the special file that does effectively the same.
(0005711)
geoffclare (manager)
2022-02-24 17:33

Proposed resolution (not finalised yet):

Page and line numbers are for Issue 8 draft 2.1

On Page 3275 line 111259 change:
/dev/stdout
to:
- or /dev/stdout


Change page 3276 lines 111285-111287 from:
If the file data header encoded by uuencode is or /dev/stdout, or the −o /dev/stdout option overrides the file data, the standard output shall be in the same format as the file originally encoded by uuencode. Otherwise, the standard output shall not be used.
to:
If the file data header encoded by uuencode or specified by the -o option is or /dev/stdout, the standard output shall be in the same format as the file originally encoded by uuencode. Otherwise, the standard output shall not be used.

On page 3276, after line 111305, Application Usage, add a paragraph:
In order to create an output file named - it needs to be specified as, for example, -o ./-, since - alone is considered a special name by uudecode.

On Page 3276 lines 111314-111318, delete:
In early drafts, the [−o outfile] option-argument allowed the use of − to mean standard output. The symbol − has only been used previously in POSIX.1-202x as a standard input indicator. The standard developers did not wish to overload the meaning of − in this manner.

On page 3278 at line 111357 change:
/dev/stdout
to:
- or /dev/stdout

On page 3281 after line 111490 add a new paragraph:
Since uudecode treats a decode_pathname of - to mean decode to standard output, in order to specify that a file named - is to be created, decode_pathname should be specified as ./-.
(0005712)
calestyo (reporter)
2022-02-24 21:34
edited on: 2022-02-25 03:08

"If the file data header encoded by uuencode or specified by the -o option is − or /dev/stdout, the standard output shall be in the same format as the file originally encoded by uuencode. Otherwise, the standard output shall not be used."

That is IMO ambigous (and even the current sentence is)...

It says basically: if either the header IR the option says '-' or '/dev/stdout', standard output should be used.

However, if the header would be set to '-' or '/dev/stdout' ... but the -o option would say /tmp/foo, than the latter should be used.



Perhaps:
"If the file data header encoded by uuencode is − or /dev/stdout (and the -o option is not used) or the -o option is − or /dev/stdout, the standard output shall be in the same format as the file originally encoded by uuencode. Otherwise, the standard output shall not be used."

(0005713)
kre (reporter)
2022-02-24 23:23

Re Note: 0005712

I understand the point, but the proposed wording is horrid. We should
just say what we mean, and not be afraid of using more than one sentence,
we don't pay a premium for using the '.' character.

As I understand it, the intent is more like

    If the -o option is used, the pathname shall be outfile, otherwise
    the pathname is that encoded in the file data header by uuencode.
    If the pathname so determined is "-" or "/dev/stdout", output
    in the same format as the file originally encoded by uuencode shall be
    sent to standard output, otherwise standard output shall not be used.

With the wording and formatting cleaned up to be nicer. Perhaps making the
earlier text more clearly contain the first sentemce, and limiting the STDOUT
section to simply the second (omitting "so determined" of course).
(0005715)
calestyo (reporter)
2022-02-25 03:13

What about your version but with:

   If the pathname determined by either way is any of the
   magic cookies "-" or "/dev/stdout" …

with the following intentions:

- making it more clear, that the 2nd sentence doesn't just relate to the "otherwise"-case of the first

- since the text uses the term "pathname", indicating again that these two are not really files, that would actually need to exist or be used
(0005717)
geoffclare (manager)
2022-02-25 10:07
edited on: 2022-02-25 10:08

I agree with calestyo and kre that the wording needs improvement. I think the best place to fix the problem is early in the description, by changing (P3275 L111240):
... attempt to create or overwrite the file described by the data (or overridden by the −o option). The pathname shall be contained in the data or specified by the −o option.
to:
... attempt to create or overwrite the file described by the data (or overridden by the -o option), or write the file contents described by the data to standard output if the pathname specified for the output file is either of the magic cookies or /dev/stdout.

The new STDOUT section can then be:
If the pathname specified for the output file is or /dev/stdout, the standard output shall be in the same format as the file originally encoded by uuencode. Otherwise, the standard output shall not be used.


(0005718)
kre (reporter)
2022-02-25 15:46

Re Note: 0005717

The division between what is in the description and stdout sections looks fine,
but the wording of the former still reads as if the implementation has an
option what it should write to (or write to in addition). I think it needs
to be more prescriptive, not "do this or that".

So for the L111240 replacement, I'd suggest something more like

    ... determine the pathname from the -o option if given, or if not,
    from the filename in the date. If that pathname is one of the magic
    cookies - or /dev/stdout then uudecode shall write the decoded file to
    standard output, otherwise it shall attempt to create or overwrite the
    file named by the pathname.
(0005722)
geoffclare (manager)
2022-03-01 11:45

Suggested new resolution:

Page and line numbers are for Issue 8 draft 2.1

On page 3275 line 111239 change:
The uudecode utility shall scan the input file, searching for data compatible with one of the formats specified in uuencode, and attempt to create or overwrite the file described by the data (or overridden by the −o option). The pathname shall be contained in the data or specified by the −o option.
to:
The uudecode utility shall scan the input file, searching for data compatible with one of the formats specified in uuencode, and determine the pathname for the output file from the -o option if given, otherwise from the input data. If the pathname for the output file is either of the magic cookies or /dev/stdout, uudecode shall write the decoded file to standard output, otherwise it shall attempt to create or overwrite the file named by the pathname.

On Page 3275 line 111259 change:
/dev/stdout
to:
- or /dev/stdout


Change page 3276 lines 111285-111287 from:
If the file data header encoded by uuencode is or /dev/stdout, or the −o /dev/stdout option overrides the file data, the standard output shall be in the same format as the file originally encoded by uuencode. Otherwise, the standard output shall not be used.
to:
If the pathname specified for the output file is or /dev/stdout, the standard output shall be in the same format as the file originally encoded by uuencode. Otherwise, the standard output shall not be used.

On page 3276, after line 111305, Application Usage, add a paragraph:
In order to create an output file named - it needs to be specified as, for example, -o ./-, since - alone is considered a special name by uudecode.

On Page 3276 lines 111314-111318, delete:
In early drafts, the [−o outfile] option-argument allowed the use of − to mean standard output. The symbol − has only been used previously in POSIX.1-202x as a standard input indicator. The standard developers did not wish to overload the meaning of − in this manner.

On page 3278 at line 111357 change:
/dev/stdout
to:
- or /dev/stdout

On page 3281 after line 111490 add a new paragraph:
Since uudecode treats a decode_pathname of - to mean decode to standard output, in order to specify that a file named - is to be created, decode_pathname should be specified as ./-.
(0005724)
quinq (reporter)
2022-03-02 09:49

Hello,

Just my naive two cents,

as it's clear that the defined behaviour will be slightly changed here,
maybe this would rather be a good time to deprecate
the usage of "/dev/stdout" as a magic cookie?

I'm not sure many implementations actually treat this as a special string
and would just try to output to the actual device.

As this device isn't specified by POSIX,
it would look more logical to me to unspecify this part
and let the user rely on the system implementation of /dev/stdout if provided,
hence the deprecation.

Using '-' as means of stdout looks like a more consistent solution.

Feel free to reject the idea ^^
(0005725)
steffen (reporter)
2022-03-02 12:50

FreeBSD uuencode only supports -o /dev/stdout not in-stream /dev/stdout, i stumbled over that in the past.
At least so when i looked last (2016), .. the commits since then do not make me think it has changed. (I could spawn a VM and test it this evening.)
(0005726)
calestyo (reporter)
2022-03-03 02:51
edited on: 2022-03-03 02:59

Re: https://austingroupbugs.net/view.php?id=1544#c5724 [^]

I don't think that would be a good idea:

The idea of this request was to a) give an alternative to /dev/stdout, which some implementations may not treat as magic cookie, and b) standardise behaviour "-" which is anyway supported by at least some implementations and often used for stdin/stdout.

This has in principle already the potential of breaking things (namely, when some implementation didn't support '-' and someone really used it as the file '-'.

But removing /dev/stdout, in the worst case, just breaks further things, namely when anyone relies on that being standard output and then it *might* not longer be so.

And at best, removing it, doesn't give much advantage:
- Implementations would just save one further string comparison.
- They would allow to actually write to the file /dev/stdout... but on many systems, this will be anyway a special file that means standard output again,... and on all others... which use-case is there, for writing a /dev/ file from contents from uuencode?

What I could agree, though, is if '-' would be named something like the "preferred" magic cookie.

But I wouldn't deprecate /dev/stdout (as magic cookie)... most probably not even in the long term.

(0005727)
calestyo (reporter)
2022-03-03 03:12
edited on: 2022-04-01 23:03

Re: https://austingroupbugs.net/view.php?id=1544#c5722 [^]

Looks quite good to me, but I'd change the following:


1) Regarding your:
> On page 3276, after line 111305, Application Usage,
> add a paragraph:
>
> In order to create an output file named - it needs
> to be specified as, for example, -o ./-, since - alone is
> considered a special name by uudecode.

- perhaps don't use "special name", when "magic cookie" is used before - or better: change "magic cookie" everywhere to "special name" or even "special name (which doesn't refer to a filename)".

- give the same example on how to create /dev/stdout, e.g. '///dev/stdout' or '/./dev/stdout'


2) And the same for uuencode at "page 3281 after line 111490 add a new paragraph", that is:

- also give the example on how to create /dev/stdout, e.g. '///dev/stdout' or '/./dev/stdout'


3) Change page 3276 lines 111316-111318 from:

Change:
"The /dev/stdout concept exists on most modern systems. The /dev/stdout syntax does not refer to a new special file."

To (something like):
"A special file /dev/stdout, that prints any data written to it to the respective process’ standard output, exists on most modern systems. The /dev/stdout syntax however does not refer to such special file."


I think that would explain better what's happening there, perhaps one could even add that the special file, would still work as a "fallback" if some implementation "forgot" to handle the special name/magic cookie.


Update:
Applied corrections from eblake and changed '//dev/stdout' to '///dev/stdout' above.

(0005732)
eblake (manager)
2022-03-03 20:35
edited on: 2022-03-03 20:36

Regarding Note: 0005727, the names '/./dev/stdout' or '///dev/stdout' are viable alternative spellings, but the name '//dev/stdout' is not (the standard specifically permits implementation-specific behavior for exactly two leading /).

(0005734)
calestyo (reporter)
2022-03-05 03:18

Oh yes, you are absolutely right. I forgot about the special cases in 3.271 Pathname.
(0005799)
geoffclare (manager)
2022-04-14 15:47
edited on: 2022-04-14 15:48

Page and line numbers are for Issue 8 draft 2.1

On page 3275 line 111239 change:
The uudecode utility shall scan the input file, searching for data compatible with one of the formats specified in uuencode, and attempt to create or overwrite the file described by the data (or overridden by the −o option). The pathname shall be contained in the data or specified by the −o option.
to:
The uudecode utility shall scan the input file, searching for data compatible with one of the formats specified in uuencode, and determine the pathname for the output file from the -o option if given, otherwise from the input data. If the pathname for the output file is either of the magic cookies or /dev/stdout, uudecode shall write the decoded file to standard output, otherwise it shall attempt to create or overwrite the file named by the pathname.

On Page 3275 line 111259 change:
/dev/stdout
to:
- or /dev/stdout

Change page 3276 lines 111285-111287 from:
If the file data header encoded by uuencode is or /dev/stdout, or the −o /dev/stdout option overrides the file data, the standard output shall be in the same format as the file originally encoded by uuencode. Otherwise, the standard output shall not be used.
to:
If the pathname specified for the output file is or /dev/stdout, the standard output shall be in the same format as the file originally encoded by uuencode. Otherwise, the standard output shall not be used.

On page 3276, after line 111305, Application Usage, add a paragraph:
In order to create an output file named - it needs to be specified using an alternative pathname, for example, -o ./-, since - alone is considered a magic cookie by uudecode. Likewise, in order to write to an output file named /dev/stdout it also needs to be specified as, for example, -o ///dev/stdout.

On Page 3276 lines 111314-111318, change:
In early drafts, the [−o outfile] option-argument allowed the use of − to mean standard output. The symbol − has only been used previously in POSIX.1-202x as a standard input indicator. The standard developers did not wish to overload the meaning of − in this manner. The /dev/stdout concept exists on most modern systems. The /dev/stdout syntax does not refer to a new special file. It is just a magic cookie to specify standard output.
to:
In early drafts, the [−o outfile] option-argument allowed the use of to mean standard output. The standard developers did not wish to overload the meaning of in this manner, resulting in previous versions only using /dev/stdout for this purpose. POSIX.1-202x now allows it as most implementations were already supporting as an extension. The file /dev/stdout exists as a special file on most modern systems. However, the /dev/stdout syntax in uudecode does not refer to a new file. It is just a magic cookie to specify standard output.

On page 3278 at line 111357 change:
/dev/stdout
to:
- or /dev/stdout

On page 3281 after line 111490 add a new paragraph:
Since uudecode treats a decode_pathname of - to mean decode to standard output, in order to specify that a file named - is to be created, decode_pathname should be specified using an alternative pathname, for example ./-. Likewise, in order to specify that a file with the pathname /dev/stdout is to be written, decode_pathname should be specified as, for example, ///dev/stdout.


(0005803)
calestyo (reporter)
2022-04-15 01:19

Reviewed and mostly good from my side, with the single exception:

When you use "///dev/stdout", I'd recommend to explicitly mention that "//dev/stdout" is *not* the same.

The standard is most likely also read by people that are not experts, just like I am, and who don't know or remember that two leading / in a pathname are actually special as per 3.271 Pathname.


Given that many people think that *any* sequence of / folds in a pathname... I'd probably even give the '/./dev/stdout' example.


What about:
"... for example, -o /./dev/stdout or -o ///dev/stdout but not -o //dev/stdout (see 3.271 Pathname)."

respectively

"... for example, /./dev/stdout or ///dev/stdout but not //dev/stdout (see 3.271 Pathname)."


Anyway... if you think it's good as it is, I wouldn't insist on any of this (not that my opinion would count much anyway ^^).

- Issue History
Date Modified Username Field Change
2022-01-08 03:21 calestyo New Issue
2022-01-08 03:21 calestyo Name => Christoph Anton Mitterer
2022-01-08 03:21 calestyo Section => uudecode
2022-01-08 03:21 calestyo Page Number => N/A
2022-01-08 03:21 calestyo Line Number => N/A
2022-01-10 10:08 geoffclare Note Added: 0005584
2022-01-10 15:16 calestyo Note Added: 0005586
2022-01-11 00:19 alanc Note Added: 0005592
2022-02-24 09:24 Don Cragun Page Number N/A => 3357
2022-02-24 09:24 Don Cragun Line Number N/A => 113061-113063
2022-02-24 09:24 Don Cragun Interp Status => ---
2022-02-24 15:51 shware_systems Note Added: 0005707
2022-02-24 16:01 calestyo Note Added: 0005708
2022-02-24 16:26 geoffclare Note Added: 0005709
2022-02-24 16:39 calestyo Note Added: 0005710
2022-02-24 17:33 geoffclare Note Added: 0005711
2022-02-24 17:35 geoffclare Final Accepted Text => Note: 0005711
2022-02-24 17:35 geoffclare Status New => Resolution Proposed
2022-02-24 21:34 calestyo Note Added: 0005712
2022-02-24 23:23 kre Note Added: 0005713
2022-02-25 03:08 calestyo Note Edited: 0005712
2022-02-25 03:13 calestyo Note Added: 0005715
2022-02-25 10:07 geoffclare Note Added: 0005717
2022-02-25 10:08 geoffclare Note Edited: 0005717
2022-02-25 15:46 kre Note Added: 0005718
2022-03-01 11:45 geoffclare Note Added: 0005722
2022-03-02 09:49 quinq Note Added: 0005724
2022-03-02 12:50 steffen Note Added: 0005725
2022-03-03 02:51 calestyo Note Added: 0005726
2022-03-03 02:59 calestyo Note Edited: 0005726
2022-03-03 03:12 calestyo Note Added: 0005727
2022-03-03 20:35 eblake Note Added: 0005732
2022-03-03 20:36 eblake Note Edited: 0005732
2022-03-03 20:36 eblake Note Edited: 0005732
2022-03-05 03:18 calestyo Note Added: 0005734
2022-03-05 03:19 calestyo Note Edited: 0005727
2022-04-01 23:03 calestyo Note Edited: 0005727
2022-04-14 15:47 geoffclare Note Added: 0005799
2022-04-14 15:48 geoffclare Note Edited: 0005799
2022-04-14 15:48 geoffclare Final Accepted Text Note: 0005711 => Note: 0005799
2022-04-14 15:48 geoffclare Status Resolution Proposed => Resolved
2022-04-14 15:48 geoffclare Resolution Open => Accepted As Marked
2022-04-14 15:49 geoffclare Tag Attached: issue8
2022-04-15 01:19 calestyo Note Added: 0005803
2022-05-19 09:05 geoffclare Status Resolved => Applied


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker