Austin Group Defect Tracker

Aardvark Mark IV

Main | My View | View Issues | Change Log | Docs
  

Viewing Issue Simple Details [ Jump to Notes ] [ Issue History ] [ Print ]
ID Category Severity Type Date Submitted Last Update
0000248 [1003.1(2008)/Issue 7] Shell and Utilities Objection Error 2010-04-30 20:20 2024-06-11 08:53
Reporter dwheeler View Status public  
Assigned To ajosey
Priority normal Resolution Accepted As Marked  
Status Closed  
Name David A. Wheeler
Organization
User Reference
Section Shell & Utilities
Page Number 2789,3003,3260,3385,and more
Line Number 91004,99054-99063,108778,113329-113334,and more
Interp Status ---
Final Accepted Text Note: 0006560
Summary 0000248: Fix the numerous filename/pathname processing errors in specification examples
Description Many examples in the specification fail, sometimes spectacularly, when given filenames or pathnames permitted by the specification. The specification currently permits filenames to include newline, return, tab, escape, leading "-", double-quotes, single-quotes, ampersand, asterisk, and other "interesting" characters. However, many of the specification's examples will fail to work correctly when such filenames are present.

In a few cases, there's a note that the example presumes that some of these will not occur, but there is no standard way to *ensure* that such filenames cannot occur. In addition, many attackers have learned to *intentionally* create such filenames, since there is no way to prevent them in general. Thus, multi-user systems, or systems that receive external data (e.g., via USB sticks and network drives) are still at risk if they used these examples. This incorrect filename handling can be the basis of security problems, as noted in:
 http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html [^]
It's worth noting that this is not a new problem. Line 82265 noted that the specification has been changed before "to remove the unreliable use of find | xargs".


The examples will need to be examined for patterns such as these:

# WRONG. Fails if newlines/tabs/space/backslash/apostrophe/double-quote/
# ampersand in filenames:
find . -type f | xargs COMMAND

# WRONG. Fails if a file in the current directory begins with "-" or
# if there are no files in the current directory:
COMMAND *
for file in * ; do
 COMMAND "$file"
done



Please either:
1. Fix the various errors. Below are some of them, to get started. It's concerning that even the experts who developed this specification (and they're definitely experts!) made these mistakes. I believe this shows that it is too difficult to correctly use systems while conforming to this specification.
2. Change the rules on filenames to forbid certain filenames. In particular, forbidding control characters in filenames (at least bytes 1-31 and maybe 127) would completely eliminate a number of the problems. In addition, forbidding leading "-" would solve some more (though not as many as the control-characters one).
Desired Action Page 2789, Line 91004:
Change:
 find . −type f | xargs hash
To:
 find . -type f -exec hash {} \;
Rationale:
 This example tries to explain why hash won't work in this case,
 but the example should only have *that* error, not *other* errors as well.

Page 3003, line 99054-99063:
Remove this entire example.
I believe it is not possible to do what this example is attempting to do
while using standard POSIX utilities, at least by using pax.
This example attempts to determine if the filenames in a pax archive
are valid, but filenames can include newline, and pax does not have any
way to disambiguate these cases (e.g., by having a -0 option) in its output.
POSIX tools simply aren't capable of doing accurate filename checking
in this case, at least not without re-implementing (say, in C) a significant
portion of pax.

Page 3260, line 108778:
Replace:
  find . −type f | xargs type
with:
  find . −type f -exec type {} \;

Page 3385, lines 113329-113334:
Remove this example, or change it to work properly
(and that may not be easy).
Although it documents that it will not work if
filenames contain newline, there's no way to be guaranteed of that.
It also presumes that the filenames don't have {}, which isn't guaranteed either. If this was executed on a system where a filename DID contain a newline,
or where the source/target directories contained {},
*disaster* could ensue, and examples should not recommend
techniques that might be disastrous.
Most of the other examples on page 3385 should arguably be
removed as well, since there's no mechanism for guaranteeing that
filenames do not include newline (if xargs -0 is added, then they
can probably be easily repaired and kept).
Tags applied_after_i8d3, issue8
Attached Files

- Relationships
related to 0000251Closedajosey Forbid newline, or even bytes 1 through 31 (inclusive), in filenames 

-  Notes
(0000885)
Don Cragun (manager)
2011-07-06 23:59

 We agree that there are problems in the examples listed.
The fixes for these issues will be delayed until 0000251 is resolved.
The resolution to 0000251 is expected to make many of these
examples much easier to implement reliably using standard features.
(0006560)
geoffclare (manager)
2023-10-30 16:34

 Page and line numbers are for Issue 8 draft 3.

Page 2973, Line 99511 (hash APPLICATION USAGE):
Change:
find . −type f | xargs hash
To:
find . -type f -exec hash {} +


Page 3441, line 117538 (type APPLICATION USAGE):
Replace:
find . −type f | xargs type
with:
find . −type f -exec type {} +


On page 3568 line 122228 (xargs EXAMPLES) change example 2 from:
The following command invokes diff with successive pairs of arguments originally typed as command line arguments. It assumes there are no embedded <newline> characters in the elements of the original argument list.
printf "%s\n" "$@" | sed 's/[^[:alnum:]]/\\&/g' |
xargs -E "" -n 2 -x diff
to:
The following command invokes diff with successive pairs of arguments originally typed as command line arguments.
printf "%s\0" "$@" | xargs -0 -n 2 -x diff --


On page 3568 line 122233 change example 3 from:
In the following commands, the user is asked which files in the current directory (excluding dotfiles) are to be archived. The files are archived into arch; a, one at a time or b, many at a time. The commands assume that no filenames contain <blank>, <newline>, <backslash>, <apostrophe>, or double-quote characters.
a. ls | xargs -E "" -p -L 1 ar -r arch
b. ls | xargs -E "" -p -L 1 | xargs -E "" ar -r arch
to:
In the following command, the user is asked which regular files below the current directory are to be archived.
find . -type f -print0 | xargs -0 -p -L 1 ar -r arch


On page 3568, lines 122242-122247 remove example 5

- Issue History
Date Modified Username Field Change
2010-04-30 20:20 dwheeler New Issue
2010-04-30 20:20 dwheeler Status New => Under Review
2010-04-30 20:20 dwheeler Assigned To => ajosey
2010-04-30 20:20 dwheeler Name => David A. Wheeler
2010-04-30 20:20 dwheeler Section => Shell & Utilities
2010-04-30 20:20 dwheeler Page Number => 2789,3003,3260,3385,and more
2010-04-30 20:20 dwheeler Line Number => 91004,99054-99063,108778,113329-113334,and more
2011-07-06 23:59 Don Cragun Note Added: 0000885
2023-10-30 15:26 eblake Relationship added related to 0000251
2023-10-30 16:34 geoffclare Note Added: 0006560
2023-10-30 16:36 geoffclare Interp Status => ---
2023-10-30 16:36 geoffclare Final Accepted Text => Note: 0006560
2023-10-30 16:36 geoffclare Status Under Review => Resolved
2023-10-30 16:36 geoffclare Resolution Open => Accepted As Marked
2023-10-30 16:37 geoffclare Tag Attached: issue8
2023-11-21 09:49 geoffclare Status Resolved => Applied
2023-11-21 09:49 geoffclare Tag Attached: applied_after_i8d3
2024-06-11 08:53 agadmin Status Applied => Closed

Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker