Austin Group Defect Tracker

Aardvark Mark III


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0000243 [1003.1(2008)/Issue 7] Shell and Utilities Objection Enhancement Request 2010-04-29 19:23 2011-11-16 18:22
Reporter dwheeler View Status public  
Assigned To ajosey
Priority normal Resolution Open  
Status Under Review  
Name David A. Wheeler
Organization IDA
User Reference
Section find
Page Number 2740
Line Number 89194
Interp Status ---
Final Accepted Text
Summary 0000243: Add -print0 to "find"
Description The POSIX specification and common implementations permit nearly all bytes to be in pathnames, and yet it is surprisingly difficult to portably and correctly process such pathnames. This is one of the more common reason for security vulnerabilities (see CERT’s "Secure Coding" item MSC09-C, CWE 78, CWE 73, and CWE 116, and the 2009 CWE/SANS Top 25 Most Dangerous Programming Errors). For more details about this problem, see:
 http://www.dwheeler.com/essays/filenames-in-shell.html [^]
 http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html [^]

The find command's "-exec...+" was intended to fix this, but it is simply inadequate. This is only practical for trivial commands. It also fails to acknowledge a very common construct, find ... -print0 | xargs -0, which is technically not portable (it's not in the spec) but is actually in wide use.

The 2008 specification notes that "Other implementations have added other ways to get around this problem, notably a -print0 primary that wrote filenames with a null byte terminator. This was considered here, but not adopted. Using a null terminator meant that any utility that was going to process find's -print0 output had to add a new option to parse the null terminators it would now be reading." I believe that this decision must be revisited. While it's true that adding null terminator support means that other extensions are necessary, the POSIX -exec...+ construct is simply inadequate to support robust filename processing. Complex commands are rediculously unreadable when placed there, for example, and xargs supports other capabilities (such as limiting the number of parameters) that find does not duplicate. Nor should find duplicate xargs; the beauty of POSIX is that different tools can be good at one job. POSIX should either completely forbid the characters such as newline in filenames, or it should be extended to adequately support such filenames.

The current situation is that it is too hard to *correctly* process filenames, leading to a number of security vulnerabilities. Expecting users and developers to use complicated constructs to handle filenames is unreasonable and dangerous; they should be given a safer and easy-to-use set of constructs for this common case.
Desired Action After line 89195, add:

-print0
The primary shall always evaluate as true; it shall cause the current pathname to be written to standard output, followed by a null byte.


In lines 89387-89401, delete the now-obsolete text "Other implementations... reading."

In line 89285, append: "(but note that pathnames may include newlines, so you cannnot be sure that each line is actually a different pathname)"

In the "STDOUT" section, after line 89257, state:

The −print0 primary shall cause the current pathnames to be written to standard output, with each pathname terminated by a null byte. The format shall be:
           "%s", <path>
  followed by a null byte for each <path>.

Note that this change is a prerequisite for several other proposals that are necessary to make "find" useful and secure for ALL pathnames permitted by POSIX.
Tags No tags attached.
Attached Files

- Relationships
related to 0000244Under Reviewajosey 1003.1(2008)/Issue 7 Add -0 to xargs 
related to 0000245Under Reviewajosey 1003.1(2008)/Issue 7 Add -0 option to shell's "read" 
has duplicate 0000903Closed 1003.1(2013)/Issue7+TC1 Please, add find -print0, xargs -0, read -d and other such options 

-  Notes
(0000882)
Don Cragun (manager)
2011-07-06 23:54

The current plan is to add a set of byte values (based on single-byte characters in
the C Locale) that will not be allowed in newly created filenames using 0000251
as the bug to make the changes. If consensus is reached on a resolution for bug
251, the plan is to reject and close bugs 243, 244, and 245. These three bugs
will remain open until bug 251 is resolved.
(0001020)
dwheeler (reporter)
2011-11-16 18:22

On further reflection, I recommend that bugs 243, 244, and 245 be accepted, regardless of the resolution of bug 251.

Adding these capabilities will make it easier to implement portable applications. Most POSIX systems today permit filenames with include anything except NUL (including newline). Even if a future version of POSIX forbids it, there's no guarantee that implementations will move quickly to implement this change to POSIX. In addition, most application developers will want to develop software that works correctly on both older and newer systems. Technically older POSIX systems need not implement bug 243, 244, and 245, but they are very widely implemented.

Adding these capabilities will make many programs - and various widely-recommended and used constructs - POSIX-compliant.

- Issue History
Date Modified Username Field Change
2010-04-29 19:23 dwheeler New Issue
2010-04-29 19:23 dwheeler Status New => Under Review
2010-04-29 19:23 dwheeler Assigned To => ajosey
2010-04-29 19:23 dwheeler Name => David A. Wheeler
2010-04-29 19:23 dwheeler Organization => IDA
2010-04-29 19:23 dwheeler Section => find
2010-04-29 19:23 dwheeler Page Number => 2740
2010-04-29 19:23 dwheeler Line Number => 89194
2011-07-06 23:42 Don Cragun Relationship added related to 0000244
2011-07-06 23:42 Don Cragun Relationship added related to 0000245
2011-07-06 23:54 Don Cragun Note Added: 0000882
2011-11-16 18:22 dwheeler Note Added: 0001020
2015-03-12 16:15 Don Cragun Relationship added has duplicate 0000903


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker