View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0001927 | 1003.1(2008)/Issue 7 | Shell and Utilities | public | 2025-06-01 01:18 | 2025-06-11 16:49 |
| Reporter | dwheeler | Assigned To | ajosey | ||
| Priority | normal | Severity | Editorial | Type | Clarification Requested |
| Status | Under Review | Resolution | Open | ||
| Name | David A. Wheeler | ||||
| Organization | The Linux Foundation | ||||
| User Reference | Utilities | ||||
| Section | Utilities | ||||
| Page Number | NA | ||||
| Line Number | NA | ||||
| Interp Status | |||||
| Final Accepted Text | |||||
| Summary | 0001927: Add sponge utility | ||||
| Description | When using only POSIX utilities, it's unnecessarily difficult to read a file, process it, and replace that file with those processed results. Novices who try to do the following will experience sadness: grep -E '^X' < foo > foo You can *sort-of* do this by redirecting the results to a temporary file, and then moving the temporary file onto the old one. However, this sequence fails to maintain the permissions of the original file. Doing that requires MORE steps. This is a common need that should have an easy solution. Real-world implementations of "sed" include a flag -i (-in-place) specifically to do this. This only works when using sed (obviously). Unfortunately, the flag has incompatible interfaces on different systems, as noted in https://austingroupbugs.net/view.php?id=530 . It'd be worth implementing a common interface for sed everywhere so this widely-used in-place functionality had a standard way to invoke it, but that would take a while to get implemented widely and again, it would *only* work with sed. The "sponge" utility has been used for years to do this in practice. It's simple to describe, simple to implement, widely available, and provides general functionality that works in any pipeline (not just for sed). I believe that standards should always strongly consider codifying existing practice. That's what I'm proposing here. | ||||
| Desired Action | NAME sponge — soak up standard input and write it to a file SYNOPSIS sponge [-a] target DESCRIPTION sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before opening the output file. This allows constricting pipelines that read from and write to the same file. sponge preserves the permissions of the output file if it already exists. When possible, sponge creates or updates the output file atomically by creating a separate temp file in TMPDIR and then renaming that temp file into place. This cannot be done if TMPDIR is not in the same filesystem. If the output file is a special file or symlink, the data will be written to it, non-atomically. If no file is specified, sponge outputs to stdout. OPTIONS -a Replace the file with a new file that contains the file's original content, with the standard input appended to it. This is done atomically when possible. OPERANDS The following operands shall be supported: target file A pathname of the file where the results will be placed. STDIN The standard input shall be used to read inputs. ENVIRONMENT VARIABLES The following environment variables shall affect the execution of cp: TMPDIR Temporary directory for use in storing results while sponge is reading data from standard input. ASYNCHRONOUS EVENTS Default. STDOUT Not used. STDERR Diagnostic messages such as an inability to duplicate permissions or an inability to modify the target file. OUTPUT FILES The output files may be of any type. EXTENDED DESCRIPTION None. EXIT STATUS The following exit values shall be returned: 0 Target file was successfully updated. >0 An error occurred. CONSEQUENCES OF ERRORS If sponge terminates before it reads the end of standard input, there may be a leftover temporary file in TMPDIR. The following sections are informative. APPLICATION USAGE EXAMPLES # Modify file foo so it only includes lines beginning with "X": grep -E '^X' < foo | sponge foo RATIONALE This utility makes it much easier to create pipelines that use some file as an input and write their results to that file, all while preserving the permissions of that file. It's possible to write to a temporary file, then mv it to the original name, but this common approach fails to preserve the original file's permissions. The goal of this utility is to make it easy to do a common action. The following example doesn't work as a novice might expect, because opening the file "foo" for writing eliminates the data in "foo" for reading: grep -E '^X' < foo > foo FUTURE DIRECTIONS A future option might be added to truncate and copy contents into the target file so that the target's inode does not change. A future option might be added to create a new file with the same permissions as an existing file (since this is the first step of implementing this utility). SEE ALSO cp, mv, sed XBD 4.7 File Access Permissions, 8. Environment Variables, 12.2 Utility Syntax Guidelines XSH open(), unlink() End of informative text. | ||||
| Tags | No tags attached. | ||||
|
|
Note: I'm sure that the proposed text above for the new utility could be improved. However, the first step is to decide whether or not to include the sponge utility at all, and the second step is to refine its definition to be appropriate for POSIX. I tried to do both, but I've focused on step 1. There's no point in refining the proposal if it will be rejected no matter what :-). However, I think "use a pipeline to update in place" is such a common use case that it's good to make it *easy*. There have been proposals for "in-place" commands that run a string passed in, but don't know of any that are fully implemented or widely used. Part of the problem is that they don't they require complex escape mechanisms for widely-used characters. As a result they require complex escaping to use that is hard to reason about, the mixing of data and command easily leads to vulnerabilities, and it's hard to create tools to do things like provide proper syntactic highlighting. In contrast, "sponge" has been used for a long time by many, it's easy to reason about, and it doesn't require complex reasoning . Instead, just write your pipeline as usual, and pipe to sponge as the final command. No separate mv command needed, and you know that the permissions are the same (if it's possible to do that). Many places highlight sponge as a useful utility. Here are some: https://www.reddit.com/r/linux/comments/18gifb8/usefull_cli_tools/ https://www.putorius.net/linux-sponge-soak-up-standard-input-and-write-to-a-file.html https://www.youtube.com/watch?v=9RXkZpmBDj0 (at time 15:10) https://rentes.github.io/unix/utilities/2015/07/27/moreutils-package/ |
