Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0000657 [1003.1(2008)/Issue 7] System Interfaces Objection Clarification Requested 2013-02-08 22:46 2023-11-21 10:35
Reporter philip-guenther View Status public  
Assigned To ajosey
Priority normal Resolution Accepted As Marked  
Status Applied  
Name Philip Guenther
Organization OpenBSD
User Reference
Section fmemopen
Page Number 867
Line Number 28775
Interp Status Approved
Final Accepted Text Note: 0006535
Summary 0000657: Conditions under which fmemopen() write a NUL to the buffer are insufficiently specified
Description As updated by XSH/TC1/D3/0149, the fmemopen() description now states:

 When a stream open for writing is flushed or closed, a null byte shall
 be written at the current position or at the end of the buffer, depending
 on the size of the contents. If a stream open for update is flushed or
 closed and the last write has advanced the current buffer size, a null
 byte shall be written at the end of the buffer if it fits.

The first sentence does not specify _how_ the choice of where to write the NUL depends on the size of the contents. Therefore, an implementation is presumably conforming if it writes the NUL to the current position whenever the size of the contents is an even number, and to the end of the buffer when it is odd, for example.

The second sentence only indicates that a NUL is (required to be) written if the last write advanced the current buffer size. So, a program that does a write which advanced the buffer size, then seeks back and does another write, and _then_ flushes or closes it cannot depend on a NUL to be written.
Desired Action Provide an actual specification for when and where the NUL is written in the "open for writing" case, presumably something about the minimum of the two offsets.

Change the second sentence to match what seems to be the glibc behavior:
 If a stream open for update is flushed or closed and the current buffer
 size has been advanced by a write since the stream was opened or flushed,
 a null byte shall be written at the end of the buffer if it fits.

Tags applied_after_i8d3, issue8
Attached Files

- Relationships
related to 0000396Closedajosey 1003.1(2008)/Issue 7 fmemopen vs. 'b' mode flag 
related to 0000456Appliedajosey 1003.1(2008)/Issue 7 mandate binary mode of fmemopen 
related to 0000818Closed 1003.1(2013)/Issue7+TC1 fmemopen should allow 0-size buffer 

-  Notes
(0001506)
philip-guenther (reporter)
2013-03-28 23:54

Another spot or two where glibc and the standard disagree:
    http://sourceware.org/bugzilla/show_bug.cgi?id=15298 [^]

I'm not sure why the text for these interfaces was put in the standard, given that the words apparently had little to do with the implementation that existed at the time. It's not clear they're converging either.

Might I suggest that if single-sourced APIs are going to be pulled into the standard, that that source be consulted with to verify what part of them can be considered stable for inclusion (vs left as unspecified or undefined behavior) and what level of "change control" they're willing to pass over? Is the submission done with the desire to improve the original implementation from the feedback (I think the *at() API family was like this) or is it pretty much cast in stone? If the final spec doesn't match the implementation, is the implementation going to change or will the spec need to change?

If the answer from the glibc people is "we'll update to match the requirements of the spec in a future release", all the better for POSIX. If their answer is "the spec is wrong on these points, but we'll work with you to bring the spec in line in a timely fashion with what we say the behavior is", that would be wonderful too, because then we would at least have a spec that matched reality.

If their answer is "sorry, we got here first and have too many apps depending on their current behavior and/or may make future changes to their behavior without consultation", then these interfaces should be deprecated from the spec with a note about why...so that the other existing implementations can go shred those pages in their copies of POSIX and then reverse engineer what glibc is doing.

Lacking visible word from the glibc maintainers on this, my assumption is that the latter is the case, but no one wants to actually say it.
(0006535)
geoffclare (manager)
2023-10-16 16:06
edited on: 2023-10-16 16:09

Interpretation response
------------------------
The standard is unclear on this issue, and no conformance distinction can be made between alternative implementations based on this. This is being referred to the sponsor.

Rationale:
-------------
None.

Notes to the Editor (not part of this interpretation):
-------------------------------------------------------

Page and line numbers are for Issue 8 draft 3.

Change "size" (when talking about the argument) to "max_size" throughout.
Change "size" (when talking about the last position in the buffer) to "end position" throughout.

Change on P961, L32741:
Open the stream for update (reading and writing). Truncate the buffer contents.
to:
Open the stream for update (reading and writing).

Change P961, L32744-32745 from:
If the mode argument includes 'b', then the stream shall be in binary mode; otherwise the stream shall be in text mode.
to:
If the mode argument begins with 'w' and max_size is not zero, the buffer contents shall be truncated by writing a null byte at the beginning. If the mode argument includes 'b', the results are implementation-defined.

Change on P961, L32758-32762:
The stream shall also maintain the size of the current buffer contents; use of fseek() or fseeko() on the stream with SEEK_END shall seek relative to this size. If mode starts with 'r' or includes 'b', the size shall be set to the value given by the size argument and shall not change. Otherwise, the stream is in text mode and writable, and the size shall be variable; for modes w and w+ the initial size shall be zero and for modes a and a+ the initial size shall be:
to:
The stream shall also maintain the end position of the current buffer contents; use of fseek() or fseeko() on the stream with SEEK_END shall seek relative to this end position. If mode starts with 'r' the end position shall be set to the value given by the max_size argument and shall not change. Otherwise, the stream is writable and the end position shall be variable; for modes w and w+ the initial end position shall be zero and for modes a and a+ the initial end position shall be:

Change on P962, L32775-32776:
When a stream open for writing in text mode is flushed or closed, a null byte shall be written at the current position or at the end of the buffer, depending on the size of the contents. If a stream open for update in text mode is flushed or closed and the last write has advanced the current buffer size, a null byte shall be written at the end of the buffer if it fits. If a stream is opened in binary mode, no additional null byte shall be written.
to:
When a stream open for update (the mode argument includes '+') or for writing only is successfully written and the write advances the current buffer end position, a null byte shall be written at the new buffer end position if it fits.

Change on P963, L32822-32828:
Unlike fopen(), where a 'b' in the mode argument is required to have no effect, fmemopen() distinguishes between text and binary modes. Text mode guarantees that the underlying memory will always be null terminated after any write operation, and tracks the growth of the largest position written to up to that point; while binary mode only modifies the underlying buffer according to direct writes, while seeking relative to the full buffer size. The combination of append and binary modes is not commonly used, since any attempt to write to such a stream will necessarily fail because the stream does not dynamically grow beyond the initial size.
to:
Implementations differ as regards how a 'b' in the mode argument affects the behavior. For some the 'b' has no effect, as is required for fopen(); others distinguish between text and binary modes.

Note that buf will not be null terminated if max_size bytes are written to the memory stream. Applications wanting to guarantee that the buffer will be null terminated need to call fmemopen() with max_size set to one byte smaller than the actual size of buf and set buf[max_size] to a null byte.


(0006536)
philip-guenther (reporter)
2023-10-16 18:29

Was any sort of buy-in from glibc maintainers obtained?
Has anyone verified that the updated wording match the current glibc behavior?

Put another way: is there any reason to be believe that the wording here is a useful reference vs just reverse engineering glibc?
(0006539)
Don Cragun (manager)
2023-10-16 22:15
edited on: 2023-10-16 22:21

re: Note: 0006536
Yes,. Except for the change of making mode "w" truncate the buffer (as well as "w+". Looking at the glibc code we found that the original implementation of fmemopen() did truncate in both cases, but that was changed along with a note saying the change was make to conform to POSIX. The original POSIX text was derived from the glibc man page for fmemopen() and it was not verified against the source at that time. We have asked the maintainers if they are willing to go back to the original implementation; if they are not we can change this back to the current behavior (or make it unspecified) before the interpretation is finalized. Note that the BSD implementation of fmemopen() does truncate the buffer for both "w" and "w+".

We made binary mode implementation-defined because glibc dropped binary mode, but BSD still supports it.

(0006541)
agadmin (administrator)
2023-10-17 05:02

Interpretation Proposed: 16 October 2023
(0006575)
agadmin (administrator)
2023-11-20 09:58

Interpretation approved: 20 November 2023

- Issue History
Date Modified Username Field Change
2013-02-08 22:46 philip-guenther New Issue
2013-02-08 22:46 philip-guenther Status New => Under Review
2013-02-08 22:46 philip-guenther Assigned To => ajosey
2013-02-08 22:46 philip-guenther Name => Philip Guenther
2013-02-08 22:46 philip-guenther Organization => OpenBSD
2013-02-08 22:46 philip-guenther Section => fmemopen
2013-02-08 22:46 philip-guenther Page Number => 867
2013-02-08 22:46 philip-guenther Line Number => 28775
2013-02-14 16:55 eblake Relationship added related to 0000396
2013-02-14 16:55 eblake Relationship added related to 0000456
2013-03-28 23:54 philip-guenther Note Added: 0001506
2014-01-30 04:06 eblake Relationship added related to 0000818
2023-10-16 16:06 geoffclare Note Added: 0006535
2023-10-16 16:07 geoffclare Interp Status => Pending
2023-10-16 16:07 geoffclare Final Accepted Text => Note: 0006535
2023-10-16 16:07 geoffclare Status Under Review => Interpretation Required
2023-10-16 16:07 geoffclare Resolution Open => Accepted As Marked
2023-10-16 16:08 geoffclare Note Edited: 0006535
2023-10-16 16:09 geoffclare Note Edited: 0006535
2023-10-16 16:09 geoffclare Tag Attached: issue8
2023-10-16 18:29 philip-guenther Note Added: 0006536
2023-10-16 22:15 Don Cragun Note Added: 0006539
2023-10-16 22:19 Don Cragun Note Edited: 0006539
2023-10-16 22:21 Don Cragun Note Edited: 0006539
2023-10-17 05:02 agadmin Interp Status Pending => Proposed
2023-10-17 05:02 agadmin Note Added: 0006541
2023-11-20 09:58 agadmin Interp Status Proposed => Approved
2023-11-20 09:58 agadmin Note Added: 0006575
2023-11-21 10:35 geoffclare Status Interpretation Required => Applied
2023-11-21 10:36 geoffclare Tag Attached: applied_after_i8d3


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker