Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0000697 [1003.1(2013)/Issue7+TC1] System Interfaces Comment Clarification Requested 2013-05-15 10:31 2021-05-07 15:06
Reporter steffen View Status public  
Assigned To
Priority normal Resolution Accepted As Marked  
Status Applied  
Name Steffen Nurpmeso
Organization
User Reference
Section none
Page Number none
Line Number none
Interp Status ---
Final Accepted Text Note: 0005332
Summary 0000697: Adding of a getdirentries() function
Description POSIX systems don't follow the Unix paradigm "everything is a file" for directories in that it is impossible to simply open(2) a directory and read(2) the descriptor to get access to the list of direntries therein.

In the current POSIX standard there is no way to gain access to directory entries but by using the completely intransparent (in respect to internal resource usage; though today there is now fdopendir(3), but still) and unpredictable ("directory status snapshot") opendir(3) / readdir(3) / closedir(3) family of functions.

On the other hand all (to the best of my knowledge) POSIX systems support some kind of getdents(2), getdirentries(2) or getdirent(2) system call (with, despite their names, almost identical semantics).
This functionality can be used to fully control reading of directory entries, at least in respect to resource usage, and including the possibility to perform bulk reads. A publically accessible example would be [1].

[1] http://code.google.com/p/plan9port/source/browse/src/lib9/dirread.c?r=6daaa8f20a12cc5eda13b0e13f293a4cd4174729 [^]
Desired Action Add a getdirentries(2) function to the standard, to catch up handling of directories with handling of files (e.g., open(2) / getdirentries(2) / close(2) <=> open(2) / read(2) / close(2)).
Or nay.
Tags issue8
Attached Files

- Relationships
related to 0000696Applied either NAME_MAX shouldn't be optional, or readdir_r() needs clarification 

-  Notes
(0001607)
jilles (reporter)
2013-05-15 22:08

With common implementation methods, all state for objects referenced by file descriptors has to be kept in the kernel (file descriptors may persist across execve() which replaces the userspace memory and may be sent across sockets). I think this should only be forced when there is a benefit to it. I don't see such a benefit here. The getdirentries() function does not make directory file descriptors more interchangeable with other file descriptors because it is specific to directories.

A concrete example of state for a directory stream occurs with stacked union filesystems. The list of directory entries in the union consists of all directory entries in the upper layer and those directory entries in the lower layer that do not match an entry in the upper layer by name. A possible implementation strategy (used in FreeBSD) is to remove the duplicates in opendir() and readdir(). This avoids storing the entire upper directory in the kernel, or reading it repeatedly whenever a block of the lower directory is read. However, an application that calls getdirentries() directly may see duplicates.

A simple solution is to call union filesystems non-compliant but this does not help the real world where people use them and expect applications to cope.
(0001608)
steffen (reporter)
2013-05-16 10:22

Hmm. But this concrete example shows the subtleties of filesystems that are layed upon each other rather than being an objection against getdirentries(2). In fact this is a bit of a pity because i didn't post a function text yesterday and wanted to make it up and use the FreeBSD version of getdirentries(2) since it would allow the most flexible definition that should work out-of-the-box on all systems (despite using off_t not long etc.; and stating that the off_t value is transparent to users).

But when i look at opendir.c (readdir.c) of FreeBSD 10.0 (last modified by your commit b4ce52f6), then it seems that you talk about features at the very first implementation iterations, just fixed so that it at least (?) works.
You use the FreeBSD getdirentries(2) (that luckily exists :) to read in all the entries in case of MNT_UNION or f_fstypename=="unionfs" and loop over that "atomic" snapshot to modify the entries to fixup things that possibly should better be done in the kernel.

Since, well, the current code doesn't even affect readdir(3); i.e., the snapshot from opendir(3) will survive a possibly infinite amount of rewinddir(3) / readdir(3) iterations, without ever being updated!
So, for this to be sane, you need at least some kind of "modification-counter-stamp" interaction with the kernel; that is, imho -- POSIX doesn't seem to state anything about the "freshness" of a DIR* directory stream, today.
... And i don't even know how the stack behaves if the surviving duplicate is removed / moved / whatever, whereas the removed one remains. I.e., it's anyway a snapshot, but it seems to me that the kernel / the union FS must be capable of dealing with it.
There are operating systems which removed all kind of stacked filesystems due to the race that are implied by such implementations. (I don't use them, but could not live without a singly-stacked mount_nullfs(8).)
(0001629)
geoffclare (manager)
2013-05-30 15:57

In the May 30th teleconference we agreed that a function of this type would be a good addition to the standard. New interfaces require a sponsor, but it is possible that the Base Working Group might be willing to do that.

It was felt that none of the three existing functions is a perfect fit for standardization; the favored approach would be to add a new function that has the best features chosen from the three.
(0004947)
geoffclare (manager)
2020-08-28 08:21
edited on: 2020-10-08 16:32

In the Oct 5th teleconference the following proposed changes were considered ready to go into a formal document to be submitted for The Open Group company review.

(Note that the first <dirent.h> change removes XSI shading from d_ino in struct dirent. This shading is already questionable because the readdir() page states an unshaded requirement about how d_ino is set.)
 
Page and line numbers are for the 2016/2018 edition.
 
On page 231 line 7773 section <dirent.h>, change:
It shall also define the structure dirent which shall include the following members:
[XSI]<tt>ino_t  d_ino</tt>       File serial number.[/XSI]
     <tt>char   d_name[]</tt>    Filename string of entry.

[XSI]The <dirent.h> header shall define the ino_t type as described in <sys/types.h>.[/XSI]
 
The array d_name is of unspecified size, but shall contain a filename of at most {NAME_MAX} bytes followed by a terminating null byte.
to:
It shall also define the structure dirent which shall include the following members:
<tt>ino_t  d_ino</tt>       File serial number.
<tt>char   d_name[]</tt>    Filename string of this entry.

and the structure posix_dent which shall include the following members:
<tt>ino_t          d_ino</tt>      File serial number.
<tt>reclen_t       d_reclen</tt>   Length of this entry, including
                                   trailing padding if necessary. See
                                   posix_getdents().
<tt>unsigned char  d_type</tt>     File type or unknown-file-type indication.
<tt>char           d_name[]</tt>   Filename string of this entry.

The array d_name in each of these structures is of unspecified size, but shall contain a filename of at most {NAME_MAX} bytes followed by a terminating null byte.
 
The <dirent.h> header shall define the ino_t, reclen_t, size_t, and ssize_t types as described in <sys/types.h>.
 
The <dirent.h> header shall define the following symbolic constants for the file types and unknown-file-type indicator returned in the d_type member of the posix_dent structure. The values shall be distinct and shall be suitable for use in #if preprocessing directives:
 
DT_BLK
Block special.

DT_CHR
Character special.

DT_DIR
Directory.

DT_FIFO
FIFO special.

DT_LNK
Symbolic link.

DT_REG
Regular.

DT_SOCK
Socket.

DT_UNKNOWN
Unknown file type.

The implementation may implement message queues, semaphores, shared memory objects [TYM]or typed memory objects[/TYM] as distinct file types. The following macros shall be provided to represent these types. The values shall be distinct from each other and from the above symbolic constants beginning with DT_, except when a distinct file type is not implemented, in which case the correponding constant shall have a value that is never returned in d_type by posix_getdents(). The values shall be suitable for use in #if preprocessing directives:
 
DT_MQ
Message queue.

DT_SEM
Semaphore.

DT_SHM
Shared memory object.

[TYM]DT_TMO
Typed memory object.[/TYM]

On page 231 line 7784 section <dirent.h>, add:
ssize_t posix_getdents(int, void *, size_t, int);

On page 231 line 7804 section <dirent.h>, add new paragraphs to RATIONALE:
The posix_dent structure was based on existing structures used by traditional getdents() functions, but the name was changed because the existing structures differed in name and in their members. Some used the dirent structure but this is not required to include a d_type member, which is the main advantage of using posix_getdents() over readdir(). The d_reclen member was included, even though some implementations return fixed-length entries and therefore do not need it, as almost all existing code that used getdents() used d_reclen to iterate through the returned entries. Implementations that return fixed-length entries can simply set d_reclen to that length in posix_getdents(). The type reclen_t for d_reclen was introduced, instead of using unsigned short, so as not to create a requirement that {NAME_MAX} cannot be greater than (a value somewhat smaller than) {SHRT_MAX}.
 
Implementations are encouraged to define a DT_FORCE_TYPE symbolic constant for use in the flags argument to posix_getdents(). See the RATIONALE for [xref to posix_getdents()].

On page 232 line 7810-7812 section <dirent.h>, change:
The array of char d_name is not a fixed size. Implementations may need to declare struct dirent with an array size for d_name of 1, but the actual number of bytes provided matches (or only slightly exceeds) the length of the filename string.
to:
The array of char d_name cannot be assumed to have a fixed size. Implementations may define the d_name array in the dirent and posix_dent structures to have size 1, or size greater than {NAME_MAX}, or use a flexible array member, but in all cases the actual number of characters used for d_name is at least the length of the filename string including the terminating NUL byte.

On page 232 line 7814 section <dirent.h>, change FUTURE DIRECTIONS from:
None.
to:
A future version of this standard may add a DT_FORCE_TYPE symbolic constant for use as described in the RATIONALE for [xref to posix_getdents()].

On page 232 line 7817 section <dirent.h>, add posix_getdents() to SEE ALSO.
 
On page 403 line 13687 section <sys/types.h>, add:
reclen_t Used for directory entry lengths.

On page 403 line 13723 section <sys/types.h>, change:
..., and ino_t shall be defined as unsigned integer types.
to:
..., reclen_t, and ino_t shall be defined as unsigned integer types.

On page 475 line 16318 section 2.2.2 The Name Space, change:
d_
to:
d_, DT_

On page 519 line 18149 section 2.9.5.2 Cancellation Points, add posix_getdents() to the "may" table.
 
On page 548 line 19289 section 2.12.1 Defined Types, add:
reclen_t Unsigned integer type used for directory entry lengths.

On page 841 line 28511 section fdopendir(), add posix_getdents() to SEE ALSO.
 
On page 1444 insert a new man page for posix_getdents():
 
NAME

posix_getdents — read directory entries

SYNOPSIS
#include <dirent.h>
 
ssize_t posix_getdents(int fildes, void *buf, size_t nbyte, int flags);

DESCRIPTION
The posix_getdents() function shall attempt to read directory entries from the directory associated with the open file descriptor fildes and shall place information about the directory entries and the files they refer to in posix_dent structures in the buffer pointed to by buf, up to a maximum of nbyte bytes. The number of posix_dent structures populated in buf may be fewer than the number that will fit in nbyte bytes, but shall be at least one if nbyte is greater than the size of the posix_dent structure plus {NAME_MAX} and fildes is not currently at end-of-file.
 
The application shall ensure that buf is aligned suitably to point to a posix_dent structure. The alignment needed shall not be more restrictive than the alignment provided by malloc(). Strictly conforming applications shall ensure that the value of flags is zero; other applications can set it to a value constructed by a bitwise-inclusive OR of implementation-defined bitwise-distinct flag values.
 
Each posix_dent structure returned in buf shall be located at an address that satisfies the implementation's alignment requirements for the posix_dent structure and shall be populated as follows:
  • The value of the d_ino member shall be set to the file serial number of the file named by the d_name member.

  • The value of the d_reclen member shall be set to the number of bytes occupied by this entry in buf, including any padding bytes needed before the next entry, if any. If this is the last entry in buf, d_reclen shall include any padding bytes needed to make the address of this entry plus d_reclen bytes satisfy the alignment requirements for the posix_dent structure.

  • The value of the d_type member shall be set to indicate the file type of the named file, if the file type can be determined without needing to use the file serial number to obtain the file's metadata; otherwise it may be set to DT_UNKNOWN.  If the file type is determined and it is one of the file types defined in this standard, the value of d_type shall be DT_BLK, DT_CHR, DT_DIR, DT_FIFO, DT_LNK, DT_REG, DT_SOCK, DT_MQ, DT_SEM, DT_SHM, [TYM]or DT_TMO[/TYM] (see [xref to <dirent.h>]). If it is determined but is not a standard file type, the value of d_type shall not equal any of those listed here.

  • The d_name member shall be a filename string, and (if not dot or dot-dot) shall contain the same byte sequence as the last pathname component of the string used to create the directory entry, plus the terminating <NUL> byte.

If the d_name member names a symbolic link, the values of the d_ino and d_type members shall be set to the values for the symbolic link itself.
 
The posix_getdents() function shall start reading at the current file offset in the open file description associated with fildes. On successful return, the file offset shall be incremented to point to the directory entry immediately following the last entry whose information was returned in buf, or to point to end-of-file if there are no more directory entries. On failure, the value of the file offset is unspecified. The current file offset can be set and retrieved using lseek() on the open file description associated with fildes. The behavior is unspecified if lseek() is used to set the file offset to a value other than zero or a value returned by a previous call to lseek() on the same open file description.
 
The posix_getdents() function shall not return directory entries containing empty names. If entries for dot or dot-dot exist, a sequence of calls that reads from offset zero to end-of-file shall return one entry for dot and one entry for dot-dot; otherwise, they shall not be returned.
 
Upon successful completion, posix_getdents() shall mark for update the last data access timestamp of the directory.
 
If fildes is a file descriptor associated with a directory stream opened using fdopendir() or opendir(), the behavior is unspecified.

If posix_getdents() is called concurrently with an operation that adds, deletes, or modifies a directory entry, the results from posix_getdents() shall reflect either all of the effects of the concurrent operation or none of them. If a sequence of calls to posix_getdents() is made that reads from offset zero to end-of-file and a file is removed from or added to the directory between the first and last of those calls, whether the sequence of calls returns an entry for that file is unspecified.

RETURN VALUES
Upon successful completion, either a non-negative integer shall be returned indicating the number of bytes occupied by the posix_dent structures placed in buf or 0 shall be returned indicating the end of the directory was reached without any directory entries being placed in buf. Otherwise, -1 shall be returned and errno shall be set to indicate the error.

ERRORS
The posix_getdents() function shall fail if:
 
[EBADF]
The fildes argument is not a valid file descriptor open for reading.

[EINVAL]
The nbyte argument is not large enough to contain the information to be returned about the directory entry located at the current file offset.

[ENOENT]
The current file offset is not located at a valid directory entry.

[ENOTDIR]
The fildes argument is associated with a non-directory file.

[EOVERFLOW]
One of the values in a structure to be placed in buf cannot be represented correctly.

The posix_getdents() function may fail if:
 
[EIO]
A physical I/O error has occurred.

[ENOMEM]
Insufficient memory was available to fulfill the request.

EXAMPLES
This example function lists the files in a specified directory with their file serial number and file type. If the file type is not available from posix_getdents(), it is obtained using fstatat().
#include <dirent.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <unistd.h>
 
#define ENTBUFSIZ 10240
 
int list_dir(const char *dirnam)
{
    int fd = open(dirnam, O_RDONLY | O_DIRECTORY);
    if (fd == -1)
        return -1;
    
    char *buf = malloc(ENTBUFSIZ);
    if (buf == NULL)
    {
        close(fd);
        return -1;
    }

    ssize_t bytesinbuf;
    for(;;)
    {
        ssize_t nextent = 0;

        bytesinbuf = posix_getdents(fd, buf, ENTBUFSIZ, 0);
        if (bytesinbuf <= 0)
            break;
 
        do {
            const char *ftype;
            struct posix_dent *entp = (void *)&buf[nextent];
            if (entp->d_type == DT_UNKNOWN)
            {
                struct stat stbuf;
                if (fstatat(fd, entp->d_name, &stbuf, AT_SYMLINK_NOFOLLOW) == -1)
                    ftype = "?";
                else
                    ftype = S_ISBLK(stbuf.st_mode) ? "b" :
                         S_ISCHR(stbuf.st_mode) ? "c" :
                         S_ISDIR(stbuf.st_mode) ? "d" :
                         S_ISFIFO(stbuf.st_mode) ? "p" :
                         S_ISLNK(stbuf.st_mode) ? "l" :
                         S_ISREG(stbuf.st_mode) ? "r" :
                         S_ISSOCK(stbuf.st_mode) ? "s" :
                         S_TYPEISMQ(&stbuf) ? "mq" :
                         S_TYPEISSEM(&stbuf) ? "sem" :
                         S_TYPEISSHM(&stbuf) ? "shm" :
#ifdef S_TYPEISTMO
                         S_TYPEISTMO(&stbuf) ? "tmo" :
#endif
                         "?";
            }
            else
            {
                ftype = entp->d_type == DT_BLK ? "b" :
                        entp->d_type == DT_CHR ? "c" :
                        entp->d_type == DT_DIR ? "d" :
                        entp->d_type == DT_FIFO ? "p" :
                        entp->d_type == DT_LNK ? "l" :
                        entp->d_type == DT_REG ? "r" :
                        entp->d_type == DT_SOCK ? "s" :
                        entp->d_type == DT_MQ ? "mq" :
                        entp->d_type == DT_SEM ? "sem" :
                        entp->d_type == DT_SHM ? "shm" :
#ifdef DT_TMO
                        entp->d_type == DT_TMO ? "tmo" :
#endif
                        "?";
            }
 
            printf("%ld\t%s\t%s\n", (long)entp->d_ino, ftype, entp->d_name);
 
            nextent += entp->d_reclen;
 
        } while (nextent < bytesinbuf);
    }

    close(fd);
    free(buf);
    return bytesinbuf;
}

APPLICATION USAGE
If an array of posix_dent structures (which is only possible on implementations where d_name is not a flexible array member) is used to provide the storage for buf in order to satisfy the alignment requirement, it should be noted that the number of array elements used to size the array may bear little or no relation to the number of directory entries that can be stored in it. It is recommended that the number of elements is calculated from the desired size in bytes, for example:
#define DESIREDSIZE 10240
struct posix_dent buf[DESIREDSIZE / sizeof(struct posix_dent) + 1];
size_t nbyte = sizeof buf;

When posix_getdents() is called with a buf that is not type char *, it is important to note that d_reclen is a byte count and therefore any pointer arithmetic involved in calculating the start of the next entry needs to use a char * pointer.
 
On implementations where directory entries in a directory take up more space than the corresponding posix_dent structures in buf, a call to posix_getdents() may read nbyte bytes from the directory, resulting (in most cases) in the actual number of bytes placed in buf being less than nbyte.
 
One advantage of posix_getdents() is that it provides the file type of each directory entry (if available), whereas readdir() only does so on implementations that have the file type as a non-standard additional member of the dirent structure. Knowing the file type can greatly reduce the number of fstatat() calls that need to be made when traversing the file hierarchy.
 
Whether or not a file's type can be determined without needing to use the file serial number to obtain the file's metadata may vary across the different file system types supported by an implementation. Therefore applications should not assume that if d_type contains known file types (i.e. not DT_UNKNOWN) for entries in a given directory then it will also contain known file types for entries in subdirectories of that directory or in its parent.

Since the d_reclen value for the last entry in buf includes padding to satisfy alignment requirements, applications can grow the buffer and call posix_getdents() again to append to it without needing to perform an alignment calculation.

RATIONALE
The posix_getdents() function was derived from existing getdents functions but the name was changed because the existing getdents functions differed in various ways, in particular the type of the second argument (structure pointer or void *), the members of the populated structures, and the error numbers used for some conditions. The name change also provided an opportunity to add a flags argument to provide for future extensibility.
 
Implementations are encouraged to include support for a DT_FORCE_TYPE flag which, when that bit is set in flags, causes posix_getdents() to look up the file type if it can not be obtained from the directory entry. This will allow applications that need to know the file type of every directory entry to keep the cost of these lookups to the minimum needed to obtain the type at the file system level, without the additional overhead of making a call to fstatat() for every file (that has d_type equal to DT_UNKNOWN).

Some existing getdents() or similar functions return directory entry structures for deleted directory entries in buf, marked with a special value of one of the structure members to distinguish them from non-deleted entries. This behavior is not allowed for posix_getdents(), although the data from a deleted directory entry may be present in buf in the form of extra padding on the end of the previous entry.

FUTURE DIRECTIONS
A future version of this standard may add a DT_FORCE_TYPE flag as described in RATIONALE.

SEE ALSO
fdopendir(), lseek(), readdir()
 
XBD <dirent.h>

CHANGE HISTORY
First released in Issue 8.

On page 1781 line 57632 section readdir(), add posix_getdents() to SEE ALSO.
 
On page 3791 line 130084 section E.1, add posix_getdents() to the POSIX_FILE_SYSTEM subprofiling group.

(0004948)
shware_systems (reporter)
2020-08-28 17:07

Since NAME_MAX can vary for subdirs that reference file systems other than the root, '/' or "//", I think it is better to characterize the possible length of a posix_dent structure in terms of "as if fpathconf( fd, PC_NAMEMAX ) is used".

Since nbytes is size_t, and maybe should be ofs_t, the function return value should be of the same type.

I think there should be a note about the Return Value, that a value of 0 is possible on a first call, indicating an empty directory on a file system that does not store '.' and '..' entries.

I feel it should also be emphasized if the fd used to access the file is not opened with O_RDWR | O_EXCL only the first call to the interface may be successful. Any additional calls looking for a 0 return may return garbage data due to another thread modifying the directories entries. An alternative is to have buf typed as posix_dent**, which if NULL on entry indicates the interface should malloc() sufficient space for all the records at once.

I would prefer to see the prototype as varargs one, so any flags an implementation defines that require additional arguments have a place for them. For example, a DT_ONLY_TYPE or DT_NOT_TYPE flag would need an additional value to specify which DT_* file type to include or exclude, respectively, from the returned records.

As a wrapper for multiple readdir() calls, conceptually, some might like an additional prototype that uses a DIR * rather than a file descriptor, e.g. posix_fgetdents(DIR *fildes, ...).
(0004949)
kre (reporter)
2020-08-28 17:52

I suspect that the intent of all of this is good, but this one
phrase's wording (wrt the d_name field, it is used for that
field in both struct direct and struct posix_dent):

     but shall contain a filename of at most {NAME_MAX} bytes

is incomprehensible to me. I can read it as saying

     no file names longer than NAME_MAX bytes can ever occur in a d_name
or as
     the d_name field must be able to contain file names at least
     NAME_MAX bytes long

I suspect that the latter is most likely what was intended, but I really
don't know for sure - that one allows implementations to support filesystems
where directories might contain names longer than can be regularly passed to
system calls, for example, and generally allowing implementation extensions
is desirable, but the former allows an application to declare an array of
NAME_MAX+1 bytes and be sure that any d_name entry will fit.

This phrase ought to be reworded to make it clear what is intended there.
Since it is talking about a variable length array, better phrasing would
probably concentrate on what bounds exist for the size of that array, rather
than the length of what might be stored within it.

And while I'm here, I don't think we need to be providing C tutorials,
so I'd drop all the stuff about arrays of posix_dent structures
completely - attempting to write a program using such a thing would be
folly (as it is for any variable sized objects, arrays (in C at least)
require equal sized objects so addresses can be calculated by arithmetic on
the index - the technique to use to handle several dirent or posix_dent
(or anything similar) that are to be combined into a single linear data
struct is the sequence (not a C data type...) where each item follows on
after the one before it, and the only way (without an ancillary data struct)
to locate the N'th item is to examine each of the preceding N-1 in order.
(Network packets are full of this kind of object). But none of this needs
to be in the standard - how to write code is the application programmer's
problem, and text books or "how to" or xxx for dummies, or whatever are
the places where programming techniques belong.
(0004952)
philip-guenther (reporter)
2020-08-28 22:42

I believe all the historical versions of this interface can return entries that did not exist (any longer) and that the caller is expected to skip. At least on the BSDs this has been with entries with d_ino == 0. I believe this had a positive side effect of making readdir() work "more often" after a seekdir() to the position of a file deleted after the telldir(), as the position could remain valid despite the deletion. I take it that this new interface intentionally does not support this and that the implementation is required to only expose valid entries.
(0004953)
philip-guenther (reporter)
2020-08-28 22:52

I think the unspecified nature of the d_name member in the new posix_dent makes writing portable software more difficult while providing only minimal benefit to programs that don't care. I would support requiring it to be a flexible array member and thus eliminating the error of declaring an array and trying to walk it via indexing instead of by advancing a char pointer by d_reclen.

(The unspecified nature of d_name in struct dirent made using readdir_r() unnecessarily painful; thankfully the push to use that API is dead with the wide recognition that readdir() is thread safe on a per-DIR basis, but the memory of that pain lingers.)
(0004956)
shware_systems (reporter)
2020-08-29 11:34
edited on: 2020-08-29 11:41

As written the intent is more the former. The function attempts to pack in to buf as many records as will fit, only adding padding between records to satisfy alignment requirements for accessing the d_inode field of a following record. If ino_t is 64 bits, the d_reclen field will probably be a multiple of 8 to reflect this. In this scenario d_reclen, and the length of the record, may be as low as 16.

To ensure success for the corner case where a filename is the maximum length, the buffer passed in should be at least large enough to hold the maximum size of one of these records, a malloc( sizeof posix_dent + fpathconf(fildes, PC_NAME_MAX) + sizeof ino_t ) should suffice. Normally the buffer will be much larger, however, based on the size reported by a fstatat(fildes, ".") call.

(0004957)
shware_systems (reporter)
2020-08-29 13:07
edited on: 2020-08-29 13:11

Re: 4952
Packing of records does not imply record removal, unless flags are defined for this, such as my suggestions. So, seeing inode values of zero isn't precluded.

The only case I see where this might not hold is if a file system does not report an entry as part of the "file" because it has failed some file system specific validity test. Something like this would be invisible to the implementation, however.

(0004958)
philip-guenther (reporter)
2020-08-30 23:06

The proposed text includes:
    The d_name member shall be a filename string, and (if not dot or dot-dot)
    shall contain the same byte sequence as the last pathname component of the
    string used to create the directory entry, plus the terminating <NUL> byte.

That would seem to require that all returned entries correspond to filenames that existed in the directory at _some_ point in time. However, I see no requirement that posix_getdents() only return currently existing files, nor any requirement that non-existing files be flagged in any way. The former at least seems like an oversight that should be corrected.

I interpret the definition of readdir(), including its description of the DIR type:
    The type DIR, which is defined in the <dirent.h> header, represents a
    directory stream, which is an ordered sequence of all the directory
    entries in a particular directory.
and later text as effectively requiring that readdir() may not return an entry for a file that did not exist at some point between when opendir() or rewinddir() was last called and when readdir() returned it.

Since posix_getdents() is all about bulk transfer with no buffering, I think its description should be updated EITHER to require that
a) all returned entries must have existed at some point during the call, OR
b) all returned entries with d_ino != 0 must have existed at some point during
   the call and specify that entries with d_ino == 0 may have d_name[0] == '\0'

Specifying (b) is more in line with historical BSD behavior, but does require additional application logic, so I'm sympathetic to a view that specifying (a) is the cleaner choice.
(0004959)
philip-guenther (reporter)
2020-08-30 23:11

Oh, and if (b) is chosen and d_ino==0 means an entry that should be skipped, then XBD's entry for "File Serial Number" should be updated to indicate that no file will have a file serial number of zero. (It appears that the standard does not currently reserve that value.)
(0004960)
shware_systems (reporter)
2020-08-30 23:44

For file systems that relate inode values to start sector index, inode 0 is the superblock for that volume, and entirely valid. Saying it is illegal I don't see happening; that code is simply buggy, conceptually. The safer value to use is (ino_t)-1, if it is to be overloaded that way, but defining a DT_DELETED value for d_type is even safer.
(0005016)
geoffclare (manager)
2020-10-02 09:11

Note: 0004947 has been updated with changes agreed in the Oct 1st teleconference.

Notable changes are:

Return type of posix_getdents() is now ssize_t (to match read()) and <dirent.h> is required to define that type.

The last entry in buf must have a d_reclen that includes any needed padding for alignment. (So applications can grow the buffer and append to it without needing to do an alignment calculation.)

The wording of the condition for d_type not being DT_UNKNOWN is now "if the file type can be determined without needing to use the file serial number to obtain the file's metadata".

Added a paragraph about concurrent file operations.

Knock-on effects of d_name[] possibly being a flexible array member, where the size of the structure is mentioned and in the first para of APPLICATION USAGE.

Added a paragraph to RATIONALE about posix_getdents() not being allowed to return directory entry structures for deleted directory entries.
(0005022)
dennisw (reporter)
2020-10-03 12:53

I think the specification should make it explicit under which circumstances (if any) the application can rely on posix_getdents() to return the current state of the directory.
Right now it only says that it is unspecified when a sequence of posix_getdents() calls reads the entire directory.
Is posix_getdents() guaranteed to return the current state after an lseek() to the beginning?

Also in the second last paragraph of the DESCRIPTION, there should be no parentheses after fildes.
(0005023)
geoffclare (manager)
2020-10-05 08:11

Re Note: 0005022, the condition in the last paragraph of DESCRIPTION, "If a sequence of calls to posix_getdents() is made that reads the entire directory" is intended to cover the case of lseek() to the beginning then reading to end-of-file. (It was based on the readdir() page saying "after the most recent call to opendir() or rewinddir()".) Perhaps this should be clarified by changing it to:
If a sequence of calls to posix_getdents() is made that reads from offset 0 to end-of-file, ...

Thanks for catching the extraneous parentheses after fildes - I have removed them from Note: 0004947.
(0005032)
geoffclare (manager)
2020-10-07 08:47

Note: 0004947 has been updated with changes agreed in the Oct 5th teleconference, changing two occurrences of "that reads the entire directory" to "that reads from offset zero to end-of-file".
(0005033)
dennisw (reporter)
2020-10-07 11:29

I noticed another issue:
The EOVERFLOW error requires the function to fail if the result would be larger than INT_MAX. But the return type was changed to ssize_t, so this error needs to be updated.
(0005034)
geoffclare (manager)
2020-10-07 13:23
edited on: 2020-10-08 16:33

Re Note: 0005033 This set me thinking about why that part of the EOVERFLOW error is there at all. There is no equivalent EOVERFLOW for read(), nor should there be.

I think if posix_getdents() reaches the point where adding another entry to the buffer would make its length unrepresentable in the return type, then it should return (successfully) at that point. I.e. the equivalent of a short read().

So I think the EOVERFLOW error should be changed to just:
One of the values in a structure to be placed in buf cannot be represented correctly.


Update: This change was agreed in the Oct 8th teleconference and has been applied to Note: 0004947.

(0005036)
shware_systems (reporter)
2020-10-07 14:28

That is an error in read(), and fread() as well; that these should have that case also as a may fail type. I remember this being discussed for some other interface where an argument was size_t or ssize_t but return was int. If an implementation has a SSIZE_MAX that needs a long long int to represent it, a value larger than INT_MAX is a plausible desired return value but is not representable. Similar holds when SIZE_MAX is larger than SSIZE_MAX. This was a non-issue when no platform had more than 2gb RAM, for int as a 32 bit type, but is one now.
(0005060)
geoffclare (manager)
2020-10-23 14:38

The posix_getdents() addition has been made in the Issue8NewAPIs branch in gitlab, based on Note: 0004947.
(0005332)
geoffclare (manager)
2021-04-29 15:11

Make the changes from "Additional APIs for Issue 8, Part 1" (Austin/1110).

- Issue History
Date Modified Username Field Change
2013-05-15 10:31 steffen New Issue
2013-05-15 10:31 steffen Name => Steffen Nurpmeso
2013-05-15 10:31 steffen Section => none
2013-05-15 10:31 steffen Page Number => none
2013-05-15 10:31 steffen Line Number => none
2013-05-15 22:08 jilles Note Added: 0001607
2013-05-16 10:22 steffen Note Added: 0001608
2013-05-30 15:37 eblake Relationship added related to 0000696
2013-05-30 15:57 geoffclare Note Added: 0001629
2014-03-30 00:33 sstewartgallus Issue Monitored: sstewartgallus
2020-08-28 08:21 geoffclare Note Added: 0004947
2020-08-28 08:27 geoffclare Note Edited: 0004947
2020-08-28 17:07 shware_systems Note Added: 0004948
2020-08-28 17:52 kre Note Added: 0004949
2020-08-28 22:42 philip-guenther Note Added: 0004952
2020-08-28 22:52 philip-guenther Note Added: 0004953
2020-08-29 11:34 shware_systems Note Added: 0004955
2020-08-29 11:34 shware_systems Note Added: 0004956
2020-08-29 11:41 shware_systems Note Edited: 0004956
2020-08-29 11:44 shware_systems Note Deleted: 0004955
2020-08-29 13:07 shware_systems Note Added: 0004957
2020-08-29 13:11 shware_systems Note Edited: 0004957
2020-08-30 23:06 philip-guenther Note Added: 0004958
2020-08-30 23:11 philip-guenther Note Added: 0004959
2020-08-30 23:44 shware_systems Note Added: 0004960
2020-10-02 09:00 geoffclare Note Edited: 0004947
2020-10-02 09:11 geoffclare Note Added: 0005016
2020-10-03 12:53 dennisw Note Added: 0005022
2020-10-05 08:03 geoffclare Note Edited: 0004947
2020-10-05 08:11 geoffclare Note Added: 0005023
2020-10-07 08:44 geoffclare Note Edited: 0004947
2020-10-07 08:47 geoffclare Note Added: 0005032
2020-10-07 11:29 dennisw Note Added: 0005033
2020-10-07 13:23 geoffclare Note Added: 0005034
2020-10-07 14:28 shware_systems Note Added: 0005036
2020-10-08 16:32 geoffclare Note Edited: 0004947
2020-10-08 16:33 geoffclare Note Edited: 0005034
2020-10-23 14:38 geoffclare Note Added: 0005060
2021-04-29 15:11 geoffclare Note Added: 0005332
2021-04-29 15:11 geoffclare Interp Status => ---
2021-04-29 15:11 geoffclare Final Accepted Text => Note: 0005332
2021-04-29 15:11 geoffclare Status New => Resolved
2021-04-29 15:11 geoffclare Resolution Open => Accepted As Marked
2021-04-29 15:12 geoffclare Tag Attached: issue8
2021-05-07 15:06 geoffclare Status Resolved => Applied


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker