Austin Group Defect Tracker

Aardvark Mark III


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0001134 [1003.1(2016)/Issue7+TC2] System Interfaces Editorial Enhancement Request 2017-03-31 07:06 2019-11-28 18:16
Reporter nmav View Status public  
Assigned To
Priority normal Resolution Open  
Status New  
Name Nikos Mavrogiannopoulos
Organization Red Hat
User Reference
Section getentropy
Page Number 0
Line Number 0
Interp Status ---
Final Accepted Text
Summary 0001134: Add getentropy interface
Description Software often requires a source of pseudo random data, that provides high quality unpredictable random data, suitable for cryptographic use.
Various operating systems provide interfaces via device files, such as
/dev/urandom, and /dev/random, as well as system calls like getrandom(),
getentropy() and others. However, there is no common convention in
systems resulting to much of compatibility code.

The proposed interface below is based on the getentropy description as entered
the GLIBC in 2016, however, it existed in OpenBSD and other systems
since long time.


SYNOPSIS
       #include <unistd.h>

       int getentropy(void *buffer, size_t length);

DESCRIPTION
The getentropy() function writes length bytes of data starting at the location pointed to by buffer. The output is unpredictable high quality random data, generated by a cryptographically secure pseudo-random number generator.
The maximum permitted value for the length argument is 256.

A successful call to getentropy() always provides the requested number of bytes of entropy.

RETURN VALUE
       On success, this function returns zero, On error, -1 is
returned, and errno is set appropriately.

ERRORS
       EFAULT Part or all of the buffer specified by buffer and length
is not in valid addressable memory.

       EIO Length is greater than 256.

       ENOSYS The system does not provide the necessary system calls for this operation.

NOTES

The getentropy () call is not a cancellation point.
Desired Action Add the getentropy interface(). This interface is related to the withdrawn #859.
Tags No tags attached.
Attached Files

- Relationships
related to 0000859Closed 1003.1(2013)/Issue7+TC1 Add posix_random family of interfaces 

-  Notes
(0003657)
EdSchouten (reporter)
2017-04-01 06:25

Though I'm still a fan of having an interface adopted that is guaranteed to work (like arc4random_buf()), any progress in this area is good.

Instead of hardcoding the 256-byte upper limit, would it make sense to add a definition 'GETENTROPY_MAX', for which we state that it has to be at least 256 bytes in size? Leaving it undefined either means there is no limit or it's compile-time undefined. We should probably also add a sysconf() option to query the run-time limit.

The description of EFAULT should probably be removed, as we don't document it for other calls specifically.

EIO is inconsistent with the rest of the functions part of the specification. EINVAL is typically used to denote that an argument passed to the function is outside of bounds.
(0003660)
nmav (reporter)
2017-04-06 09:46

> Though I'm still a fan of having an interface adopted that is guaranteed to work (like arc4random_buf()), any progress in this area is good.

Certainly, though this interface is quite orthogonal to that. It will allow crypto libraries to seed their own PRNG using the system's PRNG. A posix_random() function with the properties of arc4random_buf() would be certainly a nice complement to this one, but not necessary.

> Instead of hardcoding the 256-byte upper limit, would it make sense to add a definition 'GETENTROPY_MAX', for which we state that it has to be at least 256 bytes in size? Leaving it undefined either means there is no limit or it's compile-time undefined. We should probably also add a sysconf() option to query the run-time limit.

I like that proposal and the EIO comments, especially the part where it requires it to be more than 256 bytes (to prevent systems which set values like 1). Feel free to post a note with an updated interface incorporating your comments.
(0003662)
shware_systems (reporter)
2017-04-06 20:00

IIRC, Bug 859 was tentatively approved partly because PRNGs can be implemented entirely in software. Perceived randomness was therefore a function of algorithm quality and usage. An interface like this more assumes motherboards or CPUs all have peripherals that are designed to generate these uniformly random sequences independently, to guarantee they are 'cryptographically secure' in the black box sense, and few commercial systems, if any, that I'm aware of actually provide this type of hardware. Many have hardware that can provide values random enough looking, possibly, to be a PRNG seed value, but not securely so or dedicated to this purpose. This lack of hardware is why, I believe, many PKCS key generators rely on the effective randomness of an operator moving a mouse around as their seed source.

That said, I agree adding something along these lines is desirable for uniformity of usage, but I don't see it as a candidate for the Base. It should be marked part of the CRYPT Option Group, and the ENOSYS error should use wording consistent with the other members of that group. No POSIX system is required to use a system call paradigm, after all; hardware accesses can be coded directly into the C library.
(0003663)
nmav (reporter)
2017-04-07 08:47

I do not believe that the comment above is accurate. There is no hardware requirement such as "random generator" peripherals, or any such assumption whatsoever, to provide a cryptographic random generator. In fact, in almost none of the systems that today provide that functionality, such hardware is present. Implementing a CSPRNG requires the OS being able to gather several bits of information unpredictable to an adversary to seed a software PRNG. That is possible today even with the most constrained systems.
(0003664)
schwarze (reporter)
2017-04-07 10:35

Since nobody else from OpenBSD commented so far: Is it really wise to standardize getentropy() before standardizing arc4random()? We have seen application software abuse getentropy() as a random number generator before, and this order of standardization might possibly precipitate that kind of abuse. Wouldn't it be a better idea to standardize arc4random[_buf]() first, which is *much* more relevant for application software?

I'm not at all saying that i oppose including this interface, in particular since the use case given by nmav, "allow crypto libraries to seed their own PRNG", is probably relevant. Would it make sense to include a statement to the effect, "This function is not intended to be used by application programs as a random number generator. Its use should be restricted to seeding pseudo random number generator libraries" in the DESCRIPTION, or at least in the APPLICATION USAGE?

Besides, as far as i know, the restriction to 256 bytes was imposed deliberately and not as a matter of laziness, because that amount of data is sufficient to seed PRNGs, and allowing more would merely complicate the implementation, increase the risk of depleting the actual entropy available, and encourage abuse as a random-number generator. For that reason, GETENTROPY_MAX seems like a bad idea, not only for reasons of needless complication and incompatibility with existing implementations, but also because it encourages abuse of the function. I strongly agree with nmav's argument that getentropy() and arc4random_buf() are complimentary, and that argument implies that GETENTROPY_MAX blurs the picture rather than helping anything.

Please take all this with a grain of salt; i'm not a specialist in cryptography. I'm merely asking questions based on what i heard in discussions with the original developers of getentropy().
(0003667)
EdSchouten (reporter)
2017-04-07 17:32

I agree with Ingo. POSIX should aim at standardising functional APIs that people can use freely in their applications. It currently feels as if we're attempting to standardise the low-level intrinsic, while we should aim to add the high-level API instead.
(0003668)
steffen (reporter)
2017-04-07 18:54

I totally and completely disagree with #3667. Anyone can place mediocre high-level pseudo random code on top of a good low-level interface. I think it gets increasingly impossible to create something good on a mediocre or non-existing low-level entropy source interface.

I for one adore POSIX because of the low-level interface that (would) allow(ed) me to use -nostdinc (and not to forget -fno-exceptions -fno-rtti). Except for getdirentries() almost anything one needs is available to create truly portable applications like so, which i think cannot be rated high enough!

This is true even if vendors seem to favor more and more to put money into the Linux kernel than in their own complete distributions, in order to effectively safe money on the research side -- this is nothing but a degenerative motion towards a monoculture, and in real-life monocultures cannot survive. (In fact "safe even more money", if one looks at the manuals of some commercial vendors, which never got enough time nor love.)

I regret that we don't get a high-level interface in addition, i.e., that the corresponding issue has been closed: because like this vendors could exchange algorithms as the necessities arise, and could provide optimised variants. And because errors would be encapsulated in the good maintained C library, rather than shipped in uncountable packages which implement their own / have a copy of some pseudo-random generator because they need the functionality.

But it should be in addition, because in practice many tasks require good random entropy and systems do provide it since decades: /dev/u?random on Linux and FreeBSD, /dev/arandom on OpenBSD, increasingly often sysctl(2)s or direct system calls which provide the same data but without requiring a file-descriptor.
It would be logical to lift this to a standardized level, in my opinion.
(0003669)
shware_systems (reporter)
2017-04-07 19:01

Re: 3663 & 3667:
It's described as referencing device files, implying to me actual devices (such as fingerprint or retinal scanners as biometric "entropy" just starting to be in common use), not bits it is hoped is unpredictable masquerading as one. Many of the bits current systems use have modulo or similar problems that allow them to be brute force attacked, or can be compromised in other ways because they're not black boxes.

The RC4 algorithm I believe arc4random() is based on (I haven't looked at the code) has been shown to be susceptible to brute force attacks too, so is no longer recommended by NIST for any new code, as are the current interfaces in the CRYPT group. Moore's law has caught up with algorithms using less than 64 bits of seed, pretty much. The movement is heading towards governments requiring platforms support algorithms that use over 256 bits as seed sizes, in the public documents I've come across.
See http://www.pdflib.com/en/knowledge-base/pdf-20/deprecated-features/ [^] for the plans for PDFs, being voted on now or soon at ISO, as an example of this movement.

This is one of the faster evolving technologies, actually, and eventually I expect the experts on cryptography employed by various national bodies, and who have access to their classified documents that might pertain also, to weigh in directly on what should or shouldn't be part of the standard for Issue 8. It may even be a formal ISO-TR work activity, similar to how the regex group was a separate effort for Issue 6, but this is far from being decided yet.

An interface like this or a work-alike I don't see as being so controversial someone might want to exclude it, but who knows what some politicians might decide to enact as laws whatever our aims, given how much cyber-security issues are being bandied about various media nowadays. By the time Issue 8 is that advanced in addressing the non-security related bug reports, there may well be new hardware requirements and algorithms the industry, and then the standard or this interface, will have to accommodate.
(0003670)
EdSchouten (reporter)
2017-04-07 19:04

The security of the RC4 algorithm is not up for discussion. The API is called arc4random*() for historical reasons. There is no requirement that it's actually implemented using RC4. OpenBSD uses Salsa/ChaCha20, for example.
(0003671)
shware_systems (reporter)
2017-04-08 04:45

*shrug* FreeBSD uses both an Arc4 variant, in its libc, and ChaCha, in an OpenBSD compatibility module, so it is part of the history of arc4random(). The point is many interfaces that have seen common use in this area are not as secure nor as random as advertised, and are subject to being superseded by factors outside the control of POSIX or implementors.
(0003734)
nmav (reporter)
2017-05-24 07:39

Please avoid unrelated discussion for APIs which are not part of the proposal. The proposal is about adding getentropy().
(0004137)
ajosey (manager)
2018-09-30 18:18

See the New Work item section of http://www.opengroup.org/austin/docs/austin_sd6.txt [^]

The recommended criteria for development of new interfaces to enable
them to be considered for inclusion in a future revision are as follows:

1.There must be a written specification that has undergone a formal
consensus based approval process and is suitable for inclusion.

Parties interested in submitting new work items through one of the
three organizations within the Austin Group (The Open Group, IEEE, ISO/IEC)
should contact the appropriate Organizational Representative for further
information and advice on how each organization handles new work items.
Submissions from other organizations will also be considered.
Items 2 through 4 below apply to all submissions regardless of
origin.

2.There must be an implementation, preferably a reference implementation.

3.The specification must be "sponsored" by one of three organizations
(The Open Group, IEEE, ISO/IEC) within the Austin Group, i.e. they would
support and champion its inclusion.

4.Submitters must provide an outline plan of the editing instructions to
merge the document with the Austin Group specifications, and assistance
to the Austin Group editors as required to complete the merger.
For an example, see

https://collaboration.opengroup.org/platform/single_unix_specification/protected/mailarch.php?soph=N&action=show&archive=austin-group-l&num=434 [^] [^]

or
https://collaboration.opengroup.org/sophocles/mailarch.php?soph=Y&action=show&archive=austin-group-l&num=434 [^] [^]

Andrew J will take the action to forward to The Open Group Organizational Rep.
(0004211)
ajosey (manager)
2019-01-16 13:33
edited on: 2019-01-16 13:33

The Open Group Base Working group has agreed to sponsor this as a new work item. The next stage will be to develop a manual page entry for review and eventual approval at The Open Group.

Is there further input needed beyond that in the Description of this bug report?

(0004669)
nmav (reporter)
2019-11-27 14:52

I am updating the proposed description with the comments received. That should be:


SYNOPSIS
       #include <unistd.h>

       int getentropy(void *buffer, size_t length);

DESCRIPTION
The getentropy() function writes length bytes of data starting at the location pointed to by buffer. The output is unpredictable high quality random data, generated by a cryptographically secure pseudo-random number generator. The maximum permitted value for the length argument is GETENTROPY_MAX. GETENTROPY_MAX is always larger than 256.

A successful call to getentropy() always provides the requested number of bytes of entropy.

RETURN VALUE
       On success, this function returns zero, On error, -1 is
returned, and errno is set appropriately.

ERRORS
       EINVAL Length is greater than GETENTROPY_MAX.

       ENOSYS The system does not provide the necessary system calls for this operation.

NOTES

The getentropy () call is not a cancellation point.
(0004672)
schwarze (reporter)
2019-11-27 16:15

Re note #4669:

I think there is one outright error in the new wording. It must not say "GETENTROPY_MAX is always larger than 256." If you want to have a GETENTROPY_MAX constant at all, the text should be "is always at least 256" or "is always equal to or larger than 256."

That said, i'm aware that Ed asked about defining GETENTROPY_MAX, but in the comments above, i fail to see any rationale why inventing this constant might help anything. Even Ed did not really argue in favor but merely asked a question, and given that nobody answered that question, i conclude that nobody sees a point in it, whereas i did explain three specific reasons why i consider it detrimental. So please do reconsider making the limit the fixed value 256, which is sufficient for all relevant purposes, much simpler (which is always good in security-relevant interfaces), less prone to abuse, and reduces the risk of depleting the entropy actually available in the system. Making the limit larger feels really pointless and potentially dangerous.

Also, you said that you updated the proposed description "with the comments received." But i don't see any response to my comment that it would be desirable, to discourage abuse of the new interface, to include a sentence similar to "This function is not intended to be used by application programs as a random number generator. Its use should be restricted to seeding pseudo random number generator libraries." - at least below APPLICATION USAGE or RATIONALE, if not in the DESCRIPTION.
(0004674)
nmav (reporter)
2019-11-28 07:49

> I think there is one outright error in the new wording. It must not say "GETENTROPY_MAX is always larger than 256." If you want to have a GETENTROPY_MAX constant at all, the text should be "is always at least 256" or "is always equal to or larger than 256."

The point is that this interface guarrantees 256 bytes and the update makes it, at least 256 bytes. You've got a point, on the at equal or larger. Thanks.

> But i don't see any response to my comment that it would be desirable, to discourage abuse of the new interface, to include a sentence similar to "This function is not intended to be used by application programs as a random number generator. Its use should be restricted to seeding pseudo random number generator libraries." - at least below APPLICATION USAGE or RATIONALE, if not in the DESCRIPTION.

Sorry I missed this comment. I am not sure whether documented the intended use of the interface can change a lot on its use, but it may have some value. Yes the intention is an interface that can be used to seed a random generator, but that shouldn't prohibit an application which needs 16 random bytes on its lifetime to use it. Let me give it a try updating it.
(0004675)
nmav (reporter)
2019-11-28 07:57
edited on: 2019-11-28 08:01

SYNOPSIS
       #include <unistd.h>

       int getentropy(void *buffer, size_t length);

DESCRIPTION
The getentropy() function writes length bytes of data starting at the location pointed to by buffer. The output is unpredictable high quality random data, generated by a cryptographically secure pseudo-random number generator. The maximum permitted value for the length argument is GETENTROPY_MAX. GETENTROPY_MAX is equal to or larger than 256.

A successful call to getentropy() always provides the requested number of bytes of entropy.

RETURN VALUE
       On success, this function returns zero, On error, -1 is
returned, and errno is set appropriately.

ERRORS
       EINVAL Length is greater than GETENTROPY_MAX.

       ENOSYS The system does not provide the necessary system calls for this operation.

NOTES

The intended use of this function is to create a seed for other pseudo random number generators.

The getentropy () call is not a cancellation point.

(0004677)
schwarze (reporter)
2019-11-28 12:10

Re: #4675

Thanks, i think that's an improvement over previous versions.

But you still didn't explain why all three of the reasons i explained above why GETENTROPY_MAX is a bad idea should be disregarded. I think the following would be clearly better, without defining GETENTROPY_MAX at all: "The maximum permitted value for the length argument is 256."
(0004678)
nmav (reporter)
2019-11-28 13:25

Is that something significant you'd like to prevent this API from being considered? If not I would not like to follow up because that interface is on consideration for very long. In short my point is that 256 is an arbitrary numberm and having a definition can allow an update for systems which may want to include a future requirement.
(0004679)
shware_systems (reporter)
2019-11-28 14:16

Re: 4677
Yes, 256 is an arbitrary value, but having it as a variable future proofs the interface for when even 256 is not enough. A minimum more consistent with all practice would be 8 bytes, and I would put this in <limits.h> as _POSIX_ENTROPYBUFSIZE with 256 as XSI minimum for an increaseable ENTROPYBUFSIZE value, with appropriate _SC_* or _PC_* name in <unistd.h>,
rather than the constant directly.

The latter would be appropriate for systems with multiple devices useable as entropy sources, but with buffers of varying width, that may be opened for use with a getentropyat(fd,buf,bufsize) interface.
(0004680)
schwarze (reporter)
2019-11-28 14:44
edited on: 2019-11-28 14:58

Re: #4678

I do *not* think that the number 256 is arbitrary; it was carefully chosen by Matthew Dempsky and Theo de Raadt in June 2014 when they originally designed this interface because it is sufficient for seeding random number generators, see for example the commit message https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/dev/rnd.c#rev1.157 [^] . Which pseudo-random number generator do you have in mind that benefits from more than 2048 bits of entropy for seeding, or that might plausibly require more in the forseeable future? I'm not a cryptographer, so i would gladly be educated by one if there is a plausible need, but the designers of this interface, who clearly understand requirements of random number generation, explicitly said back then that they did *not* see a need to make this larger (let alone variable). Making something variable without a plausible need "just in case" does not make an interface better, only more complicated, harder to implement, and more error-prone; i won't repeat the other two arguments against allowing larger buffers. And yes, i do think these arguments are significant, or i wouldn't bother mentioning them.

Edit: And by the way, isn't GETENTROPY_MAX new invention, which the Austin Group usually refrains from doing? Glibc hardcodes 256, FreeBSD hardcodes 256, Illumos hardcodes 256, a Google search for "GETENTROPY_MAX" comes up with exactly one result: this very Aardvark ticket. Where are the widely deployed reference implementations that the Austin Group typically asks for before standardizing a new feature?

(0004681)
nmav (reporter)
2019-11-28 14:56
edited on: 2019-11-28 15:13

I've taken into consideration all input, and my proposal remains the same as in Note: 0004675.

(0004682)
shware_systems (reporter)
2019-11-28 18:16
edited on: 2019-11-28 18:18

Re: 4680
Yes, it is arbitrary. That it reflects those authors understanding of current cryptography requirements doesn't make it less so. At one time 48 bits was considered "adequate", as embodied in the current CRYPT interfaces, then it moved to 128 bits, SHA-64 to 512, and PKCS with 1024 and 2048 bits. Who's to say someone won't devise a means that these are found susceptible to cracking, so 4k, 16k or even larger might be seen as needed in future. Isn't me, or them, or anyone else for that matter.

As to constant names, the standard recommends implementations do not use hard coded limit values anywhere as, paraphrasing <limits.h>, different implementations may need other values to accommodate hardware variations. Introducing a name for this purpose to <limits.h> with a _MAX, or _MIN, suffix is permitted, by XSH 2.2.2, the table on P475. While not explicit there, when the standard moves optional features to the base any _MAX values relevant to those options that were previously not defined because a platform did not support the option get all these as new required names, that to that platform also counts as requiring invention. Ergo, the standard is free to add _MAX suffixed names as well, though preferably with a _POSIX or POSIX prefix in accordance with what is explicit in that section.


- Issue History
Date Modified Username Field Change
2017-03-31 07:06 nmav New Issue
2017-03-31 07:06 nmav Name => Nikos Mavrogiannopoulos
2017-03-31 07:06 nmav Organization => Red Hat
2017-03-31 07:06 nmav Section => getentropy
2017-03-31 07:06 nmav Page Number => 0
2017-03-31 07:06 nmav Line Number => 0
2017-03-31 07:17 Don Cragun Relationship added related to 0000859
2017-04-01 06:25 EdSchouten Note Added: 0003657
2017-04-06 09:46 nmav Note Added: 0003660
2017-04-06 20:00 shware_systems Note Added: 0003662
2017-04-07 08:47 nmav Note Added: 0003663
2017-04-07 10:35 schwarze Note Added: 0003664
2017-04-07 17:32 EdSchouten Note Added: 0003667
2017-04-07 18:54 steffen Note Added: 0003668
2017-04-07 19:01 shware_systems Note Added: 0003669
2017-04-07 19:04 EdSchouten Note Added: 0003670
2017-04-08 04:45 shware_systems Note Added: 0003671
2017-05-24 07:39 nmav Note Added: 0003734
2018-09-30 18:18 ajosey Note Added: 0004137
2019-01-16 13:33 ajosey Interp Status => ---
2019-01-16 13:33 ajosey Note Added: 0004211
2019-01-16 13:33 ajosey Note Edited: 0004211
2019-11-27 14:52 nmav Note Added: 0004669
2019-11-27 16:15 schwarze Note Added: 0004672
2019-11-27 16:15 schwarze Note Added: 0004673
2019-11-27 16:16 schwarze Note Deleted: 0004673
2019-11-28 07:49 nmav Note Added: 0004674
2019-11-28 07:57 nmav Note Added: 0004675
2019-11-28 08:01 nmav Note Edited: 0004675
2019-11-28 12:10 schwarze Note Added: 0004677
2019-11-28 13:25 nmav Note Added: 0004678
2019-11-28 14:16 shware_systems Note Added: 0004679
2019-11-28 14:44 schwarze Note Added: 0004680
2019-11-28 14:56 nmav Note Added: 0004681
2019-11-28 14:58 schwarze Note Edited: 0004680
2019-11-28 15:13 Don Cragun Note Edited: 0004681
2019-11-28 18:16 shware_systems Note Added: 0004682
2019-11-28 18:18 shware_systems Note Edited: 0004682


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker