Austin Group Defect Tracker

Aardvark Mark IV

Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0001812 [Issue 8 drafts] Shell and Utilities Editorial Enhancement Request 2024-02-15 20:53 2024-04-19 20:49
Reporter dwheeler View Status public  
Assigned To ajosey
Priority normal Resolution Accepted As Marked  
Status Resolved   Product Version
Name David A. Wheeler
Organization The Linux Foundation
User Reference
Section xargs
Page Number 3600-3601
Line Number 123252
Final Accepted Text Note: 0006751
Summary 0001812: Support xargs -P 0
Description Issue [^] modifies xargs to support "-P". This is good, since *many* computers have multiple processors available.

However, the proposed text doesn't define "-P 0". I don't think I've ever used an argument *other* than 0. The meaning of "0" seems to be widely agreed on as well, that is, "use as many as you can". Having a standard way to say "parallelize as much as you can" is extremely useful.
Desired Action Whenever this is eventually added, instead of:

-P maxprocs Parallel mode: execute at most maxprocs invocations of utility concurrently. If the value of maxprocs is non-positive, the behavior is unspecified.

Say something like this:

-P maxprocs Parallel mode: execute at most maxprocs invocations of utility concurrently. If the value of maxprocs is 0, run as many processes as possible (this is often the number of processors available). If the value of maxprocs is negative, the behavior is unspecified.
Tags issue9
Attached Files

- Relationships
related to 0001801Resolved xargs: add -P option 

-  Notes
geoffclare (manager)
2024-02-16 10:28

The wording "as many processes as possible" is problematic. An implementation could conform to this by just forking until it gets EAGAIN, which would not be good.

Slightly better would be "an implementation-defined maximum number of processes", but this would still allow the same poor implementation: it would just need to document that number as being CHILD_MAX.

In Issue 8 we have sysconf(SC_NPROCESSORS_ONLN) so perhaps we should involve that somehow, but we would need to assess whether existing implementations would conform to the new requirement.

The best time to make a decision on this would probably be after we begin work on Issue 9, rather than now.
kre (reporter)
2024-02-16 11:39

I agree with Note: 0006659 (had intended to say much the same thing), other
than the possible use of SC_NPROCESSORS_ONLN probably (maybe that's better
than nothing, but probably not).

While that value has its uses, apportioning cpus to tasks isn't one of them,
as not all cpus in a system are created equal any more - some are more suited
for one kind of task, and others for different activities (which doesn't even
begin to address whether the peculiar "hyperthreading" cpus count for any of

I certainly agree with deferring a decision on this, and all other issue 9
matters (probably even Issue 8 TC1 matters) until after issue 8 is published
(no reason why discussions cannot happen leading to a potential resolution
dwheeler (reporter)
2024-02-19 19:58

In my experience, the most common value for xargs -P is *0*, as that basically says "please maximize throughput" in a practically portable way. It *is* hard to formally specify though!

I think a better approach would be to attempt to describe, from a *human* perspective, what is being requested. How about this, instead of:

> If the value of maxprocs is 0, run as many processes as possible (this is often the number of processors available).

how about this:

> If the value of maxprocs is 0, the system heuristically attempts to minimize total runtime by executing many processes (e.g., by executing the number of processes equal to the number of processors).
steffen (reporter)
2024-02-19 21:28

In my very humble opinion in this respect i want to point out that many programs use many different approaches to this.
Reasonable i personally find zstd:

           # is 0, attempt to detect and use the number of physical CPU cores.
           In all cases, the nb of threads is capped to ZSTDMT_NBWORKERS_MAX,
           which is either 64 in 32-bit mode, or 256 for 64-bit environments.
           This modifier does nothing if zstd is compiled without multithread

Very unreasonable certain make(1) implementation where 0 means, if i recall correctly, to unfold the entire tree, leading to hundreds of parallel invocations. (Again: iirc.)

I think the standard, who extensively documented the make(1) -j parameter with even an implementation guideline, should possibly add an entry to "3. Definitions", and point to the according entry for xarg(1)s -P (0).

I also want to point out that the sysctl's and such used to do the necessary "calculations" often return false numbers.
This is true in particular for containers ("bound namespaces" more), where there are even open bug reports (Linux glibc; FreeBSD) which are not resolved because also readers of this list chose to go this route, continuing to actively report false numbers in order not to have a need to adjust a possibly large number of test cases in their test framework. (I am too lazy to search for the numbers now; but would on request.)

Ie. The standardized and available possibilities to actually get a notion of the number of available CPU threads does not reflect actual reality.
For xargs(1) in particular, also make(1), the number of tools actually spawned is anyway most likely much greater than the number given to -P or -j.
hvd (reporter)
2024-02-19 22:44

Note also that in current implementations where xargs -P 0 is supported (GNU, FreeBSD), there is no limit based on anything like number of processors, and for some commands, it may not be meaningful to have such a limit because the processes are in no way CPU bound. If xargs -P 0 is currently used like that, and its behaviour is changed to start imposing a limit based on number of processors, the obvious question that a user would ask would be how to restore the old behaviour, and in my opinion a rationale should be provided to explain what such a user can do. From my reading of what is proposed here, such a user would simply be out of luck and should keep using an old version of xargs. I hope such a user can get a better alternative instead.
gabravier (reporter)
2024-02-21 00:41

I can confirm that all current implementations I can check that have `-P0` don't limit themselves based on the number of processors - i.e. all of GNU findutils, FreeBSD, illumos, busybox and toybox.

A simple of these, Busybox, does have a limitation, though, but this doesn't seem to have anything to do with the amount of processors on a machine: Busybox limits the amount of concurrent invocations to 100 with a code comment indicating the rationale as simply being "let's not go crazy high".
geoffclare (manager)
2024-04-15 16:29

After making the page 3601 line 123232 change from bug 1801, change:
If the value of maxprocs is non-positive, the behavior is unspecified.

If the value of maxprocs is zero, the system heuristically attempts to minimize total runtime by executing multiple invocations of utility without waiting for previous executions to complete (e.g., by choosing the number of such invocations based on the number of available processors).

If the value of maxprocs is negative, the behavior is unspecified.
mohd_akram (reporter)
2024-04-19 20:49

FYI, I didn't add this in the other issue despite wanting it because of what hvd and gabravier mentioned. The option does not seem to behave reasonably in current implementations and IMO shouldn't be included because of that. This description is misleading because the current behavior is to maximize the number of processes without regard to runtime or anything else which can lead to hundreds of processes that slow down the entire OS. A user who is simply looking for a "parallelize this" flag can use `-P$(getconf NPROCESSORS_ONLN)`.

- Issue History
Date Modified Username Field Change
2024-02-15 20:53 dwheeler New Issue
2024-02-15 20:53 dwheeler Status New => Under Review
2024-02-15 20:53 dwheeler Assigned To => ajosey
2024-02-15 20:53 dwheeler Name => David A. Wheeler
2024-02-15 20:53 dwheeler Organization => The Linux Foundation
2024-02-15 20:53 dwheeler Section => xargs
2024-02-15 20:53 dwheeler Page Number => 3600-3601
2024-02-15 20:53 dwheeler Line Number => 123252
2024-02-16 10:28 geoffclare Note Added: 0006659
2024-02-16 11:39 kre Note Added: 0006661
2024-02-19 19:58 dwheeler Note Added: 0006667
2024-02-19 21:28 steffen Note Added: 0006668
2024-02-19 22:44 hvd Note Added: 0006669
2024-02-21 00:41 gabravier Note Added: 0006671
2024-03-01 11:48 geoffclare Project 1003.1(2008)/Issue 7 => Issue 8 drafts
2024-04-15 16:04 geoffclare Relationship added related to 0001801
2024-04-15 16:29 geoffclare Note Added: 0006751
2024-04-15 16:30 geoffclare Final Accepted Text => Note: 0006751
2024-04-15 16:30 geoffclare Status Under Review => Resolved
2024-04-15 16:30 geoffclare Resolution Open => Accepted As Marked
2024-04-15 16:31 geoffclare Tag Attached: issue9
2024-04-19 20:49 mohd_akram Note Added: 0006771

Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker