Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0001631 [Issue 8 drafts] Shell and Utilities Editorial Error 2023-01-21 18:45 2023-03-10 10:53
Reporter dmitry_goncharov View Status public  
Assigned To
Priority normal Resolution Accepted As Marked  
Status Applied   Product Version
Name Dmitry Goncharov
Organization
User Reference
Section make
Page Number https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/nframe.html [^]
Line Number 0
Final Accepted Text See Note: 0006169
Summary 0001631: Default archive rules are incompatible with parallel processing.
Description The default archive rules specified in the standard are

".c.a:
    $(CC) -c $(CFLAGS) $<
    $(AR) $(ARFLAGS) $@ $*.o
    rm -f $*.o


.f.a:
    $(FC) -c $(FFLAGS) $<
    $(AR) $(ARFLAGS) $@ $*.o
    rm -f $*.o
"

Parallel processing was introduced in
https://austingroupbugs.net/view.php?id=1436. [^]

With -j specified these rules can result in an incomplete archive, because multiple instances of ar are updating the same archive simultaneously.
Desired Action One option is to state that these rules are incompatible with parallel processing.
Another option is to remove these rules from the standard.
Tags issue8
Attached Files

- Relationships

-  Notes
(0006124)
kre (reporter)
2023-01-21 21:48

I don't know make implementations, or whether any does this, but
another option might be to require that when running sub-builds
in parallel (whatever the correct terminology is for that) that
no two such parallel sub-builds can be building the same target,
simultaneously.

To me that seems like a sane rule for all parallel builds, regardless
of whether there are archives (or anything similar) involved.
(0006125)
rillig (reporter)
2023-01-21 22:10

@dmitry_goncharov
Could you please elaborate why the archiving is the problem with these rules?

As far as I understand these rules, the same problem occurs when two competing files named `source.c` and `source.f` are compiled to `.o` files.

Since the inference "shall be searched in the order defined", `source.c` is chosen by both parallel makes, and `source.f` is effectively ignored.

Or are you talking about a different scenario in which one of the makefiles has deactivated the `.c.o` rule? But then, compilation would be as broken as archiving.

Or did I miss something?
(0006126)
dmitry_goncharov (reporter)
2023-01-22 02:08

> but another option might be to require that when running sub-builds
in parallel (whatever the correct terminology is for that) that
no two such parallel sub-builds can be building the same target,
simultaneously.

This looks a reasonable idea to me. Should be relatively easy to implement in top make alone. However, this is more difficult when submake is involved. One submake does not know which target another submake or top make is building. One technique could be to use a form of file locking on the target (as long as the target is an actual file in the fs). However, this is not cheap.
(0006127)
dmitry_goncharov (reporter)
2023-01-22 03:02
edited on: 2023-01-22 03:13

> Could you please elaborate why the archiving is the problem with these rules?

Let us assume we have hello.c and bye.c and have to build libhello.a which contains hello.o and bye.o.
With these rules make will first compile hello.c and bye.c in parallel. Once an object file (either hello.o or bye.o) is ready make will fork-exec an instance of ar to insert that object file to libhello.a. Which possibly results in these two instances of ar running in parallel.

(0006128)
psmith (developer)
2023-01-22 18:42

My preference would be to change the default rules to remove the $(AR) from the individual compilation, and add a new default rule to build the archive from object files using the "$?" (only updated files). But I'm not sure this can be easily done in the standard version of make, that doesn't have the complexity of intermediate files.
(0006131)
steffen (reporter)
2023-01-23 21:59

My fault.

Personally i think the standard inference rules are very much overcome, especially anything regarding SCCS that has been deprecated by life (except maybe for some private use cases). (And Jörg is dead to improve his SCCS.)

The syntax regarding archives i have never seen used in real life.

I would go with what the GNU make manual says, "either write some dedicated rules, or do not use -j for archives" (more or less cited correctly).
So maybe a sentence of warning?
(0006132)
steffen (reporter)
2023-01-23 22:02

P.S.: BSD make and smake (from Jörg) do not even get it right. BSD make also does not support the standard-documented way to output inference rules.
But eg

  lib: lib(t1.o) lib(t2.o)

only works with GNU make. (I have no Solaris to test.)
And GNU make may fail for -j, as it documents (in its info manual).

#?0|kent:x$ bmake
`lib' is up to date.
#?0|kent:x$ smake
smake: Can't find any source for 'lib(t1.o)'.
smake: Couldn't make 'lib'.
#?1|kent:x$ make
gcc -O1 -g -c -o t1.o t1.c
ar -rv lib t1.o
ar: creating lib
a - t1.o
gcc -O1 -g -c -o t2.o t2.c
ar -rv lib t2.o
a - t2.o
ok
rm t2.o t1.o
(0006133)
steffen (reporter)
2023-01-23 22:04

P.P.S: even with "lib.a" it gets worse:

#?0|kent:x$ smake
smake: Can't find any source for 'lib.a(t1.o)'.
smake: Couldn't make 'lib.a'.
#?1|kent:x$ bmake
ok
#?0|kent:x$ ll
total 12K
-rw-r----- 1 steffen steffen 61 Jan 23 22:26 t1.c
-rw-r----- 1 steffen steffen 61 Jan 23 22:26 t2.c
drwxrwxrwt 10 root root 240 Jan 23 23:01 ../
drwxr-x--- 2 steffen steffen 100 Jan 23 23:02 ./
-rw-r----- 1 steffen steffen 41 Jan 23 23:02 makefile
#?0|kent:x$ make
gcc -O1 -g -c -o t1.o t1.c
ar -rv lib.a t1.o
ar: creating lib.a
a - t1.o
gcc -O1 -g -c -o t2.o t2.c
ar -rv lib.a t2.o
a - t2.o
ok
rm t2.o t1.o
#?0|kent:x$ ll
total 20K
-rw-r----- 1 steffen steffen 61 Jan 23 22:26 t1.c
-rw-r----- 1 steffen steffen 61 Jan 23 22:26 t2.c
-rw-r----- 1 steffen steffen 41 Jan 23 23:02 makefile
drwxrwxrwt 10 root root 240 Jan 23 23:02 ../
-rw-r----- 1 steffen steffen 7218 Jan 23 23:02 lib.a
drwxr-x--- 2 steffen steffen 120 Jan 23 23:02 ./
#?0|kent:x$

(But i have problems grasping inference rules, maybe it is my fault. I only used them very superficially.)
(0006134)
hvd (reporter)
2023-01-23 22:11

> Let us assume we have hello.c and bye.c and have to build libhello.a which contains hello.o and bye.o.

The predefined rules do not support that even without -j, do they? The predefined .c.a rule supports building hello.a from hello.c, using hello.o as an intermediate file. That's it. If you want a different name, such as libhello.a, or if you want to include multiple object files, you need custom rules, and it's then your own responsibility to making them parallel-build-safe.

Whether the predefined rule should be kept or not, the parallel processing should have no impact on that. For the very limited scenarios that the built-in rules handle, the parallel processing should not be an issue.
(0006135)
dmitry_goncharov (reporter)
2023-01-24 02:24

> The predefined rules do not support that even without -j, do they?
...
If you want a different name, such as libhello.a, or if you want to include multiple object files, you need custom rules, and it's then your own responsibility to making them parallel-build-safe.

You'll need to tell make that libhello.a depends on hello.o and bye.o.
But, the default rules, along with their default recipes, can still be used to build libhello.a.


$ ls
bye.c hello.c makefile
$ cat makefile
libhello.a: libhello.a(hello.o) libhello.a(bye.o)
$ make
cc -c -o hello.o hello.c
ar -rv libhello.a hello.o
ar: creating libhello.a
a - hello.o
cc -c -o bye.o bye.c
ar -rv libhello.a bye.o
a - bye.o
rm hello.o bye.o
$ ar tv libhello.a
rw-r--r-- 1001/1004 952 Jan 23 21:06 2023 hello.o
rw-r--r-- 1001/1004 952 Jan 23 21:06 2023 bye.o


> Whether the predefined rule should be kept or not, the parallel processing should have no impact on that.

People use parallel processing a lot. With these rules we expect every user to read the manual and notice the warning and avoid parallel processing.
(0006136)
hvd (reporter)
2023-01-24 09:56

> But, the default rules, along with their default recipes, can still be used to build libhello.a.

Thanks for spelling it out.

> My preference would be to change the default rules to remove the $(AR) from the individual compilation, and add a new default rule to build the archive from object files using the "$?" (only updated files). But I'm not sure this can be easily done in the standard version of make, that doesn't have the complexity of intermediate files.

Right, they complicate things. It would be easy if the .o files did not need to be removed. Then, the default .c.o rule could be used to make the object file, and a rule to build the archive from the .o files could use $? to get the updated object files (as opposed to the updated source files). But the .o files getting removed complicates things.

kre's idea to require that no two rules for the same target run in parallel seems like a good idea, but prevents hello.c and bye.c from being compiled in parallel. With the current rule, compiling them in parallel is unsafe as there is no way to not also cause the $(AR) commands to run in parallel, which then causes issues. If there is a way to separate the commands into separate rules, like with the referenced concept of intermediate files, that would become safe.

Yet another option would be to leave make as is, but require ar to use file locking to allow concurrent invocations, where a second invocation blocks until a first one completes. This is probably a bad idea, but mentioning it for completeness.
(0006137)
psmith (developer)
2023-01-24 18:55

If the .o files don't need to be removed then you don't need any special handling for archives at all. You can just write standard rules and you don't need the magical libfoo.a(foo.o) syntax or the special behaviors of the .a suffix rules, because you can just base things on the timestamp of the archive itself compared to the timestamp of the .o files.

The only way this extra behavior is needed or useful is when you want to update the archive _without_ having to rebuild all the object files (like you do, for example, when you link a program).
(0006152)
shware_systems (reporter)
2023-02-16 17:05

An alternative is possibly a change to ar requirements: that it check if an existing archive is already open for processing by another instance or application and then pause until it is capable of gaining such exclusive use. It is unknown whether any current implementation does this checking, however, but it would make use of -j with this rule plausible as the accesses and object adds would be self serializing.
(0006153)
steffen (reporter)
2023-02-16 20:41

re @6152
You mean this would be the right time to standarIdize flock(1)?
I use it often


       flock [options] file|directory command [arguments]
       flock [options] file|directory -c command
       flock [options] number

DESCRIPTION
       This utility manages flock(2) locks from within shell scripts or from
       the command line.

       The first and second of the above forms wrap the lock around the
       execution of a command, in a manner similar to su(1) or newgrp(1). They
       lock a specified file or directory, which is created (assuming
       appropriate permissions) if it does not already exist. By default, if
       the lock cannot be immediately acquired, flock waits until the lock is
       available.

       The third form uses an open file by its file descriptor number. See the
       examples below for how that can be used.
(0006169)
nick (manager)
2023-02-23 17:18
edited on: 2023-02-23 17:22

Add to APPLICATION USAGE (D2.1 P2952, after line 99106):
It is important to be careful when using parallel execution (the -j switch) and archives. If multiple $(AR) commands run at the same time on the same archive file, they will not know about each other and can corrupt the file. If the -j option is used, it is necessary to use .WAIT in between archive member prerequisites to prevent this (see EXAMPLES).


On D2.1 page 2953 line 99164 section make (EXAMPLES), change:
lib: lib(file1.o) lib(file2.o) lib(file3.o)
    @echo lib is now up-to-date


The .c.a rule is used to make file1.o, file2.o, and file3.o and insert them into lib.

to:
lib.a: lib.a(file1.o) .WAIT lib.a(file2.o) .WAIT lib.a(file3.o)
    @echo lib is now up-to-date


The .c.a rule is used to make file1.o, file2.o, and file3.o and insert them into lib.a.



- Issue History
Date Modified Username Field Change
2023-01-21 18:45 dmitry_goncharov New Issue
2023-01-21 18:45 dmitry_goncharov Name => Dmitry Goncharov
2023-01-21 18:45 dmitry_goncharov Section => make
2023-01-21 18:45 dmitry_goncharov Page Number => https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/nframe.html [^]
2023-01-21 18:45 dmitry_goncharov Line Number => 0
2023-01-21 21:48 kre Note Added: 0006124
2023-01-21 22:10 rillig Note Added: 0006125
2023-01-22 02:08 dmitry_goncharov Note Added: 0006126
2023-01-22 03:02 dmitry_goncharov Note Added: 0006127
2023-01-22 03:13 dmitry_goncharov Note Edited: 0006127
2023-01-22 03:13 dmitry_goncharov Note Edited: 0006127
2023-01-22 18:42 psmith Note Added: 0006128
2023-01-23 21:59 steffen Note Added: 0006131
2023-01-23 22:02 steffen Note Added: 0006132
2023-01-23 22:04 steffen Note Added: 0006133
2023-01-23 22:11 hvd Note Added: 0006134
2023-01-24 02:24 dmitry_goncharov Note Added: 0006135
2023-01-24 09:56 hvd Note Added: 0006136
2023-01-24 18:55 psmith Note Added: 0006137
2023-02-16 17:05 shware_systems Note Added: 0006152
2023-02-16 20:41 steffen Note Added: 0006153
2023-02-23 17:18 nick Note Added: 0006169
2023-02-23 17:19 nick Note Added: 0006170
2023-02-23 17:19 nick Note Deleted: 0006170
2023-02-23 17:20 nick Final Accepted Text => See Note: 0006169
2023-02-23 17:20 nick Status New => Resolved
2023-02-23 17:20 nick Resolution Open => Accepted As Marked
2023-02-23 17:22 nick Note Edited: 0006169
2023-02-23 17:22 nick Tag Attached: issue8
2023-03-10 10:53 geoffclare Status Resolved => Applied


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker