Anonymous | Login | 2024-10-09 03:49 UTC |
Main | My View | View Issues | Change Log | Docs |
Viewing Issue Simple Details [ Jump to Notes ] | [ Issue History ] [ Print ] | ||||||
ID | Category | Severity | Type | Date Submitted | Last Update | ||
0001437 | [1003.1(2016/18)/Issue7+TC2] Shell and Utilities | Editorial | Enhancement Request | 2020-12-15 21:02 | 2024-06-11 09:08 | ||
Reporter | steffen | View Status | public | ||||
Assigned To | |||||||
Priority | normal | Resolution | Accepted As Marked | ||||
Status | Closed | ||||||
Name | steffen | ||||||
Organization | |||||||
User Reference | |||||||
Section | Vol. 3: Shell and Utilities, Issue 7, make | ||||||
Page Number | (2975) 2989 | ||||||
Line Number | (98708, 98742) 99308 | ||||||
Interp Status | --- | ||||||
Final Accepted Text | See Note: 0005489. | ||||||
Summary | 0001437: make: (document .NOTPARALLEL and .WAIT special targets) in RATIONALE | ||||||
Description |
Parallel, even massively parallel processing has become the widely supported and used default, yet the standard make(1) does not document it. If the -j command line option as requested in issue #1436 is standardized, the standard should also mention ways to synchronize rule processing, in order to support more complicated build relationships seen in real-life projects. The suggestion given here can be used with the most widely used make(1) implementations (GNU make, BSD make) on any tested platform (Linux, *BSD, SunOS / Solaris). The make family rooted on the Sun side of likfe has has at least .WAIT: reserved for decades, and in practice .NOTPARALLEL: .WAIT: # Luckily BSD make supports specifying this as target, too tangerine: config .WAIT build .WAIT test .WAIT install citron: config .WAIT build .WAIT install all: config .WAIT build produces the desired outcome. |
||||||
Desired Action |
( On page 2975, insert before line 98708 .NOTPARALLEL When specified all rules specified in the given makefile are processed sequentially, as if -j has not been given. On page 2975, insert before line 98472 .WAIT When .WAIT appears in a dependency line, prerequisites that appear after it are built no sooner until all preceeding prerequisites have been built successfully. On page 2975, change on line 98472 ff. The special targets .IGNORE, .POSIX, .PRECIOUS, .SILENT, and .SUFFIXES shall be specified without commands. to The special targets .IGNORE, .NOTPARALLEL, .POSIX, .PRECIOUS, .SILENT, .SUFFIXES and .WAIT shall be specified without commands. ) On page 2989, insert before line 99308 The special targets .NOTPARALLEL and .WAIT represent different, rather incompatible synchronisation concepts of parallel build environments as used by the diverse implementations. By using an indirection both approaches can be supported: for implementations which use .NOTPARALLEL in a makefile to ensure sequential rule processing in the given file the full power of parallel processing will be available in recursive make instances invoked on makefiles without the directive, implementations which synchronize on the .WAIT special target will synchronize on it, for example: .NOTPARALLEL: # Process all rules sequentially .WAIT: # Define to satisfy "the other implementation" install: config .WAIT all echo Installing.. .. all: config .WAIT build build: make -f makefile.build config: ./configure |
||||||
Tags | issue8 | ||||||
Attached Files |
make.diff [^] (16,063 bytes) 2021-08-28 19:11 easier.diff [^] (7,414 bytes) 2021-09-02 23:20 gmake-4.3.wait.diff [^] (6,555 bytes) 2021-09-03 13:33 |
||||||
|
Relationships | |||||||||||||||||||
|
Notes | |
(0005184) psmith (developer) 2020-12-17 17:10 |
Just to note, GNU make does not support the .WAIT special target. |
(0005185) steffen (reporter) 2020-12-17 19:23 |
Ach! I wish it would, i think the .WAIT approach "is better" (as in explicit, and does not need an indirection). But you said on the ML that you do not plan to support it either. This is why the first part above is in parenthesis, and this issue actually requests some words in RATIONALE instead. The presented solution has the desired effect also in GNU make ("build" target uses parallel processing). |
(0005186) psmith (developer) 2020-12-17 21:16 |
I don't believe I said I didn't want GNU make to support it; I agree that it seems like a useful feature. I just said that given the way make's parallel support works I suspect implementation is not so easy and I didn't plan to work on it as I have other things on my (relatively small) plate. But, if someone else were to get motivated to try to get this working that would be fine with me :). I wonder if creating some kind of internal/implicit prerequisite relationship that wasn't visible to users would be a simpler way to implement this... or not. Well, something for someone to consider. |
(0005444) joerg (reporter) 2021-08-14 12:19 |
Well, implementing .WAIT is not really hard. You need to stop creating new active process slots if you see a .WAIT in the dependency list but wait until all running jobs for that target are finished before starting new jobs from the list to the right of .WAIT. You also need to remove .WAIT from the list of dependencies, if that list is needed for things like e.g. $^ or $?. In cases of doubt, SunPro Make is OpenSource... |
(0005446) rhansen (manager) 2021-08-19 16:22 edited on: 2021-08-19 16:22 |
On page 2975, insert before line 98708:.NOTPARALLELThe application shall ensure that this special target is specified without prerequisites or commands. When specified, make shall update one target at a time, regardless of whether the -j maxjobs option is specified. If the -j maxjobs option is specified, the option shall continue to be passed unchanged to sub-make invocations via MAKEFLAGS. On page 2975, insert before line 98742: .WAITThe application shall ensure that this special target, if specified as a target, is specified without prerequisites or commands. When .WAIT appears as a target, it shall have no effect. When .WAIT appears in a target rule as a prerequisite, it shall not itself be treated as a prerequisite; however, prerequisites that appear after it shall not be brought up-to-date until after all preceding prerequisites for the same target have been brought up-to-date. On page 2975, change line 98742 from: The special targets .IGNORE, .POSIX, .PRECIOUS, .SILENT, and .SUFFIXES shall be specified without commands.to: The special targets .IGNORE, .NOTPARALLEL, .POSIX, .PRECIOUS, .SILENT, .SUFFIXES and .WAIT shall be specified without commands. On page 2975 lines 98744-98747, change: Targets with names consisting of a leading <period> followed by the uppercase letters "POSIX" and then any other characters are reserved for future standardization. Targets with names consisting of a leading <period> followed by one or more uppercase letters are reserved for implementation extensions.to: Targets and prerequisites consisting of a leading <period> followed by the uppercase letters "POSIX" and then any other characters are reserved for future standardization. Targets and prerequisites consisting of a leading <period> followed by one or more uppercase letters, that are not described above, are reserved for implementation extensions. |
(0005447) psmith (developer) 2021-08-19 18:39 |
I'm assuming the title of this issue should reference the RATIONALE section not the RATIONAL section. I don't have a copy of the document that includes these line numbers so I can't quite understand what is being proposed here. The original proposal was to add this to the RATIONALE section which means it would not be binding and required (as I understand things) but instead only a suggestion, and probably reserving these targets for future implementations. As far as I can tell, though, these changes are being made to the "Extended Description" section, not the RATIONALE section, which makes these new features that are required of all conforming implementations. Am I wrong about that? As I said above, GNU make doesn't support the .WAIT feature. As I said above, it is likely not actually trivial to implement .WAIT in GNU make. I don't know how Sun make implements parallelism but in GNU make we don't require an entire branch of the graph to complete before moving to the next sibling branch, if parallel builds are enabled. For example: all: ONE TWO ONE: THREE FOUR ONE TWO THREE FOUR: ; echo $@; sleep 2 If we run this as "make -j4" then we'll get the following output: echo THREE; sleep 1 echo FOUR; sleep 1 echo TWO; sleep 1 THREE FOUR TWO echo ONE; sleep 1 ONE Note that GNU make runs TWO in parallel with THREE and FOUR, even though ONE has not completed yet. You can imagine that TWO itself is a complex dependency graph and the actual target built in parallel may well be a long ways away (in the graph). If we change this makefile to add the putative .WAIT: all: ONE TWO ONE: THREE .WAIT FOUR ONE TWO THREE FOUR: ; echo $@; sleep 2 what we'd like to see (one assumes) is: echo THREE; sleep 1 echo TWO; sleep 1 THREE TWO echo FOUR; sleep 1 FOUR echo ONE; sleep 1 ONE so that we'd run TWO in parallel with THREE, rather than waiting to start TWO until after FOUR is complete. This means we need to be able to detect a .WAIT target and stop searching the DAG at that point, but rather than just waiting for the previous dependencies to complete we want to move on to other siblings of the current target and see if there are any more targets that might be buildable, then come back later after THREE has completed. |
(0005448) rhansen (manager) 2021-08-19 18:55 edited on: 2021-08-19 19:04 |
> As far as I can tell, though, these changes are being made to the "Extended Description" section, not the RATIONALE section, which makes these new features that are required of all conforming implementations. Correct, that was our intention. I asked about making .WAIT non-normative during today's telecon (see line 68 of https://posix.rhansen.org/p/2021-08-19 [^] ), and the group decided that it was useful enough to include as a normative requirement. But thinking about this more, isn't: target: $(leftstuff) .WAIT $(rightstuff); do_thingequivalent to: $(rightstuff): $(leftstuff) target: $(leftstuff) $(rightstuff); do_thing? If so, two questions:
|
(0005449) shware_systems (reporter) 2021-08-19 19:27 |
If this was to be a TC type addition, then yes, keeping it as Rationale and Future Direction would be the way to go. However, as most modern CPUs do have multiple cores capable of parallelism, the consensus was more the future is now, rather than wait until Issue 9, to make them normative. Your concerns were part of the discussion, including a paraphrase of the example you provided, and the feeling was more non-trivial is not the same as non-doable too, it just may take longer for someone to do it, so was not a bar to adopting it. |
(0005450) psmith (developer) 2021-08-19 22:47 |
> But thinking about this more, isn't: ... equivalent to: Yes, basically: that's what I meant when I wrote in my comment last December, above: > I wonder if creating some kind of internal/implicit prerequisite relationship that wasn't visible to users would be a simpler way to implement this... or not. However your construction is not quite correct because it says that $(rightstuff) will be rebuilt if $(leftstuff) is rebuilt, which is wrong. If you use GNU make's "order-only prerequisite" feature, though, you could write it like this and it would work properly: target: $(leftstuff) $(rightstuff); do_thing $(rightstuff): | $(leftstuff) I think a real implementation of .WAIT is more effort than is implied by the term "syntactic sugar", but this seems to be a promising way to think about the problem. At this time I have no plans, or time, to implement either the new :::= operator, which I continue to maintain as per my comments in that issue is simply not useful, or certainly not useful enough to rise to the level of being added to the standard; or the .WAIT feature, which I agree is a good idea but which is a non-trivial amount of work. So, I guess GNU make will just become a non-conforming implementation until such time as someone puts forward the effort to change that. > this was to be a TC type addition, then yes, keeping it as Rationale and Future Direction would be the way to go. However, as most modern CPUs do have multiple cores capable of parallelism, the consensus was more the future is now, rather than wait until Issue 9, to make them normative. This seems like a strawman to me: I have not suggested that adding -j was a problem: indeed, I welcome the addition of -j and agree it allows better use of modern systems. However, people have been writing makefiles supporting parallelism using GNU make for 25+ years without the .WAIT capability. It definitely makes some (fairly rare) things easier, but there are other ways to manage those situations and not having .WAIT has not prevented massive build systems from being successful using parallel GNU make. > Your concerns were part of the discussion, including a paraphrase of the example you provided, and the feeling was more non-trivial is not the same as non-doable too, it just may take longer for someone to do it, so was not a bar to adopting it. Pretty much everything is a SMOP, in the end. If the only barrier to standardization is what is possible then indeed there's virtually no limit. |
(0005451) shware_systems (reporter) 2021-08-19 23:09 |
Re: 5448 No, I don't see them as equivalent. The version with .WAIT has the greater of left or right count pool removals starting in parallel. The replacement you suggest has potentially left count times right count starting that way. |
(0005452) psmith (developer) 2021-08-20 18:38 |
Sorry, I didn't understand the comment #5451: I don't know what a "left or right count pool removal" is, or why anything would be "left count times right count". The comment in #5448 IS an equivalent way to represent the .WAIT version with one significant difference, as I said in my comment: it declares that if any of the targets in $(leftstuff) are updated, then all the targets in $(rightstuff) will be considered out of date. Other than that it behaves identically. The extra dependency relationship is a fundamental change in the behavior, though, so that's not good. GNU make has a feature which avoids it. Nevertheless, the .WAIT version is easier to read and write so it's helpful. Just, not necessarily trivial to implement. |
(0005453) rhansen (manager) 2021-08-20 19:28 edited on: 2021-08-20 19:43 |
> If you use GNU make's "order-only prerequisite" feature, though, you could write it like this and it would work properly: TIL about GNU make's order-only prerequisites, thank you. It's good to know that all of the make implementations provide a way to order prerequisites, just with different syntax (.WAIT vs. pipe) and slightly different semantics (but similar enough that one can fairly easily convert a makefile from one format to the other). Given that all makes provide a way to order prerequisites, I think that adding the feature to POSIX in some form is a worthwhile activity. If everyone agrees, the questions I think we should consider futher are:
|
(0005454) shware_systems (reporter) 2021-08-20 19:39 |
The description of a pool is part of the resolution of Bug 1436, basically the number of data allocations necessary to manage parallelism when -j value greater than one. The count is the subset of that number actually needed to start the invocations the makefile specifies to bring targets or prereqs up to date. With: rightstuff=foo bar baz leftstuff=w x y z $(rightstuff):$(leftstuff) you can have 12 invokes started in parallel, with 3 each trying to update w, x, y, or z independently. If -j 10 specified 2 of those are waiting anyways so other executions complete. With the .WAIT version the max in parallel is four, the count of leftstuff. Then rightstuff can reuse the allocations leftstuff needed to be tracked as completed or not. |
(0005455) psmith (developer) 2021-08-20 20:20 |
Re 5454: Sorry but I don't see anything in issue #1436 which talks about "the number of data allocations" or anything that describes a "pool", and I still didn't really understand your comment. I think there's a fundamental confusion here. The makefile: foo bar baz: w x y z is 100% identical to writing: foo: w x y z bar: w x y z baz: w x y z That is, you're declaring 3 targets and giving them all the same 4 prerequisites. That's ALL it means. In fact once the makefile is parsed there's no way to differentiate the above two constructs based on the in-memory graph representation (at least, not in GNU make...) and that equivalency is required by POSIX. There's no change to this idea introduced by issue #1436 and -j. Given the above makefile then per invocation of make you may build a maximum of 7 targets (w, x, y, z, foo, bar, baz), and have a maximum of 4 jobs running in parallel (w, x, y, z). Never more than that. You will definitely never need to run 12 jobs, in parallel or not. If you add .WAIT, say between "w x" and "y z", you're only reducing the maximum number of possible parallel jobs because now make can't start y or z until after w and x are complete. But nothing about .WAIT or -j can ever increase the maximum number of targets that could be built. |
(0005456) psmith (developer) 2021-08-20 20:57 |
Regarding comment #5453: Just to point out that GNU make's order-only prerequisites is a more powerful construct than .WAIT. .WAIT is used when you can't allow two targets to run in parallel BUT at the same time you don't want them to depend on each other (if they could depend on each other you'd just use dependencies, not .WAIT). That's a different and fundamentally weaker guarantee than OO prerequisites provide: they give a true ordered relationship represented in the dependency graph, just with a differently colored edge that doesn't imply dependency. Consider this example: all: foo bar foo: one .WAIT two bar: two one Here we see that while we're building "foo", "two" won't be built in parallel with "one". However, while we're building "bar", "one" and "two" CAN be built in parallel because there's no .WAIT between them. If you set -j high enough (here, we only need -j2) then you'll still get "one" and "two" building at the same time; "one" is built as a dependency of "foo", then the updating of dependencies of "foo" is paused due to .WAIT so make continues looking for something else to build and finds "two" as a dependency of "bar"; there's no .WAIT here so make will go ahead and build "two", while "one" is still (potentially) running. Although easy to write, my opinion is that .WAIT is actually pretty hard to use in a reliable way due to this behavior. I consider it a kind of a hacky / quick-and-dirty shortcut that avoids implementing a robustly correct solution in your makefile. With order-only prerequisites you would write this makefile: all: foo bar foo: one one: | two bar: two one Here, "one" actually depends on "two" and that's represented in the dependency graph as an edge, so no matter where "one" appears it can never be updated before "two" is complete. However I was not really suggesting that OO prerequisites should be chosen instead of .WAIT: OO prerequisites were not actually created for this purpose (at least not directly). I'm just saying that their existence in GNU make allows people to implement sort-of-.WAIT-like features in GNU make makefiles, if they want to. And I'm also saying that, even though OO prequisites exist in GNU make, I'm not convinced it's completely straightforward to use it to implement .WAIT. However, I am open to patches implementing it (assuming the appropriate copyright assignment paperwork is completed etc.) |
(0005457) joerg (reporter) 2021-08-21 17:06 edited on: 2021-08-21 23:12 |
I am not sure whether the gmake method with the pipe symbol is able to be usable as a complete replacement for features offered by .WAIT. I also see no chance that this gmake specific method will ever be supported by other make implementations. So adding that method to POSIX would definitely weaken the POSIX standard. BTW: gmake still has various other bugs that prevent me personally from being able to use gmake in parallel mode. I am talking about a makefile system that offers portability to all currently available platforms (z/OS/EBCDIC just has been made working a month ago). The makefile system currently works without problems using smake and SunPro Make and (in serial mode) using gmake. The make implementation specific definitions are inside special files that are included based on the name of the make implementation in use. That concept of course requires to only have features that may be used in such a system without a need to have make implementation specific rules in the general makefiles. With these experiences in mind, I worked on the recent POSIX standardization to get something that matches the criteria for using different existing make implementations. With this makefile system and trying to use gmake in parallel mode, this are the current show stoppers: - gmakes calls readdir()/stat() at startup time and caches all timestamps. Unfortunately, gmake does never update these cached timestamps and as a result, it incorrectly claims that files are not present even though they just have been created from a rule that just has been executed - just because these files have been missing at startup time. The gmake bug tracking system claims that this bug has been fixed, but I cannot see a new version for testing, nor got a report from Paul that he did test whether that fix works for the schily makefile system. - missing include files with rules that allow to remake them in many cases do not work because gmake executes the rule commands in the inverse expected order and gmake does not offer a way to enforce the intended order. Paul has been given the chance to verify that gmake offers a way to deal with that problem by sending a patch to the affected makefiles that does not prevent other make programs to work with these makefiles, but he preferred to do nothing. The gmake bug tracking system claims that this bug has been fixed, but I cannot see a new version for testing, nor got a report from Paul that he did test whether that fix works for the schily makefile system. BTW: Other make implementations just work fine even though there is no .WAIT pseudo-target in use at the locations that currently fail with gmake. With the current version of gmake, I need to call gmake 5 times from the top level directory (starting from am fresh unpacked source) in order be able to finish with the compilation in parallem mode. If there is a fix for the above problems, it would really help to see a new gmake version now, since the last version is from January 2020. Finally, I remember that I had a discussion with Paul where he offered to implement .WAIT in case I implement .NOTPARALLEL into SunPro Make. I unfortunately cannot find that anymore. But given the fact that I spent 3 weeks for implementing things like ::= and .NOTPARALLEL into SunPro Make, even though these features are not needed because there are better existing methods, it would be a nice gesture from Paul to spend some time in making gmake more compatible to other make implementations. .WAIT is available in SunPro Make and BSD make since 30 years and that should be a verification for a useful feature. |
(0005458) rhansen (manager) 2021-08-23 06:41 edited on: 2021-08-23 06:45 |
In response to Note: 0005456:> Although easy to write, my opinion is that .WAIT is actually pretty hard to use in a reliable way due to this behavior. I consider it a kind of a hacky / quick-and-dirty shortcut that avoids implementing a robustly correct solution in your makefile.OK, it seems like my understanding of how .WAIT works was wrong. If .WAIT implementations behave as you describe, then you have a very compelling point. Note: 0005446 was written under the assumption that .WAIT created an order-only prerequisite relationship: "prerequisites that appear after it shall not be brought up-to-date until after all preceding prerequisites for the same target have been brought up-to-date." If existing implementations behave as you describe, then none of them would be conforming. I'm now leaning against including a normative requirement for .WAIT, at least as it is currently written. (Something in "future directions" could still be useful.) If existing implementations change .WAIT so that it creates an ordered relationship in the dependency graph like GNU make's OO prerequisites feature, then I'd be more interested in adding .WAIT. And if that happened, then it would be easier to implement in GNU make, yes? In response to Note: 0005457: > I am not sure whether the gmake method with the pipe symbol is able to be usable as a complete replacement for features offered by .WAIT.I now see that there isn't a 100% semantic match, but why wouldn't it be usable as a replacement for .WAIT? I agree with Paul; it seems stronger than .WAIT to me. > BTW: gmake still has various other bugs that prevent me personally from being able to use gmake in parallel mode.Please keep the discussion on-topic. Your bugnote derails the conversation, which makes it harder to properly address this issue. > .WAIT is available in SunPro Make and BSD make since 30 years and that should be a verification for a useful feature.This argument is disingenuous. First, there are lots of old features that aren't standardized. Second, GNU make is the 800-pound gorilla here, and it became the 800-pound gorilla despite not having .WAIT. It is disrespectful to GNU make's sizeable userbase to not take Paul's input seriously; if he doesn't see a compelling reason to add .WAIT then we should take a moment to really consider its inclusion. Maybe the other make implementations should learn from GNU make's success and replace .WAIT with a clone of GNU make's OO prerequisites feature. (I'm kidding. Mostly.) |
(0005459) joerg (reporter) 2021-08-23 12:36 |
Re: Note: 0005448 There is a fundamental difference between target: $(leftstuff) .WAIT $(rightstuff); do_thing and $(rightstuff): $(leftstuff) The first one says "build leftstuff before rightstuff" The second one says "build rightstuff from leftstuff" That alone would make both variants non-equivalent. But even worse, if you have: a: l echo a a b c: x y z This results amongst others in this rule: a: l x y z echo a which is what the standard requires but not what you expect in this case... as you may have unexpected rule commands involved. From a theoretical view, problems in a parallel build are missing dependencies that would need to be defined. From practical experience it is usually hard to find the missing dependency in special if the dependency graph is large already. .WAIT on the other side delivers the method that fits to the observed problems and does not create rules, you do not want to have. If you need more than one .WAIT in a dependency list, this could be (if at all) only written as many dependencies that at some point cannot be understood anymore and of course enhances the chance that they are unexpectedly added as dependencies to existing rule commands as in the example above. |
(0005460) psmith (developer) 2021-08-23 20:18 edited on: 2021-08-23 21:07 |
In response to Note: 0005458 My understanding of the text "prerequisites that appear after it shall not be brought up-to-date until after all preceding prerequisites for the same target have been brought up-to-date." in particular "for the same target", is that it's specifically intended to limit the effects of .WAIT to the prerequisites within a given target. If the same target appears as a prerequisite of a different target, then the .WAIT doesn't (necessarily) apply. However, I suppose this statement is open to interpretation and maybe this should be addressed in the standard, to reduce the chance of confusion. In response to Note: 0005457 We have had this discussion before. If you do want to have it again we should restrict it to the mailing list (or better yet, the GNU make mailing lists) and not have it here. The statements are at least misleading. All I will say here is that, IF you state all prerequisites of your targets explicitly in your makefiles then GNU make will rebuild your targets 100% accurately 100% of the time, regardless of -j level. If you write your rules so that they have "undeclared side effects" that other targets rely on, then it's true that your targets may not build correctly in some situations in GNU make. I understand that your makefile systems rely heavily on this type of "undeclared side effect", so it is hard to port them to be usable by GNU make. I have described the changes you'd need to make to convert the "undeclared side effects" into declared explicit prerequisites and you have declined to make those changes in order to facilitate the portability, and instead decided to run make multiple times, which is perfectly fine. I have agreed to make some changes to GNU make to reduce this issue, and those changes are in fact checked into the Git source code control for the next release of GNU make. These changes are not very important to the current userbase of GNU make, however, because GNU make makefiles generally DO explicitly declare all their prerequisites and so don't run into these problems. In response to Note: 0005459 > From a theoretical view, problems in a parallel build are missing dependencies that would need to be defined. From practical experience it is usually hard to find the missing dependency in special if the dependency graph is large already. I simply disagree with this. If you know what a target does, then it should be quite clear what its prerequisites are. I need to build a target FOO. I know what commands are needed to build FOO. I know what other targets in my makefile need to be available to run those commands. Therefore I know which targets need to be created before FOO's commands can be run. So I declare those targets as prerequisites of FOO. There's no magic or deep insight needed here. Maybe if you are limited to .WAIT, where you have to actually modify the prerequisites list of a specific target and list things in a specific order, and especially you have to add the .WAIT to a rule (or all rules) that itself lists FOO as a prerequisite rather than declaring the relationship on FOO itself, it can be more difficult. However, with order-only prerequisites you can define the prerequisite relationship directly between FOO and whatever targets it requires, ANYWHERE in the makefile, just as you can define normal prerequisite relationships anywhere. For example if it's easier you can define the relationship next to the prerequisite, rather than next to the target. |
(0005461) joerg (reporter) 2021-08-23 20:39 |
Re: Note: 0005458 First, the arguments against .WAIT are just wrong. In contrary, the gmake method is unreliable to use, see Note: 0005459 This is because it is hard to use a feature reliably that is hard to understand in case that the makefile, where it is used, already has a complex dependency graph. Most makefiles from today match that criteria. If a makefile fails in parallel mode, you know where it fails and thus you know where to add a .WAIT, but you usually do not know how/where to add more dependencies to satisfy make's wishes. Regarding .WAIT in combination with -j, the effects to parallel execution are the same as with the gnu only method. This is because the sum of all make instances that run in a common make pool share a common job pool and if one targes cannot start a new job for certain dependencies for a limited time, then other targets in the same make instance or other targets in other make instances in the same make pool will run other jobs from the available slots in the job pool. Finally, gmake definitely is not the gorilla that convinces because of it's features but it is rather used because a lot of vendors only offer/advertize gmake to their customers. Gmake offers a lot of features that do not give added value and deviates in it's behavior from other make implementations for features that did exist long before gmake was initially written. In case you did not get the background of the sum of all feature enhancements to "make" in POSIX that I worked on in the past years, this is only in order to avoid vendor lock in effects to specific make implementations and to foster more commonly usable features. If we (after adding .WAIT) add the simplified pattern rules I mentioned in Note: 0005126 from 0000513 and BSD make would implement that, we would have the minimum set of features that are needed for todays makefiles and SunPro Make, smake, gmake, bmake would offer a sufficient common set of features to finally come to the usual POSIX grant of portability for the make utility as well. |
(0005462) rhansen (manager) 2021-08-23 20:58 edited on: 2021-08-23 21:10 |
Re: Note: 0005459:> There is a fundamental difference betweenI know; Paul already explained the difference in Note: 0005450. In Note: 0005453 I was proposing adopting GNU make's order-only prerequisites, not regular prerequisites.target: $(leftstuff) .WAIT $(rightstuff); do_thingand$(rightstuff): $(leftstuff) Re: Note: 0005460: > However, I suppose this statement is open to interpretation and maybe this should be addressed in the standard, to reduce the chance of confusion.Agreed. The intention (or at least *my* intention) was to establish an order-only prerequisite in the dep graph. Either way, the wording needs to be clarified, if it is to be kept at all. Do you think there would be more interest in implementing .WAIT in GNU make if it did establish an order-only prerequisite? For example, the following: $(a): $(b) .WAIT $(c) .WAIT $(d); do_thingwould be 100% equivalent to: $(c): | $(b) $(d): | $(c) $(a): $(b) $(c) $(d); do_thing |
(0005464) shware_systems (reporter) 2021-08-24 04:03 edited on: 2021-08-24 04:05 |
Re: 5455 With: rightstuff=foo bar baz leftstuff=w x y z $(rightstuff):$(leftstuff) target: $(rightstuff) // missing line // gets leftstuff up to date indirectly is where you get the 12, as the rightstuff as prereq of target starts 3 parallel invokes, and these are put on hold while the 4 prereqs of each invoke are evaluated, all in parallel. Before the initial ones go on hold all it cares about is all the prereqs for its particular target are out of date, they have no way to determine that the other targets also have the same prereqs, as if they were on separate lines like you expanded. |
(0005465) psmith (developer) 2021-08-25 12:53 |
Re: 5464 I'm assuming here you meant to have the "target" rule come before the "$(rightstuff):$(leftstuff)" rule, or else invoke "make target". The method you suggest is not how make works. Make creates a directed graph and walks the graph. It doesn't try to build any target (node in the graph) until AFTER all that target's prerequisites are completely created. Here we have a graph with a root of "target" and 3 sub-nodes, "foo", "bar", and "baz". There are then 4 leaves, "w", "x", "y", and "z". Each of the three sub-nodes connect to each of the four leaves. When make runs it walks the graph from whatever starting point is chosen using a depth-first, right-to-left ordering among siblings. So if we start at "target" then first it will build "w", then "x", then "y", then "z", then "foo" (because now foo's prerequisites are complete), then "bar" (because "bar"'s prerequisites were already built, as they are also prerequisites of "foo"), then "baz", and finally "target" (the root of the graph). With -j the walk of the graph is identical as without -j, only instead of waiting for each node to complete before moving on to the next one we start N nodes (for -jN) in parallel before we are forced to wait (or, we wait when there are no nodes left that can be built until after already-in-process nodes are complete). |
(0005466) psmith (developer) 2021-08-25 14:19 |
Re Note: 5462 I don't think it will work to declare that equivalency, because I don't think these things are equivalent. The problem is that order-only prerequisites create an explicit "always present" relationship in the graph, and .WAIT does not. Suppose we have this: all: one .WAIT two one two: ; @echo $@ versus this: all: one two two: | one one two: ; @echo $@ What happens if you run "make two"? In GNU make you have declared a dependency relationship here, so make will first build "one" (if needed) and then it will decide to build "two" or not, without considering whether "one" was updated (the relationship is "order-only" so it doesn't impact the "out of date" decision). In the .WAIT version, only "two" will be built. "one" will not be considered, because "one" is not a dependency of "two". However, I've downloaded FreeBSD make (the only make I could easily install that supports both -j and .WAIT) and it appears the behavior of .WAIT is different than I thought. I don't know whether this behavior is specific to the FreeBSD version or not. Let's consider this makefile: all: foo bar foo: one .WAIT two bar: one two foo bar one two: ; @echo $@; sleep 1 Suppose we invoke this makefile with -j10 so that the -j value is not a limiting factor. What do we expect to build in parallel, and where will we wait? My original assumption was that the .WAIT is a feature of the prerequisite list of the target, so that we would try to build "foo", which would build "one", then it would not build "two" here because "one" was still running. Then make would try to build "bar", and see that "one" and "two" could be built, and since there's no .WAIT here it would build "two" immediately. So I assumed that the output would be this: one two bar <pause> foo However, that's not what FreeBSD make does. It seems that this .WAIT does indeed create some kind of globally visible relationship between "one" and "two", so what I see is: one <pause> two <pause> foo bar This relationship persists even if we don't build "foo" at all! If I run "make bar" here we still get: one <pause> two <pause> bar The implementation has some strange subtleties. If I change the graph so that "two" appears before "one" in the graph, FreeBSD make still runs "one" first and waits for it: all: bar foo bar: two foo: one .WAIT two one two foo bar: ; @echo $@; sleep 1 In this graph, make will consider targets in the order "two", "bar", "one", "foo". But running this gives: one <pause> two <pause> bar foo So, somehow FreeBSD make realizes, when it wants to build "two", that sometime in the future it will also want to build "one" and that "one" should be completed before "two" starts. This is very mysterious to me. Certainly implementing something like that in GNU make will be very difficult, because GNU make doesn't "look ahead" while walking the graph. It walks the graph in order and simply goes from beginning to end; at the time that make wants to build "two" it doesn't know whether it will want to build "one" or not. And of course note that in GNU make, where we can have long chains of implicit rules which are not resolved until runtime, the full shape of the graph can't even be known until we actually try to resolve the implicit rules. That "one" dependency might be found through a long chain of implicit rules, and not be known until all that work is done. However maybe I'm missing something. In any event it's clear to me that the current wording proposed here certainly does not capture the actual behavior of FreeBSD make's .WAIT with anything close to specificity. I think that if we want to add .WAIT to the standard we will have to make it much LESS specific, so that all the subtleties are not actually required by POSIX and instead we create the simplest possible behavior. |
(0005468) joerg (reporter) 2021-08-25 15:34 edited on: 2021-08-25 15:49 |
Re: Note: 0005466 Please note that SunPro Make is easier to compile/install than BSD make. Just fetch a recent schilytools from http://sourceforge.net/projects/schilytools/files/ [^] today, the most recent one is: https://sourceforge.net/projects/schilytools/files/schily-2021-08-14.tar.bz2 [^] unpack that archive chdir into the tree and call "make", then "make install" Then call either /opt/schily/bin/make or /opt/schily/bin/dmake "make" is the variant that is installed in the base OS and "dmake" is the variant that by default runs in parallel mode. BSD make is not fully POSIX compliant, SunPro Make of course is, as it has been certified with Solaris. BTW: where did you get a compilable FreeBSD make from? I am not aware of such a source.... I verified the makefile from your example and both SunPro Make and bmake (The portable version from NetBSD make) behave as you expect. They print one and two with no delay and then wait, before printing bar and foo. |
(0005469) psmith (developer) 2021-08-25 17:52 |
I installed the pre-built freebsd-buildutils package onto my GNU/Linux system. It contains an "fmake" program. I installed the schily version. It didn't quite build so easily, but I was able to build it. However, there is no "dmake" or "make" in the bin directory (or anywhere else in /opt/schily) after I run install. I was able to find an uninstalled command ./sunpro/Make/bin/make/common/OBJ/x86_64-linux-gcc/make in the build directory and that one does support .WAIT (unlike the "smake" program). Although the output is a little bit weird (to me), I can see that it does indeed behave the way I expected in my note 5466, and not the way FreeBSD make works. |
(0005470) joerg (reporter) 2021-08-25 18:03 edited on: 2021-08-25 18:09 |
Well you did probably not run "make install" as root. If your Linux system contains unknown modifications that prevent compilation of single programs, I recommand to use "make -i" and to make a bug report. This is an actively maintained project that responds to bug reports. smake currently does not support parallel compilation and since .WAIT was added to POSIX recently, there is no support yet. This is why the schily makefile system uses $(WAIT) that may be empty. BTW: SunPro Make always collects the output from rule commands and prints serialized output, so output of course looks different from simple make implementaions. |
(0005471) psmith (developer) 2021-08-26 13:09 |
I never install software I download and build myself as root. I have a separate user account I use for all that. The problem I ran into is that I built with GNU make, then during it build it attempted to invoke the make that was built (I suppose smake), but there were GNU make options in MAKEFLAGS that weren't recognized by the make that was built and it failed. However that was my own fault because I have some MAKEFLAGS options I set in my environment before I invoke make. Re Note: 5461 We'll just have to disagree about which feature is easier to use. Nothing said here convinces me that .WAIT is easy to use: on the contrary to me it seems very difficult to use RELIABLY, EXCEPT perhaps in the very limited context that the BSD build system uses it in (where it's used to mix in pause points into prerequisite lists contained in variables like SUBDIRS). > In contrary, the gmake method is unreliable to use, see Note: 0005459 There are no examples in that note showing that OO prereqs are hard to use. In fact, there are no OO prereqs mentioned in that note at all. > This is because it is hard to use a feature reliably that is hard to understand in case that the makefile, where it is used, already has a complex dependency graph. There is nothing about OO prereqs that make them hard to understand in complex dependency graphs. On the contrary, since OO prereqs create an actual edge in the graph while .WAIT does not, the behavior of OO prereqs are easier to understand. > If a makefile fails in parallel mode, you know where it fails and thus you know where to add a .WAIT, but you usually do not know how/where to add more dependencies to satisfy make's wishes. This is exactly the reverse of reality. When a target fails, you know what target failed and you know why it failed (which prerequisite did not exist that should have existed). With GNU make, you merely add this to your makefile: target: | prereq and now you've added a new edge to your graph that ensures that prereq will be complete before the target is invoked. Note that unlike normal prerequisites, order-only prerequisites DO say only "build prereq BEFORE target" and they do NOT say "build prereq FROM target". That's what makes them different from "normal" prerequisites. But unlike .WAIT, this relationship is defined as an edge in the graph between the target and prerequisite so it always exists, regardless of where target is used in the makefile. I'm not exactly sure what you mean by "where to add more dependencies"; like any other dependency relationship it can be added ANYWHERE in the makefile. On the other hand, to solve this issue with .WAIT you must search all your makefiles and envision the entire dependency graph so you can be sure that you have found all the other dependency lists that contain "target", and which can be reached from a place where "prereq" might have been invoked first, so you can add a .WAIT somewhere between "target" and "prereq". This is because .WAIT is not associated with target and prereq, and it doesn't apply to other places where these targets might exist. You can see this issue even with my very simple example above: I didn't add the .WAIT between the "one" and "two" targets when they were prerequisites of "bar", and so they were built in parallel. With .WAIT I have to search the entire graph looking for these types of relationships (which, unlike my example, could be very far apart in the graph and hard to see in the makefiles). Alternatively, you could try to simplify things by just searching all makefiles to find all places where "target" appears as a prerequisite and change them all to ".WAIT target", which would solve the problem but could introduce many more wait points than are needed or optimal. And, of course, when we consider things like pattern rules etc. it might not always be obvious at all that a given prerequisite list will have "target" as a prerequisite. |
(0005472) joerg (reporter) 2021-08-26 19:23 edited on: 2021-08-26 19:24 |
Re: Note: 0005466 I did test your makefiles on a FreeBSD instance using "make". I canot repeat the problem you describe, however both makefiles are written in a way that makes them instable as their behavior depends on chance when in parallel mode. On FreeBSD, I see a comparable behavior with SunPro Make and BSD make. Re: Note: 0005471 When you do not use the official makefiles of a trustworthy project, it is up to you, to do the right things. This is something you cannot blame the project for. Even the Linux distros, that support binary packages based on schilytools use "make install" (see README.install) to create the prototype trees for their packages. If you are using a private environment MAKEFLAGS, you are doing something that is being warned for. If your MAKEFLAGS environment even contains non-standard content, you should never expect that to work. Smake implements workarounds that deal with the non-POSIX structure from gmake's MAKEFLAGS, but it does not support to work around non-standard gmake options. And BTW: you are right, if you call "make" at top level, this first compiles a bootstrap smake and then continues with that compiled smake in order to be able to compile schilytools with make implementations that do not support the minimum feature set for the makefile system. |
(0005473) rhansen (manager) 2021-08-26 20:22 |
There was some spirited discussion about this bug during today's telecon. For now, we're going to wait to see if Steffen (or anyone else) can assemble a rough proof of concept patch for GNU make that takes an approach acceptable to Paul. If so, then it seems much more likely to us that GNU make would eventually support .WAIT. That would alleviate concerns about indefinite non-conformance by a major implementation. |
(0005474) psmith (developer) 2021-08-26 20:40 edited on: 2021-08-26 20:52 |
I'm not sure which "problem" you refer to for FreeBSD make. Are you saying your FreeBSD make doesn't put the "pauses" in the same place as mine? Let's be very clear. If I have this makefile: all: bar foo bar: two one foo: one .WAIT two one two foo bar: ; @echo $@; sleep 10 where I've increased the sleep time to be more obvious even on slow or busy systems, and I run FreeBSD make via "time", I get this: $ time fmake -j10 one <pause> two <pause> bar foo <pause> real 0m30.018s user 0m0.019s sys 0m0.005s It takes 30 seconds to run this makefile. If I do the same with the SunPro make from schily, I get this: $ time ./schily/schily-2021-08-14/sunpro/Make/bin/make/common/OBJ/x86_64-linux-gcc/make -j10 make: Warning: Can't find `make.rules': No such file or directory dmake: defaulting to parallel mode. local --> 1 job local --> 2 jobs <pause> local --> Job output two local --> Job output one local --> 1 job local --> 2 jobs <pause> local --> Job output bar local --> Job output foo real 0m20.018s user 0m0.019s sys 0m0.008s and the total time taken is 20 seconds. Are you saying you get different behavior from your version of FreeBSD make? That it also takes 20 seconds, not 30 seconds? I don't care about the _order_ in which the output appears, that's irrelevant. The important thing for this experiment is how many pauses there are: what is run in parallel and what is run serially. > If your MAKEFLAGS environment even contains non-standard content, you should never expect that to work. Uh, and...?? I did say, "However that was my own fault because I have some MAKEFLAGS options ..." |
(0005475) psmith (developer) 2021-08-26 20:50 |
My main concern at this point is that the text added to the standard be very clear about what is required by the standard, and what is left unspecified. See my example above and how the two different make versions that already support .WAIT behave. I assume that the text means to say that .WAIT only must take effect between two sets of dependencies when they are processed as prerequisites of the specific target where the .WAIT appears in a prerequisite list. If those same two targets appear as dependencies of some other target where .WAIT is not present, then the standard makes no statement about whether they should be ordered or not. Let me offer up a potential implementation and you can say whether it's intended to be allowed by the standard: Suppose a version of make is created where, when it sees a .WAIT in a prerequisit list, it pauses and waits for ALL currently-running jobs to complete, then keeps going. Ignoring quality of implementation issues, would such an implementation be POSIX conforming under the text being proposed? |
(0005476) rhansen (manager) 2021-08-26 21:00 edited on: 2021-08-26 21:08 |
Re: Note: 0005474> Are you saying you get different behavior from your version of FreeBSD make? That it also takes 20 seconds, not 30 seconds?Using the Debian bmake-20181221-2 package (supposedly copied from NetBSD): $ time bmake -f - -j10 <<\EOF all: bar foo bar: two one foo: one .WAIT two one two foo bar: ; @echo $@; sleep 10 EOF --- two --- --- one --- --- two --- two --- one --- one # pauses here for 10s --- bar --- --- foo --- --- bar --- bar --- foo --- foo # pauses here for 10s real 0m20.032s user 0m0.018s sys 0m0.010sMaybe this is a bug in NetBSD's make? Re: Note: 0005466 > [...] order-only prerequisites create an explicit "always present" relationship in the graphOnce again my understanding of how implementations actually behave is upended. :) I was under the impression that "|" only affected when the stuff on the right was built, not whether it was built. For example, I had assumed that the following would not build 'one': $ make -f - -j10 two <<\EOF two: | one one two: ; @echo $@; sleep 2; echo done sleeping $@ EOFGiven that GNU make does build 'one', the name "order-only prerequisite" seems like a misnomer. Maybe "existence-only prerequisite" would be a better name. If GNU make did have a true order-only prerequisite feature, then I think that .WAIT could be implemented as syntactic sugar. The resulting behavior would be more conservative than BSD's .WAIT behavior (less parallelism), but that's OK. Re: Note: 0005475 > My main concern at this point is that the text added to the standard be very clear about what is required by the standard, and what is left unspecified.Agreed. I think everyone understands that the wording in Note: 0005446 is problematic. Coming up with precise and correct wording might be tricky, but now that we collectively understand the problem better I am confident that we can come up with something suitable. > [...] would such an implementation be POSIX conforming under the text being proposed?Yes, POSIX should allow that behavior. Implementations should always be permitted to have less concurrency than is technically possible. For example, POSIX should allow an implementation to dynamically reduce parallelism if the current system load is too high. |
(0005477) joerg (reporter) 2021-08-26 21:12 edited on: 2021-08-26 21:17 |
Re: Note: 0005474 Running the FreeBSD make on FreeBSD with your makefile indeed results in a total time of 20 seconds for the make run. This is the FreeBSD-13 beta from September 24 2020. --- two --- --- one --- --- two --- two --- one --- one delay... --- bar --- --- foo --- --- bar --- bar --- foo --- foo delay... Since even the output looks different from yours, you seem to use a different make implementation. P.S. The results have been verified to be identical to the results fetched with a FreeBSD installation from April 2021. |
(0005478) psmith (developer) 2021-08-28 12:42 |
RE Note: 5476 > Given that GNU make does build 'one', the name "order-only prerequisite" seems like a misnomer. Maybe "existence-only prerequisite" would be a better name. It's possible I've been living with it too long (order-only prerequisites have existed in GNU make for almost 20 years, having been added in GNU make 3.80 in 2002) but I don't agree. The term "prerequisite" to me means that it must be brought up to date before "target" can be built. The "order-only" modifier means that only the order of targets, but not the out-of-date computation, is relevant for that edge in the graph. If a feature were created that told make that "targetB" should be built before "targetA" but if and only if "targetB" needed to be built for some other reason, that shouldn't be called a prerequisite at all IMO. You would need some other term that didn't contain "prerequisite". Such a feature wouldn't be so easy to create, but could be done. It would need to have some kind of "undefined state" edge in the graph which, when seen, would mean that the target node could not be built until that state was no longer undefined. The state would change to defined when either (a) the prerequisite node were considered (for some other reason than this undefined edge), or else (b) the entire graph has already been walked: at that time all "undefined" edges would revert to "considered with no changes" and we would have to build all the target nodes that could not previously be considered due to undefined states. Or, something like that. |
(0005479) joerg (reporter) 2021-08-28 15:15 |
If that gmake feature has been added late, I did expect gmake to see beyond the edge of the plate and to check what other make implementations did offer as solution already instead of introducing something incompatible new. UNIX was a success story because implementors from vendor A did look at implementations from vendor B and reimplemented goot ideas. In the 1980s a good idea from another vendor typically did take 2-4 years to appear everywhere. With that background, .WAIT should have been implemented in gmake no later than 1995. |
(0005480) psmith (developer) 2021-08-28 17:21 |
As I hope the discussion here makes obvious, .WAIT is not the same thing as an order-only prerequisite. As far as I'm aware there isn't anything similar to OO prerequisites in other versions of make so GNU make isn't implementing this feature in some incompatible way to other versions of make that already supported it. The question of whether or not GNU make supports .WAIT for compatibility with other versions of make is a completely separate issue. |
(0005481) steffen (reporter) 2021-08-28 19:21 |
I added a Public Domain diff to make(1) that may be used freely shall it can be applied to the source of your program. It implements a true .WAIT that ensures the chosen ordering of prerequisites is stable across the entire makefile. For example, for the example above: <code> #?0|kent:tmp$ cat y/makefile all: bar foo bar: two one foo: one .WAIT two one two foo bar: ; @echo $@; sleep 3 #?0|kent:tmp$ ~/src/.gmake.git/make -j4 -f y/makefile one two bar foo </code> or for the more sophisticated <code> #?0|kent:z$ cat makefile all: lib ham .WAIT bin bin2 ;@echo all command clean: ; rm -f bin/.stamp lib/.stamp lib: lib/.stamp; @echo lib command ham: ;@echo ham command lib/.stamp: ;cd lib && $(MAKE) bin: bin/.stamp; @echo bin command bin/.stamp: ;cd bin && $(MAKE) bin2: ; @echo bin2 command #?0|kent:z$ ~/src/.gmake.git/make -j4 ham command cd lib && /home/steffen/src/.gmake.git/make make[1]: Entering directory '/tmp/z/lib' lib-x command lib-x after sleep lib/all command make[1]: Leaving directory '/tmp/z/lib' lib command cd bin && /home/steffen/src/.gmake.git/make bin2 command make[1]: Entering directory '/tmp/z/bin' bin-x command bin/all command make[1]: Leaving directory '/tmp/z/bin' bin command all command </code> It may be up to the core developers to leave the exact meaning of .WAIT an implementation issue, but i think this way of doing things -- extending a chosen barrier to be global -- is right. I want to note that this likely does not cause issues in practice, since Makefiles yet making use of .WAIT are likely ordered in a way that would avoid the race condition which does occur if make(1) does not take care for ensuring barrier validity gracefully. Since most use cases of .WAIT are in the BSD make system this is a non-issue, since there so-called "singleton"s (thanks Paul Smith) are used, and .WAIT thus occurs naturally only to guarantee correct ordering in this singleton. P.S.: survived -fsanitize=address testing. |
(0005482) joerg (reporter) 2021-08-29 12:37 |
With the proposal from Steffen, we would need to add a hint to the standard that a makefile like this: all: bar foo bar: two one foo: one .WAIT two one two foo bar: ; @echo $@; sleep 3 is non-conforming, since the target "bar" and it's prerequisite list is in conflict with the definitions for target "foo". Steffens code causes an implicit dependency and even though this may result in working code while using gmake, it will not work with SunPro Make and BSD Make in case that "all" is used as the main target to build. This is because SunPro Make and BSD make evaluate .WAIT at run time, based only on the list of dependencies for the current target. |
(0005484) rhansen (manager) 2021-09-02 06:14 edited on: 2021-09-02 06:17 |
Re: Note: 0005478> You would need some other term that didn't contain "prerequisite".Good point. Re: Note: 0005482 > [...] the target "bar" and it's prerequisite list is in conflict with the definitions for target "foo".How so? It looks OK to me. Some experiments with Steffen's patch with the following makefile: Building 'all' works as expected:all: bar foo bar: two one foo: one .WAIT two one two foo bar: ; @echo $@; sleep 2 Building just 'bar' causes 'one' to be built before 'two' even though the BSD makes build 'one' and 'two' in parallel. I personally prefer this behavior over the BSD make behavior, so I don't consider it to be a bug. Output:$ time ./install/bin/make -j10 -f - <<\EOF all: bar foo bar: two one foo: one .WAIT two one two foo bar: ; @echo $@; sleep 2 EOF one <2s pause here> two <2s pause here> bar foo <2s pause here> real 0m6.021s user 0m0.019s sys 0m0.004s Building just 'two' causes 'one' to be built. This feels like a bug to me, and it doesn't match the behavior of BSD makes. Output:$ time ./install/bin/make -j10 -f - bar <<\EOF all: bar foo bar: two one foo: one .WAIT two one two foo bar: ; @echo $@; sleep 2 EOF one <2s pause here> two <2s pause here> bar <2s pause here> real 0m6.028s user 0m0.012s sys 0m0.016s Based on the above, it looks like the patch effectively converts .WAIT into order-only prerequisites.$ time ./install/bin/make -j10 -f - two <<\EOF all: bar foo bar: two one foo: one .WAIT two one two foo bar: ; @echo $@; sleep 2 EOF one <2s pause here> two <2s pause here> real 0m4.022s user 0m0.014s sys 0m0.008s |
(0005485) psmith (developer) 2021-09-02 12:23 |
Re Note 5482: I don't think I'd like to say that a makefile like that is non-conforming. Rather I think we should say that .WAIT is only guaranteed to create a wait point in the prerequisites while building the target in which the .WAIT appears. It's undefined (by the standard) whether it will add a wait point between these prerequisites if they appear in other rules. Maybe that amounts to the same thing, more or less. |
(0005486) joerg (reporter) 2021-09-02 13:22 edited on: 2021-09-02 13:25 |
Re: Note: 0005484 Your makefile (using dmake == SunPro Make) ==> mostly starts with two on Solaris mostly starts with one on FreeBSD on a RaspPi more or less alternatingly starts with one or the other on Linux This is because the "all: bar foo" dependency has no order and bar has no order as well. Another important reason for that behavior is the different fork() timing on the different operating systems. I believe it is important that if we allow implementations that use a prerequisite-like implementaion, the standard explicitely mentions that in such a makefile, the order is undefined. |
(0005487) shware_systems (reporter) 2021-09-02 15:11 edited on: 2021-09-02 15:26 |
Re: 5486 This applies to non-parallel builds too, that I see, there's no requirement prereqs are evaluated left to right, just commands top to bottom. An order can be specified for both in the standard as to when they start, but a parallel build still has no control over when a prereq finishes updating. |
(0005488) psmith (developer) 2021-09-02 15:30 |
Re Note 5487: No, you're not correct here. The standard explicitly says: > The make utility shall treat all prerequisites as targets themselves and recursively ensure that they are up-to-date, processing them in the order in which they appear in the rule. Prerequisites MUST be considered in the order in which they are listed in the makefile. Also, I'm not sure I understood your second sentence but note make orders things by when jobs COMPLETE, not by when they start. |
(0005489) rhansen (manager) 2021-09-02 16:28 edited on: 2021-09-02 16:32 |
Here's a new attempt: On page 2975, insert before line 98708: .NOTPARALLELThe application shall ensure that this special target is specified without prerequisites or commands. When specified, make shall update one target at a time, regardless of whether the -j maxjobs option is specified. If the -j maxjobs option is specified, the option shall continue to be passed unchanged to sub-make invocations via MAKEFLAGS. On page 2975, insert before line 98742: .WAITThe application shall ensure that this special target, if specified as a target, is specified without prerequisites or commands. When .WAIT appears as a target, it shall have no effect. When .WAIT appears in a target rule as a prerequisite, it shall not itself be treated as a prerequisite; however, make shall not recursively process the prerequisites to the right of the .WAIT until the prerequisites to the left of it have been brought up-to-date. Implementations may also enforce the same ordering between the affected prerequisites while processing other target rules that have some or all of the same affected prerequisites. On page 2975, change line 98742 from: The special targets .IGNORE, .POSIX, .PRECIOUS, .SILENT, and .SUFFIXES shall be specified without commands.to: The special targets .IGNORE, .NOTPARALLEL, .POSIX, .PRECIOUS, .SILENT, .SUFFIXES and .WAIT shall be specified without commands. On page 2975 lines 98744-98747, change: Targets with names consisting of a leading <period> followed by the uppercase letters "POSIX" and then any other characters are reserved for future standardization. Targets with names consisting of a leading <period> followed by one or more uppercase letters are reserved for implementation extensions.to: Targets and prerequisites consisting of a leading <period> followed by the uppercase letters "POSIX" and then any other characters are reserved for future standardization. Targets and prerequisites consisting of a leading <period> followed by one or more uppercase letters, that are not described above, are reserved for implementation extensions. On page 2985 after line 99167 (make EXAMPLES) append a new example: 7. With the following makefile, <tt>make -j 10 all</tt> may bring <tt>one</tt> and <tt>two</tt> up-to-date in parallel despite the .WAIT in the prerequisite list for <tt>foo</tt>. This is because the .WAIT does not stop make from recursively processing <tt>bar</tt> and its prerequisites in parallel. However, if only <tt>foo</tt> is specified (<tt>make -j 10 foo</tt>), make will wait for <tt>one</tt> to be brought up-to-date before bringing <tt>two</tt> up-to-date. Note also that the .WAIT does not create a prerequisite relationship between <tt>one</tt> and <tt>two</tt>, so <tt>make -j 10 two</tt> will not build <tt>one</tt>.all: foo bar foo: one .WAIT two bar: one two foo bar one two: ; @echo $@ |
(0005490) shware_systems (reporter) 2021-09-02 16:36 edited on: 2021-09-02 16:43 |
Re 5488: Yes, that was pointed out during today's phone call; I was looking at the Target Rule text more, skipped last part of that sentence. make has no control over, once it establishes what prereqs of a target are not up-to-date, how long the commands specified to bring them up-to-date will need to execute. An early prereq may be a compile that takes 10 minutes, the last one may simply be a touch that takes milliseconds. Processed as parallel targets that last prereq will finish first is what I was getting at. |
(0005492) steffen (reporter) 2021-09-02 23:25 |
Re: 5484 Well ok, then this is even easier. The patch easier.diff should do just that. With my example it still prints "lib command" before stepping over to "bin" (different to unpatched make), but the "two one" example happens in that order (but when used for "foo", but then race free). It surely could be optimized, and it likely is possible to extend this to global level nonetheless (with some effort). Re 5489: That example is false, then, it should be "bar: two one" Ciao! |
(0005493) steffen (reporter) 2021-09-03 13:37 |
P.S.: i added yet another public domain file, gmake-4.3.wait.diff, which can be patch(1)ed onto the currently released version of Paul Smith's version of make without conflicts. It is not so easy to build and use development versions of software in today's software world, unfortunately. It is identical to easier.diff but for removal of useless cruft hunks. (Just flags targets which should be waited for, then later recurses into these to also mark all their dependencies like that, so that in effect they are waited for. Works for me!?! Ciao.) |
(0005494) psmith (developer) 2021-09-03 14:48 |
Re Note 5489: That text seems acceptable to me. I can't say for sure when this feature will be made available in GNU make however. |
Issue History | |||
Date Modified | Username | Field | Change |
2020-12-15 21:02 | steffen | New Issue | |
2020-12-15 21:02 | steffen | Name | => steffen |
2020-12-15 21:02 | steffen | Section | => Vol. 3: Shell and Utilities, Issue 7, make |
2020-12-15 21:02 | steffen | Page Number | => (2975) 2989 |
2020-12-15 21:02 | steffen | Line Number | => (98708, 98742) 99308 |
2020-12-17 17:10 | psmith | Note Added: 0005184 | |
2020-12-17 19:23 | steffen | Note Added: 0005185 | |
2020-12-17 21:16 | psmith | Note Added: 0005186 | |
2021-08-14 12:19 | joerg | Note Added: 0005444 | |
2021-08-19 16:22 | rhansen | Note Added: 0005446 | |
2021-08-19 16:22 | rhansen | Note Edited: 0005446 | |
2021-08-19 16:23 | rhansen | Interp Status | => --- |
2021-08-19 16:23 | rhansen | Final Accepted Text | => Note: 0005446 |
2021-08-19 16:23 | rhansen | Status | New => Resolved |
2021-08-19 16:23 | rhansen | Resolution | Open => Accepted As Marked |
2021-08-19 16:24 | rhansen | Tag Attached: issue8 | |
2021-08-19 16:59 | rhansen | Relationship added | related to 0001436 |
2021-08-19 18:39 | psmith | Note Added: 0005447 | |
2021-08-19 18:55 | rhansen | Note Added: 0005448 | |
2021-08-19 19:00 | rhansen | Summary | make: (document .NOTPARALLEL and .WAIT special targets) in RATIONAL => make: (document .NOTPARALLEL and .WAIT special targets) in RATIONALE |
2021-08-19 19:03 | rhansen | Note Edited: 0005448 | |
2021-08-19 19:04 | rhansen | Note Edited: 0005448 | |
2021-08-19 19:04 | rhansen | Note Edited: 0005448 | |
2021-08-19 19:27 | shware_systems | Note Added: 0005449 | |
2021-08-19 22:47 | psmith | Note Added: 0005450 | |
2021-08-19 23:09 | shware_systems | Note Added: 0005451 | |
2021-08-20 18:38 | psmith | Note Added: 0005452 | |
2021-08-20 19:28 | rhansen | Note Added: 0005453 | |
2021-08-20 19:30 | rhansen | Note Edited: 0005453 | |
2021-08-20 19:39 | shware_systems | Note Added: 0005454 | |
2021-08-20 19:43 | rhansen | Note Edited: 0005453 | |
2021-08-20 20:20 | psmith | Note Added: 0005455 | |
2021-08-20 20:57 | psmith | Note Added: 0005456 | |
2021-08-21 17:06 | joerg | Note Added: 0005457 | |
2021-08-21 17:07 | joerg | Note Edited: 0005457 | |
2021-08-21 17:08 | joerg | Note Edited: 0005457 | |
2021-08-21 23:12 | joerg | Note Edited: 0005457 | |
2021-08-23 06:41 | rhansen | Note Added: 0005458 | |
2021-08-23 06:42 | rhansen | Note Edited: 0005458 | |
2021-08-23 06:45 | rhansen | Note Edited: 0005458 | |
2021-08-23 06:51 | rhansen | Status | Resolved => New |
2021-08-23 06:51 | rhansen | Resolution | Accepted As Marked => Open |
2021-08-23 12:36 | joerg | Note Added: 0005459 | |
2021-08-23 20:18 | psmith | Note Added: 0005460 | |
2021-08-23 20:39 | joerg | Note Added: 0005461 | |
2021-08-23 20:58 | rhansen | Note Added: 0005462 | |
2021-08-23 20:58 | rhansen | Note Edited: 0005462 | |
2021-08-23 21:02 | rhansen | Note Edited: 0005462 | |
2021-08-23 21:06 | rhansen | Note Edited: 0005462 | |
2021-08-23 21:07 | rhansen | Note Edited: 0005460 | |
2021-08-23 21:08 | rhansen | Note Edited: 0005462 | |
2021-08-23 21:08 | rhansen | Note Edited: 0005462 | |
2021-08-23 21:10 | rhansen | Note Edited: 0005462 | |
2021-08-24 04:03 | shware_systems | Note Added: 0005464 | |
2021-08-24 04:05 | shware_systems | Note Edited: 0005464 | |
2021-08-25 12:53 | psmith | Note Added: 0005465 | |
2021-08-25 14:19 | psmith | Note Added: 0005466 | |
2021-08-25 14:19 | psmith | Note Added: 0005467 | |
2021-08-25 14:19 | psmith | Note Deleted: 0005467 | |
2021-08-25 15:34 | joerg | Note Added: 0005468 | |
2021-08-25 15:35 | joerg | Note Edited: 0005468 | |
2021-08-25 15:42 | joerg | Note Edited: 0005468 | |
2021-08-25 15:46 | joerg | Note Edited: 0005468 | |
2021-08-25 15:47 | joerg | Note Edited: 0005468 | |
2021-08-25 15:47 | joerg | Note Edited: 0005468 | |
2021-08-25 15:49 | joerg | Note Edited: 0005468 | |
2021-08-25 17:52 | psmith | Note Added: 0005469 | |
2021-08-25 18:03 | joerg | Note Added: 0005470 | |
2021-08-25 18:05 | joerg | Note Edited: 0005470 | |
2021-08-25 18:09 | joerg | Note Edited: 0005470 | |
2021-08-26 13:09 | psmith | Note Added: 0005471 | |
2021-08-26 19:23 | joerg | Note Added: 0005472 | |
2021-08-26 19:24 | joerg | Note Edited: 0005472 | |
2021-08-26 20:22 | rhansen | Note Added: 0005473 | |
2021-08-26 20:40 | psmith | Note Added: 0005474 | |
2021-08-26 20:50 | psmith | Note Added: 0005475 | |
2021-08-26 20:52 | psmith | Note Edited: 0005474 | |
2021-08-26 21:00 | rhansen | Note Added: 0005476 | |
2021-08-26 21:01 | rhansen | Note Edited: 0005476 | |
2021-08-26 21:01 | rhansen | Note Edited: 0005476 | |
2021-08-26 21:08 | rhansen | Note Edited: 0005476 | |
2021-08-26 21:12 | joerg | Note Added: 0005477 | |
2021-08-26 21:16 | joerg | Note Edited: 0005477 | |
2021-08-26 21:17 | joerg | Note Edited: 0005477 | |
2021-08-28 12:42 | psmith | Note Added: 0005478 | |
2021-08-28 15:15 | joerg | Note Added: 0005479 | |
2021-08-28 17:21 | psmith | Note Added: 0005480 | |
2021-08-28 19:11 | steffen | File Added: make.diff | |
2021-08-28 19:21 | steffen | Note Added: 0005481 | |
2021-08-29 12:37 | joerg | Note Added: 0005482 | |
2021-09-02 06:14 | rhansen | Note Added: 0005484 | |
2021-09-02 06:15 | rhansen | Note Edited: 0005484 | |
2021-09-02 06:15 | rhansen | Note Edited: 0005484 | |
2021-09-02 06:17 | rhansen | Note Edited: 0005484 | |
2021-09-02 12:23 | psmith | Note Added: 0005485 | |
2021-09-02 13:22 | joerg | Note Added: 0005486 | |
2021-09-02 13:22 | joerg | Note Edited: 0005486 | |
2021-09-02 13:24 | joerg | Note Edited: 0005486 | |
2021-09-02 13:25 | joerg | Note Edited: 0005486 | |
2021-09-02 15:11 | shware_systems | Note Added: 0005487 | |
2021-09-02 15:26 | shware_systems | Note Edited: 0005487 | |
2021-09-02 15:26 | shware_systems | Note Edited: 0005487 | |
2021-09-02 15:26 | shware_systems | Note Edited: 0005487 | |
2021-09-02 15:26 | shware_systems | Note Edited: 0005487 | |
2021-09-02 15:30 | psmith | Note Added: 0005488 | |
2021-09-02 16:28 | rhansen | Note Added: 0005489 | |
2021-09-02 16:29 | rhansen | Note Edited: 0005489 | |
2021-09-02 16:31 | rhansen | Note Edited: 0005489 | |
2021-09-02 16:32 | rhansen | Note Edited: 0005489 | |
2021-09-02 16:36 | shware_systems | Note Added: 0005490 | |
2021-09-02 16:36 | shware_systems | Note Added: 0005491 | |
2021-09-02 16:36 | shware_systems | Note Deleted: 0005491 | |
2021-09-02 16:37 | rhansen | Final Accepted Text | Note: 0005446 => |
2021-09-02 16:43 | Don Cragun | Note Edited: 0005490 | |
2021-09-02 23:20 | steffen | File Added: easier.diff | |
2021-09-02 23:25 | steffen | Note Added: 0005492 | |
2021-09-03 13:33 | steffen | File Added: gmake-4.3.wait.diff | |
2021-09-03 13:37 | steffen | Note Added: 0005493 | |
2021-09-03 14:48 | psmith | Note Added: 0005494 | |
2021-09-09 15:13 | Don Cragun | Final Accepted Text | => See Note: 0005489. |
2021-09-09 15:13 | Don Cragun | Status | New => Resolved |
2021-09-09 15:13 | Don Cragun | Resolution | Open => Accepted As Marked |
2021-11-26 15:03 | geoffclare | Status | Resolved => Applied |
2021-12-16 16:57 | eblake | Relationship added | related to 0001520 |
2023-04-06 08:58 | geoffclare | Relationship added | related to 0001661 |
2024-06-11 09:08 | agadmin | Status | Applied => Closed |
Mantis 1.1.6[^] Copyright © 2000 - 2008 Mantis Group |