Austin Group Defect Tracker

Aardvark Mark III


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0000513 [1003.1(2008)/Issue 7] Shell and Utilities Objection Enhancement Request 2011-11-17 00:02 2011-11-18 18:05
Reporter dwheeler View Status public  
Assigned To ajosey
Priority normal Resolution Open  
Status Under Review  
Name David A. Wheeler
Organization
User Reference
Section make
Page Number 2916
Line Number 95850
Interp Status ---
Final Accepted Text
Summary 0000513: Add pattern rules (metarules) to make
Description Pattern rules (aka "metarules") are implemented in several makes, including GNU make and Oracle/Sun make (http://download.oracle.com/docs/cd/E23824_01/html/821-1461/make-1s.html). [^] They were previously considered by the committee (lines 96347-96352). It was not added at that time, but as noted, the rules for target names have been set to reserve the '%' character, and that "Implementors are strongly encouraged to use these characters only for these purposes". This text sets up a situation where pattern rules can be added at some future time.

Here is an example of a pattern rule:
lib/%.o: lib/%.c
        $(CC) -fPIC -c $(CFLAGS) $(CPPFLAGS) $< -o $@
(Note: For this example to work, we also need to allow $< to be used in commands.)

I believe now is the time to add pattern rules to standard make:
* They are more flexible. In particular, they can properly handle prefixes (including directory prefixes) and filenames with multiple suffixes (like ".tar.gz"); basic inference rules cannot.
* They can support automatic chaining.
* They are clearer; using "%.o: %.c" to represent ".o depends on .c" is more consistent with the general target: precondition syntax than ".c.o".
* They are implemented in multiple implementations (as noted above).


Desired Action On page 2916, before the text on "Inference rules", add a new section called "Pattern rules".

If at least one target in a target rule includes the '%' character, it defines a "pattern rule". A pattern rule is used if no more specific target rule can be found to generate a target (if no pattern rule can be found, then an inference rule is used as described below where available).

The "%" matches 0 or more characters; it must be prefixed, suffixed, or both with other characters (which must match a filename where present). The characters that match the '%' character in each use of a pattern rule in a target are termed the "stem". A '%' character in the prerequisites, if any, must match the same stem.

If any prerequisites do not exist, make must determine if it can make those prerequisites (this is called "chaining"). The prerequisites must exist, or make must determine a chain of other rules that could make the prerequisites, for the pattern rule to apply. A rule whose prerequisites actually exist or are mentioned always takes priority over a rule with prerequisites that must be made by chaining other implicit rules.

If more than one rule can apply, the rule with the shortest stem is used. If there is more than one such rule, the one first mentioned in the makefile is used.

Tags No tags attached.
Attached Files

- Relationships

-  Notes
(0001023)
joerg (reporter)
2011-11-17 10:32

The proposed text is far from being a sufficient explanation.

See the SunOS make manual page (but this is unfortunately not complete). SunPro make introduced this feature in 1986 and there are some important constraints that need to be honored even in a simplified implenmentaion:

- Pattern matching rules must have a higher precedence than suffix rules. This is mentioned in the SunOS manual page.

- Pattern matching rules must have a defined precedence order. SunPro Make defines the order of evaluation == the order of definition of pattern rules. This is infortunately not mentioned in the SunOS man page.
(0001029)
Konrad_Schwarz (reporter)
2011-11-17 16:06
edited on: 2011-11-17 16:07

The pattern matching operator is insufficient to specify retrieval of SCCS or RCS files from directories other than the current one.

E.g., GNU Make has the following rules built in:
%:: RCS/%,v
%:: RCS/%
%:: s.%
%:: SCCS/s.%
This works only for files located in the current directory.

To eliminate recursive make, an extension is required.

For example, this could take the form (%D) and (%F),
allowing rules such as:
%:: (%D)/RCS/(%F),v
etc.

(0001032)
joerg (reporter)
2011-11-17 16:47

I am not sure whether it is apropriate to try to eliminate recursive make.

There are many other problems with non-resurcive make systems.

SunPro make introduced a technology that allows to deal with all needed dependencies in recursive build systems. This is done by not only auto-creating C-proprocessor dependencies (via environment SUNPRO_DEPENDENCIES= that is understood by Sun C and gcc). SunPro make in addition supports (together with the Solaris linker) via SGS_SUPPORT=libmakestate.so.1 to get a list of path names for all linked files (including libraries). This does not seem to leave any unclosed hole.
(0001033)
dwheeler (reporter)
2011-11-17 17:08

Regarding comment #1023:
The precedence rules are already there.
The first rule noted is embodied in this text:
"A pattern rule is used if no more specific target rule can be found to generate a target (if no pattern rule can be found, then an inference rule is used as described below where available)."
The second rule is also noted in a more complex way. The shortest stem is used first, and if that fails, THEN the first one defined is used. So the proposal adds another precedence rule (shortest stem), which is I think the right semantic (basically, "the most specific rule" is used).

Regarding comment #1029:
A bare "%" could be supported by changing the proposal slightly. Change:
  The "%" matches 0 or more characters; it must be prefixed, suffixed, or both with other characters (which must match a filename where present).
to:
  The "%" matches 0 or more characters; it may be prefixed, suffixed, or both with other characters (which must match a filename where present).

Regarding "(%D)" - I presume you mean "$(%D)" and "$(%F)", right? We could certainly add the various special variables from say GNU make, since those are already implemented (I much prefer stealing stuff from some existing make, instead of coming up with something new.) Let me know if there's some subset you think is especially important.

Regarding comment #1032:
I don't think we should try to PREVENT recursive make. But "make" should have enough functionality so that it is not considered NECESSARY if people don't want to do it. In any case, adding pattern rules to the standard helps both recursive and non-recursive make use cases.
(0001034)
joerg (reporter)
2011-11-17 17:26

Let me answer one by one. First avoiding recursion:

In order to avoid recursion, there is a need to be able to specify target specific macro definitions. The current POSIX make does not have this feature. SunPro make has it via

tgt list := macro = value

and

tgt list := macro += value

In addition, the current definitions for include statements is not sufficuent.
There is a need to define that and how rules are evaluated while trying to include non-existent or outdated include files and there is a need to define how/when make could enable re-evaluation of rules read from included files.

GNU make does not behave as expected in this area, smake implements something that has been verified as working as expected during the past 15 years.
(0001035)
psmith (reporter)
2011-11-17 18:27

Make recursion is a tangent we don't need to follow here: it doesn't have anything to do with this proposal IMO. However, I'll say that I don't agree with comment #1034 about what is required for non-recursive implementations. There are lots of ways to manage macro namespaces, for example, in standard make without requiring target-specific macro definitions. That's a discussion best had elsewhere.

Regarding comments #1029 and #1033, first note that $(%D) etc. is NOT a standard GNU make macro, and in fact cannot be implemented as such because the standard requires target and prerequisite lists are expanded during makefile parsing, but the expansion of such a variable cannot be performed correctly until much later when make is walking the dependency graph and knows what the stem will be. Both Sun make and GNU make support a deferred expansion that helps with this (GNU make calls it "secondary expansion") but I don't think we want to make that a prerequisite for this proposal.

GNU make addresses the problem in comment #1029 for some situations by being clever about how it handles stems that contain slashes. See http://www.gnu.org/software/make/manual/html_node/Pattern-Match.html [^] This won't meet every possible need. Also it's a commonly misunderstood feature, which possibly says more about the quality of the documentation in this area but may also signal that the feature is too opaque to be a good candidate for standardization.

However I do agree that the proposal should say something about how file names containing slashes are handled... as worded here GNU make doesn't meet the proposal in this situation.
(0001036)
dwheeler (reporter)
2011-11-17 18:29

Replying to comment #1034:

A lot of the comment text seems to be an argument for a conditional capability in make. That should be a separate proposal. This proposal is about pattern rules, which are useful whether there are conditionals or not.

I agree that the standard make's include statements are not sufficient, but again, that is a separate issue. In fact, there are currently two other comments about include:
* One is to have a way to gracefully handle include files if they don't exist. A change has already been accepted to address this, in http://austingroupbugs.net/view.php?id=333 [^] - so I think that problem has been helped at least.
* Another is to automatically re-evaluate rules from included files, so that dependencies can be automatically determined and then applied. That was proposed in http://austingroupbugs.net/view.php?id=332 [^] and rejected, though the only reason stated was "no consensus". I have no idea WHY there was no consensus.
I'd like to see bug 332 re-opened, because this forces multiple invocations of make to do builds (one to figure out the dependencies, the other to actually use them), which is really unfortunate. Users should be able to just type "make" and be certain that everything necessary happen automatically; many current make implementations provide this, but it's not in the standard. But I digress; bug 513 isn't about that at all. If you want automatic rebuilding, I encourage you to revisit and argue for bug 332 (I'd love to see it!).
(0001037)
dwheeler (reporter)
2011-11-17 22:11

Regarding comment #1035:

I agree that deferred expansion is not necessary for this to be very useful, and it complicates things. I don't currently plan to add that to this proposal.

The "clever approach" GNU make takes to handling stems actually results in very intuitive results. Basically, if you don't include a "/" in the pattern, then the pattern will work in any directory at any level and produce the pathname with the right prefixed directories. I think this is important to include.

I think the challenge is that it's hard to express this idea precisely and clearly. Here's the GNU text: "When the target pattern does not contain a slash (and it usually does not), directory names in the file names are removed from the file name before it is compared with the target prefix and suffix. After the comparison of the file name to the target pattern, the directory names, along with the slash that ends them, are added on to the prerequisite file names generated from the pattern rule's prerequisite patterns and the file name. The directories are ignored only for the purpose of finding an implicit rule to use, not in the application of that rule. Thus, ‘e%t’ matches the file name src/eat, with ‘src/a’ as the stem. When prerequisites are turned into file names, the directories from the stem are added at the front, while the rest of the stem is substituted for the ‘%’. The stem ‘src/a’ with a prerequisite pattern ‘c%r’ gives the file name src/car."
(http://www.gnu.org/software/make/manual/html_node/Pattern-Match.html). [^]

Making this precise is tricky. Here's my try, please append this to the proposal:

If no target pattern in a given pattern rule contains a slash, then this rule is used to match targets and generate prerequisite pathnames:
* Only the target's basename compared with the target pattern (including its prefix and suffix).
* If there is a match and the target pathname includes slash, the stem begins with the target's dirname and ending slash. Then, whether there is a slash or not, the stem includes all of the target pathname that did not match any prefix or suffix in the target pattern.
* Any prerequisite with a '%' is formed in the following manner: If the stem includes slash, it begins with the stem's dirname and ending slash. This is followed by the prerequisite prefix (if any), the rest of the stem, and the prerequisite suffix (if any).
In contrast, if the target pattern does contain a slash, then the full pathname (including directory names if any) is compared to determine if there is a match.

There's probably a much simpler way to express this precisely.
(0001038)
joerg (reporter)
2011-11-18 10:46

Replying to comment #1036:

I have not been in the teleconference that decided "non consensus", but I know that gmake implements a method that does not evaluate include dependencies before doing the include and for this reason cannot include files that are created as a result from an existing make rule.

gmake instead evaluates these rules after doing all includes and then restarts itself. This is in conflict with the treatment of makefiles, where related rules are evaluated before a makefile is opened for reading.

I've implemented a working method in smake in 1997 and this method did not cause any problem since then. I am willing to write down the basics of this method.
(0001039)
joerg (reporter)
2011-11-18 11:00

Replying to comment #1035:

It is nice to see the explanation for gmake pattern rules, but please keep in mind that gmake tries to reimplement a feature that has been introduced in 1986 by SunPro make. In case of diverting behavior (I did not test so far whether there is one), I tend to standardize on the behavior of the initial implementation.
(0001040)
psmith (reporter)
2011-11-18 13:37

Regarding comment #1038: that explanation of the GNU make behavior is confusing and misleading. However this isn't the place to discuss that issue: please use the comments of http://austingroupbugs.net/view.php?id=332 [^] for further discussion.

Regarding comment #1039: do we have access to Sunpro make? Unless it's generally available we'll rely on you or someone else with access to verify whether the proposals here would match its behavior and if not, describe the differences.
(0001041)
joerg (reporter)
2011-11-18 14:17

Regarding comment #1040:

The gmake behavior regarding includes is confusing and missleading. I am sure that for this reason the behavior has been accepted as a bug that needs a fix in 1998 by the gmake maintainer.

Bug #332 has been closed and does not seem to be apropriate as this would be a discussion about enhancing the include syntax in order to make include better suitable for portable makefiles. It seems that we need a way to migrate off-topic discussions to new bug numbers. Andrew, do you have a proposal how to do this best?

I have no idea who is ment by "we" but anybody who is interested can have access to SunPro Make. SunPro make replaced the AT&T make in 1986 on SunOS 3.x and it still is the standard make on Solaris/OpenSolaris. An enhanced version called "dmake" is part of the Sun compiler collection that even exists for Linux. The complete sources for "dmake" are not available but the sources for the basic software have been opensourced by Sun in December 2006 at my request.
(0001042)
psmith (reporter)
2011-11-18 18:05

I will use the mailing list to discuss topics not related to this issue.

By "we" I mean anyone who is interested in investigating how the SunPro make implementation of pattern rules may or may not meet the description David proposes, and add that information here so that it can be considered when evaluating this issue.

- Issue History
Date Modified Username Field Change
2011-11-17 00:02 dwheeler New Issue
2011-11-17 00:02 dwheeler Status New => Under Review
2011-11-17 00:02 dwheeler Assigned To => ajosey
2011-11-17 00:02 dwheeler Name => David A. Wheeler
2011-11-17 00:02 dwheeler Section => make
2011-11-17 00:02 dwheeler Page Number => 2916
2011-11-17 00:02 dwheeler Line Number => 95850
2011-11-17 10:32 joerg Note Added: 0001023
2011-11-17 16:06 Konrad_Schwarz Note Added: 0001029
2011-11-17 16:07 Konrad_Schwarz Note Edited: 0001029
2011-11-17 16:47 joerg Note Added: 0001032
2011-11-17 17:08 dwheeler Note Added: 0001033
2011-11-17 17:26 joerg Note Added: 0001034
2011-11-17 18:27 psmith Note Added: 0001035
2011-11-17 18:29 dwheeler Note Added: 0001036
2011-11-17 22:11 dwheeler Note Added: 0001037
2011-11-18 10:46 joerg Note Added: 0001038
2011-11-18 11:00 joerg Note Added: 0001039
2011-11-18 13:37 psmith Note Added: 0001040
2011-11-18 14:17 joerg Note Added: 0001041
2011-11-18 18:05 psmith Note Added: 0001042


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker