Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0000766 [1003.1(2008)/Issue 7] Shell and Utilities Objection Enhancement Request 2013-10-10 16:16 2013-11-07 17:13
Reporter dwheeler View Status public  
Assigned To ajosey
Priority normal Resolution Rejected  
Status Closed  
Name David A. Wheeler
Organization
User Reference
Section make
Page Number 2911
Line Number 95628-95636
Interp Status ---
Final Accepted Text
Summary 0000766: Add VPATH to make
Description A common request from users is that they want to confine all derived files to a single directory, to keep their source directories uncluttered (source: Automake documentation). This is especially helpful when doing cross-compilations to multiple targets, or if the source directory is read-only. Some users also want a way clearly and simply create a system that selectively overrides a few specific source files. Creating multiple copies (or link farms) of big source code systems is unnecessary, and it is also error-prone if there are selected changes.

Many "make" implementations, including GNU make and FreeBSD make, provide VPATH which can do all these things with a simple, easy-to-use mechanism (see http://www.freebsd.org/cgi/man.cgi?query=make&sektion=1 [^] and http://www.gnu.org/software/make/manual/html_node/General-Search.html#General-Search). [^]

VPATH is a colon-delimited list of directories to search (similar to sh's PATH) if the filename is not found directly as stated. Many build systems, such as those generated by automake, work hard to support VPATH (see http://www.gnu.org/software/automake/manual/html_node/VPATH-Builds.html). [^]

VPATH is in wide use, but not in the POSIX standard; I believe it's time to add VPATH to POSIX. One minor complication is that implementations don't agree on what happens if there is a target in a VPATH-identified directory, but it's to be made up-to-date. In this case GNU make reverts to the non-VPATH name for the target, while others do not. I think GNU make semantics should be preferred, since only its semantics work correctly for read-only source directories containing an outdated VPATH value. However, I think we should permit the alternative, since VPATH users don't normally include a target in a VPATH'ed directory could be out-of-date in the first place.

Desired Action In the EXTENDED DESCRIPTION, after "A target is considered out-of-date if it is older than any of its prerequisites or if it
does not exist." add the following sentence: "If a target or prerequisite does not exist, it is not listed in .PHONY, and VPATH is set, then the make utility will search for the prerequisite or target using the list of colon-separated directories listed in VPATH as if they were the current directory, and use that file if found."

After the sentence, "After make has ensured that all of the prerequisites of a target are up-to-date and if the target is
out-of-date, the commands associated with the target entry shall be executed." add the following sentence: "If the target is out-of-date but was found using VPATH, the target SHOULD be named as if it was not found using VPATH, but the target MAY use the name as found via VPATH."
Tags No tags attached.
Attached Files

- Relationships

-  Notes
(0001872)
joerg (reporter)
2013-10-10 16:38
edited on: 2013-10-10 16:59

Your description is ambiguous as it does not mention what should happen
to the internal macros when VPATH is in use.

AFAIR, different make implementations do not set up macros like $* $@ $< $?
in a uniform way after a successful VPATH expansion did occur.

There also should be a definition on whether a name that contains slashes (if this is not an absoloute path name) should be VATH expanded or not.

(0001874)
dwheeler (reporter)
2013-10-10 17:30

Joerg: Great points! Thanks!

It seems to me that internal macros ($*, etc.) should be updated to use the VPATH entries, with the exception of a target being regenerated as I noted above. Different make implementation may not set them up uniformly today, but those look like bugs, not intended features. It's hard to imagine why "make" should find some prerequisites (for example) using VPATH, yet never update $?. Any name that's not absolute should be expanded, in my view.

Here's a revised description, given those comments:

In the EXTENDED DESCRIPTION, after "A target is considered out-of-date if it is older than any of its prerequisites or if it
does not exist." add the following sentence: "If a target or prerequisite with a relative pathname does not exist, it is not listed in .PHONY, and VPATH is set, then the make utility will search for the prerequisite or target using the list of colon-separated directories listed in VPATH as if they were the current directory, and use that file if found."

After the sentence, "After make has ensured that all of the prerequisites of a target are up-to-date and if the target is
out-of-date, the commands associated with the target entry shall be executed." add the following sentence: "If the target is out-of-date but was found using VPATH, the target's pathname (as reported in $@ and $*) should be given as if it was not found using VPATH, but the target MAY use the pathname as found via VPATH. The pathnames of prerequisites found using VPATH must prepend the VPATH directory chosen (as reported in $? and $<)"


An interesting discussion about current VPATH differences is here:
http://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/VPATH-and-Make.html#VPATH-and-Make [^]
These variances are exactly the kind of problem that a standard should try to resolve :-).
(0001903)
dwheeler (reporter)
2013-10-14 21:55

P.S.: Another "make" implementation with VPATH support is AT&T nmake, described here:

http://web.archive.org/web/20130605191027/http://www2.research.att.com/~gsf/nmake/nmake.html [^]
(0001904)
shware_systems (reporter)
2013-10-15 00:59

If I recall properly, it was noted in some discussion about VPATH that while its syntax and behavior in various environments might be well defined, makefiles could not presume that any fixed paths in one environment had the same purpose in another environment. So, all use of VPATH had to be considered implementation specific as an administrative matter beyond the scope of the standard. This is because there exists the possibility that a generated dependency or target might overwrite files from a separate package with the same names, especially where make is used to customize application scripts. This differs from INCLUDE, LIB, and PATH specs as they are used for finding files generally treated as marked read or execute only, not files expected to stay as writable.

The current language is geared towards, as I see it, that make will write generated dependencies or targets either to the source code directory or the directory the makefile is in, or a sibling or sub-dir of those directories for out-of-tree type builds, and these have a common parent directory that doesn't conflict with any implementation specific dirs to avoid those kinds of accidental overwrites. Installation of final target files from there is also considered an administrative function that takes into account how the implementation wants things organized, not have the standard impose an organization on all implementations.
(0001910)
dwheeler (reporter)
2013-10-15 18:28

Yes, the current language is focused more on dealing with dependencies and targets that are either in the source code directory or descended from the current directory. So the variances in paths are not a problem. Most of the VPATH builds I see are where there are multiple "build" subdirectories for different environments, and the source directory is some variation of "../".

So in short, environment variation isn't a problem for standardization.
(0001916)
joerg (reporter)
2013-10-16 10:50

This is why I mentioned that we first need to identify what is different
with different make implementatiins. It does not help at all to standardize
on VPATH and to do this in a way that does not allow to use this feature
with different make implementation.

When I last testes BSD make (aprox. in y2000), I could e.g. not use the make
from FreeBSD for the schily makefile system because pattern macro expansions
have not been implemented in a compatible way and because path expansions
for VPATH have been done in an incompatible way to other make implementation.
BTW: IIRC, gmake also has some minor deviations from UNIX make implementation
with VPATH support here.

BTW: smake implements VPATH as a compatibility courtesy only as there is a
more powerful mechanism since 1982:

.SEARCHLIST:

holds a list of src-dir obbj-dir pairs that is used for searching.

.OBJSEARCH:

controls the search method in that list if present.

Where value may be one of src, obj, all. causes smake to
look for left-hand-sides of rules in only the source-
directories, the object-directories or both. The default
value for .OBJSEARCH: is: all. That causes smake to search
for results in object-directories and source-directories.

If no .SEARCHLIST target exists, but a VPATH= macro is
found, smake transforms the content of the VPATH= macro into
a form suitable for .SEARCHLIST:. This is done by putting
each VPATH= entry twice into the .SEARCHLIST:.
(0001918)
dwheeler (reporter)
2013-10-16 12:51

"It does not help at all to standardize on VPATH and to do this in a way that does not allow to use this feature with different make implementation."

I agree. So what needs fixing?

"BTW: smake implements VPATH as a compatibility courtesy only as there is a
more powerful mechanism since 1982:..."

That's my point. Lots of people implement VPATH, because users expect it.

If there's a better mechanism that "make" implementers will agree to implement, that's fantastic. But if we can't gain agreement on a "better" mechanism, then let's get agreement on mechanisms that users use and implementations generally support.

If you'd like to propose .SEARCHLIST and .OBJSEARCH list, that's fine. But as you noted, you had to also implement VPATH because users expect it. So, sounds like it's time to standardize it, so that users have SOME useful mechanism.
(0001926)
joerg (reporter)
2013-10-17 10:36
edited on: 2013-10-17 10:37

In order to create a standard for this feature, we need to have a complete
description of the way it should work.

Before we have this, we should not standardize it.

If you did ask me 10+ years before, I could answer these questions,
but now I only remembered that gmake causes problems with the feature
and BSD make is so different, that it is impossible to find a common
denominator.

What I have in mind:

BSD make expands dynamic macros to some unexpected values if the
target name contains slashes.

gmake does not expand all dynamic macros the same way.

For this reason, I believe that we need to define exactly how
VPATH should work and how dynamic macros should be expanded in
case the prerequisite search did use entries in VPATH.

I am nowever not sure anymore and it may be that the problem
was path expansion for named prerequisites that have been
located via VPATH. It also may be that my problem was that
there is no workaround that is usable for gmake and BSD make
at the same time.

(0001927)
dwheeler (reporter)
2013-10-17 14:49

Are you *sure* that those are the differences? Can you point us to any more info?

I know there's a difference between BSD and GNU make's VPATH implementation in they determine what the target "really" is, if it was found using VPATH. Perhaps that's what you're remembering. It turns out that the GNU make documentation specifically notes this different algorithm and documents it:
http://www.gnu.org/software/make/manual/make.html#Search-Algorithm [^]

In typical use cases I've seen, the semantic differences between the BSD and GNU make approaches actually don't actually matter. Typically if you find a target using VPATH it would be up-to-date regardless. Even if it isn't, it doesn't matter for the build where the target is put, since that result will be used either way for that and all future builds. I think the GNU folks have a point that their algorithm is more likely to produce the "expected" results, but I don't see the need to mandate it. Thus, I think POSIX can leave that aspect unspecified and just move on.

In my proposed action, I specifically stated that "the target MAY use the name as found via VPATH". That's the point of contention between the GNU and BSD makes, and my proposed text permits either behavior. At least, that was the intention :-).
(0001935)
ranjit (reporter)
2013-10-19 12:52

I think the problem joerg is alluding to is that where there are two variant definitions, either the Committee picks the better one (such as sockets from BSD, and mmap from SVR4) or it defines a new function/interface where there is no clear consensus on what is best; examples would be anything starting posix_*, pthread_* and so on; in this context ::= should make the situation clear.

It is unusual for an operator to be defined, since most languages were defined with a fairly complete set. make is different: as its author said shortly after, he wrote it as he did because he hadn't learnt yacc at that point. It's not his fault no-one paid a blind bit of notice.

In this context, AIUI GNU consider their algorithm superior since it enables read-only source trees to remain read-only as it never even attempts to write therein. This is particularly useful for nfs builds, as well as for centralised patching and developer/automation purposes.

Unless the alternative has some compelling advantage no-one has written up yet, or I am simply ignorant thereto, that would seem to indicate a clear end-user benefit to me, and so gets my "vote" for plain VPATH until such time as said compelling advantage is presented, and deemed overwhelming.

I do not agree that make is by definition constrained to any directory: however a portable application in this context is a set of sources delivered to be built via a [mM]akefile. Thus all paths within the source cpio/pax archive MUST be portable for conformant usage, afaict. As such, can we at least agree that wherever the sources are read from, the main makefile is typically used to build a tree of objects, starting from a parent which is the end-user's responsibility. Below that path, all paths SHOULD be portable, unless the user or implementation overrides them, in which case it's out of scope.
(0001969)
msbrown (manager)
2013-11-07 17:13
edited on: 2013-11-07 17:15

Due to lack of consensus on the part of implementers on the particulars of how VPATH works, this change is rejected. If the developers of various versions of make agree on a new feature that will be implemented in all versions of make, we will certainly consider a proposal to standardize that new feature.


- Issue History
Date Modified Username Field Change
2013-10-10 16:16 dwheeler New Issue
2013-10-10 16:16 dwheeler Status New => Under Review
2013-10-10 16:16 dwheeler Assigned To => ajosey
2013-10-10 16:16 dwheeler Name => David A. Wheeler
2013-10-10 16:16 dwheeler Section => make
2013-10-10 16:16 dwheeler Page Number => 2911
2013-10-10 16:16 dwheeler Line Number => 95628-95636
2013-10-10 16:38 joerg Note Added: 0001872
2013-10-10 16:59 joerg Note Edited: 0001872
2013-10-10 17:30 dwheeler Note Added: 0001874
2013-10-14 21:55 dwheeler Note Added: 0001903
2013-10-15 00:59 shware_systems Note Added: 0001904
2013-10-15 18:28 dwheeler Note Added: 0001910
2013-10-16 10:50 joerg Note Added: 0001916
2013-10-16 12:51 dwheeler Note Added: 0001918
2013-10-17 10:36 joerg Note Added: 0001926
2013-10-17 10:37 joerg Note Edited: 0001926
2013-10-17 14:49 dwheeler Note Added: 0001927
2013-10-19 12:52 ranjit Note Added: 0001935
2013-11-07 17:13 msbrown Interp Status => ---
2013-11-07 17:13 msbrown Note Added: 0001969
2013-11-07 17:13 msbrown Status Under Review => Closed
2013-11-07 17:13 msbrown Resolution Open => Rejected
2013-11-07 17:15 msbrown Note Edited: 0001969


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker