Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0001325 [Issue 8 drafts] Shell and Utilities Editorial Clarification Requested 2020-02-09 17:17 2020-12-09 09:56
Reporter dmitry_goncharov View Status public  
Assigned To
Priority normal Resolution Accepted As Marked  
Status Applied   Product Version Draft 1
Name Dmitry Goncharov
Organization
User Reference
Section (section number or name, can be interface name)
Page Number (page or range of pages)
Line Number (Line or range of lines)
Final Accepted Text Note: 0005087
Summary 0001325: Allow make to remake an included file
Description https://austingroupbugs.net/view.php?id=333 [^]

proposes the following wording
 
"If the file cannot be opened, and if the word include was prefixed with a
<hyphen> character, the file shall be ignored. Otherwise, if the file cannot be
opened an error occurs."

If this wording is added to the standard it'll outlaw a widely used scenario.
Specifically a makefile has a rule to build a dependency file and an include
directive without hyphen to include this freshly built dependency file.
If the file is missing the rule builds it and include includes it.
include rather than -include is more useful for this scenario because if the
file cannot be remade the user will know.
This mechanism is supported by gnu make and is used by automake and thus by a
miriad projects that use autotools.
Desired Action After
"If the word include"
append
", optionally prefixed with a <hyphen> character,".

At the end, append:
"If the file cannot be opened and the makefile has a rule to build the file
make may remake the file.
If the file cannot be opened and cannot be remade, and if the word include was
prefixed with a <hyphen> character, this -include directive shall be ignored.
If the file cannot be opened and cannot be remade, and if the word include was
not prefixed with a <hyphen> character, an error occurs."
Tags issue8
Attached Files

- Relationships
parent of 0001415Closedajosey Issue 8 drafts The text added as a result of Issue #1325 is unacceptable 
has duplicate 0000328Closed 1003.1(2008)/Issue 7 Auto-Dependency Generation 

-  Notes
(0004780)
shware_systems (reporter)
2020-02-09 18:29

As a potential breaking change to some implementations I'm not sure this is desirable. If such a workflow is needed the same effect is produced by having the file as a target of its own before the include line, with the current wording. Having include fail indicates the commands associated with an explicit target were not successful in creating it. I can see an App Usage note explaining this, not modifying the include argument to be treated fully as a prerequisite file; matching targets and inference rules, especially if the file suffix matches additional inference rules built-in to a particular implementation. The gnu make behavior would still be allowed to makefile's that don't have the .POSIX special target.
(0004781)
joerg (reporter)
2020-02-10 13:27
edited on: 2020-02-10 13:28

Re: Note: 0004780 Nobody will implement a behavior since SunPro Make introduced the behavior of first trying to evaluate an existing rule before the include statement is handled.

Prepending "include" by a minus thus just can be seen as "do not cause an error if the file cannot be included after a potential rule to create it has been run".

A big problem in this area however is that GNU make did missbehave for this statement for a really long time. The new gmake version that appeared recently fixed this in serial mode, but fails even more miserably in parallel mode.

But it may be of interest that SunPro Make see schilytools: http://sourceforge.net/projects/schilytools/files/ [^]

supports "include" since SunOS-3.5 (January 1986) and my enhanced version supports "-include" since January 2018, the way I documented above.

My smake (first written by me in 1985) supports include and -include since 24 years.

I cannot speak for GNU make, but SunPro Make passed the POSIX compliance tests together with the Solaris certification. It would be a really bad idea to make the standard in conflict with the current behavior of the most important make implementations.

BTW: I was in the assumption, that the new POSIX text is compatible with the SunPro Make implementation.

(0005066)
geoffclare (manager)
2020-10-26 15:53

In Issue 8 draft 1 page 2892 lines 97098-97102 (make extended description), change:
If the file does not exist in this location, it is unspecified whether additional directories are searched. If the file cannot be opened, and if the word include was prefixed with a <hyphen-minus> character, the file shall be ignored. Otherwise, if the file cannot be opened an error shall occur.
to:
If a target rule or inference rule for the pathname has been parsed before the include line is parsed make shall use the rule to attempt to create the file or to bring it up to date before opening the file. If the pathname is relative, the file does not exist, and it cannot be created using such a rule, it is unspecified whether additional directories are searched for an existing file of the same relative pathname. If, after applying such a rule (if any), the file cannot be opened, and if the word include was prefixed with a <hyphen-minus> character, the file shall be ignored. Otherwise, if the file cannot be opened an error shall occur.

In issue 8 draft 1 page 2911 after line 97926 (make rationale), insert a new paragraph:
This standard specifies that if a target rule or inference rule for an include line pathname has been parsed before the include line is parsed, make will use the rule to attempt to create the file or to bring it up to date before opening the file. Historical versions of GNU make delay the use of the rule until after all makefiles have been read, which means that a rule for an include file cannot be defined in an earlier include file (that itself needs to be made via a rule). Historical versions of BSD make do not attempt to (re-)make the include file. This standard also specifies that an include file is first (re-)made if possible, and only if not possible can other directories be searched. Historical versions of GNU make first search the include directories, then attempt to (re-)make the include file(s) in reverse order of the include line's appearance in the makefile.
(0005068)
rhansen (manager)
2020-10-26 16:57
edited on: 2020-10-26 17:00

> Historical versions of GNU make delay the use of the rule until after all makefiles have been read,

That is true, but...

> which means that a rule for an include file cannot be defined in an earlier include file (that itself needs to be made via a rule).

I don't think that is correct. With the following files, <tt>make foo</tt> (GNU Make 4.3) properly builds <tt>a.mk</tt> and <tt>foo</tt>:

# Makefile
include a.mk
-include b.mk


# a.mk
b.mk:
    printf 'foo:\n\techo foo >foo\n' >b.mk


Here is the script I used to test:

#!/bin/sh
d=$(mktemp -d) || exit 1
trap 'rm -rf "$d"' EXIT
cd "$d" &&
printf -- 'include a.mk\n-include b.mk\n' >Makefile &&
printf 'b.mk:\n\tprintf '\''foo:\\n\\techo foo >foo\\n'\'' >b.mk\n' >a.mk &&
make foo &&
[ -f foo ]


Furthermore, GNU Make 4.3 allows the rule for an include file to be defined after the include line like this:

# Makefile
-include a.mk
include b.mk


# b.mk
a.mk:
    printf 'foo:\n\techo foo >foo\n' >a.mk


Here is the script I used to test:

#!/bin/sh
d=$(mktemp -d) || exit 1
trap 'rm -rf "$d"' EXIT
cd "$d" &&
printf -- '-include a.mk\ninclude b.mk\n' >Makefile &&
printf 'a.mk:\n\tprintf '\''foo:\\n\\techo foo >foo\\n'\'' >a.mk\n' >b.mk &&
make foo &&
[ -f foo ]


(0005069)
nick (manager)
2020-10-26 17:08

Reopening in the light of 0001415
(0005070)
geoffclare (manager)
2020-10-27 10:09

Re Note: 0005068 your example doesn't fit the situation described in the text, as a.mk already exists when you run make, whereas the text has the caveat "that itself needs to be made via a rule".
(0005071)
joerg (reporter)
2020-10-27 13:09
edited on: 2020-10-27 13:10

Re: Note: 0005068 also note that the background for deciding to make the documented behavior mandatory was that our various tests discovered non-orthogonal behavior in GNU make and even an unexplained incompatibility between gmake-4.1 and gmake-4.3.

Given that gmake implements other deviations from the POSIX standard (e.g. space and backslash handling) gmake cannot be used for POSIX makefiles the way it currently behaves.

POSIX requires a POSIX compliant behavior only in case that .POSIX appears in the Makefile. Gmake therefore could use .POSIX to switch to compliant behavior that matches SunPro Make and smake.

(0005072)
psmith (developer)
2020-10-27 14:04

GNU make's algorithm for rebuilding makefiles, which has been in place for probably 30 years (certainly far enough back that I have no history from before it was available), works like this:

1. Parse all makefiles; if any included makefiles are missing, make a note of them
2. After parsing is complete, try to rebuild all makefiles regardless of whether they are missing or not (even existing makefiles are rebuilt, if they are out of date)
3. If any required makefile does not exist and could not be created, fail
4. If any makefile was rebuilt, re-exec make and start again.

There are some trade-offs between this algorithm and that implemented by SunPro make; however GNU make's algorithm provides capability that the SunPro make algorithm cannot and that capability is considered valuable and is in very wide-spread use among GNU make users. I won't change it.
(0005073)
psmith (developer)
2020-10-27 14:36

> POSIX requires a POSIX compliant behavior only in case that .POSIX appears in the Makefile. Gmake therefore could use .POSIX to switch to compliant behavior that matches SunPro Make and smake.

No, that won't happen.

Practically, there are many thousands of makefiles out there which set .POSIX in order to enable certain types of behavior (primarily the "set -e" breakage) but which still rely on GNU make's method of rebuilding makefiles. I'm definitely not going to break all those makefiles.

Also practically, GNU make's implementation creates a very bright line between "parsing makefiles" and "building targets", and the transition across that line is one way. There is no way to go back to parsing makefiles (in particular, defining new rules and modifying the DAG) once GNU make starts building targets. Supporting this would require complex changes. The only way to implement this behavior otherwise would be to re-exec make after each included makefile has been rebuilt, so its rules could be parsed in a new invocation.

Philosophically, the POSIX standard should not be changed to require a major new feature which immediately causes arguably the most widely-used implementation of make to become non-standard in a fundamental way.
(0005074)
joerg (reporter)
2020-10-27 15:24
edited on: 2020-10-27 15:26

The include directive has been introduced by SunPro Make in January 1986 together with other features like conditional macro assignments, pattern macro substitutions and pattern matching implicit rules. The behavior of SunPro Make did not change since then and is consistent.

gmake first appeared three years later and copied many of the new SunPro Make features. People would assume that this has been done in a compatible way.

The current behavior of gmake is not self consistent and even changed compared to previous gmake versions. The current POSIX text does not break anything that was not broken before by incompatible changes in gmake already. You did not mention a single project that would no longer work if you did make gmake compatible to the include implementation in SunPro Make and smake.

I am not aware of a single gmake based project that fails with SunPro Make just beause of the ideosyncratic implementation for the include feature in gmake. The makefiles designed for gmake typically fail because they are using non-standad features like "if" and a Makefile that contains .POSIX and "if" at the same time is definitely non-compliant. In other words: it seem to be your problem, if you or your customers missuse .POSIX for purposes other than 100% POSIX compliance. Since SunPro Make is POSIX certified (via Solaris), I expect that conforming Makefiles that contain a .POSIX line work with SunPro Make. If they do not, they are not fully conforming and beyond the scope of this discussion.

Useful make implementations call "sh -ce cmd" since 1976 and never changed that. If you really like to permit people to use different shell flags, you could implement the method from smake that is based on MAKE_SHELL_FLAG and MAKE_SHELL_IFLAG.

Finally: gmake does apply known rules for Makefiles already (e.g. by calling sccs get s.Makefile) before trying to open the Makefiles. The strategy for include files should be compatible.

(0005075)
psmith (developer)
2020-10-27 18:28

I have seen no indication anywhere from either Roland or Richard that their intent was to copy (as in, duplicate) the behavior of SunPro make. That is an assumption on your part which you repeatedly state, that I have never seen anyone corroborate. It seems unlikely to me that this would have been the intent since the behaviors are so different: such difference seems highly unlikely to have been a mistake. It's quite possible that they heard of the idea of rebuilding makefiles in the abstract, but consciously decided to implement it in a way that they felt was better.

Put another way, it is absolutely not true that GNU make's implementation of include is somehow just a buggy attempt to duplicate SunPro make's implementation where we could standardize SunPro make's behavior and "fix some bugs" in GNU make's implementation. It is a completely different implementation with different intentions and real advantages.

In any event, the intent of people back in the 1980's is irrelevant to us here. The reality today is that, regardless of provenance, the algorithm I stated above is the way that GNU make handles rebuilding included makefiles and has done for 30 years.

I have no intention of changing it in a totally non-backward-compatible way, nor will I agree to implement a radically different behavior that happens only in POSIX mode.

In summary, either the POSIX standard should provide wording which allows for rebuilding makefiles in a way that does not contradict the current GNU make behavior, or else we should not add any text about rebuilding makefiles to the standard and allow the status quo to continue.


As for all the rest of your comments above, needless to say my position is that you are wrong about some of your statements of fact and I disagree with your characterization of many other things. Just one example: your final paragraph, implying that "the strategy for include files" is different than that for Makefiles is not correct. All makefiles, including default, provided with -f, provided with the MAKEFILES environment variable, and via include, are treated the same way.

However this is not the right place to discuss things which aren't related to the standard. If you want to have a conversation about things like that feel free to email me privately or use one of the GNU make mailing lists.
(0005076)
psmith (developer)
2020-10-27 18:30

The contention above that it is currently possible for existing makefiles to work with both SunPro make and GNU make shows that it should be possible to provide compatible wording.

The phrase "make will use the rule to attempt to create the file or to bring it up to date before opening the file" should be removed, and of course the commentary regarding "historical" versions of GNU make should also be removed.

For example, I would be OK with wording that:

(a) required rules that rebuilt makefiles to appear before the include itself, and
(b) required that missing makefiles are rebuilt and the contents of the rebuilt makefiles are read by make, before the goal targets are checked

For (a), GNU make would simply implement an enhancement over the standard which allowed rules to also appear after the include line. For (b), that requirement is satisfied today by both major implementations.

If you wanted to add in a statement that this is only required for makefiles that are missing that's OK as long as it doesn't preclude rebuilding makefiles which are present but out of date, and it doesn't preclude using normal prerequisite detection to decide which include files are out of date; these can be enhancements provided by GNU make.
(0005077)
geoffclare (manager)
2020-10-28 09:14

Re Note: 0005076 The suggested alternative wording would still require a change to GNU make in order for it to conform. That's because it writes to standard error when its exit status is 0 and the standard does not allow that. In order to conform it would have to somehow defer writing diagnostic messages about include files until it knows what the exit status is going to be.
(0005078)
joerg (reporter)
2020-10-28 11:23
edited on: 2020-10-28 12:16

Re: Note: 0005076 The current wording from Note: 0005066 is not in conflict with the current behavior of GNU make and for this reason, there is no need to change the wording.

Given that there are many existing makefiles that fail with the first run of GNU make on a maiden file tree, I see no benefit with the current behavior from GNU make. But....

If there are really benefits with the current behavior in GNU make, GNU make of course could continue to implement its current behavior as an extension. GNU make would however be required to add an attempt to remake an include file just before it is going to be opened. This would fix bugs reported against GNU make since 1998 and this would permit to grant a common behavior amongst various make implementations.

There are currently four actively maintained make implementations that implement POSIX compliance or near POSIX compliance and that need to be taken into consideration:

smake exists since 40 years and implements the method from SunPro Make for the include directive since 1995

SunPro Make exists since nearly 35 years. It introduced the include feature on UNIX.

BSD make exists since 32 years (originally as pmake) and currently does not apply rules for the include directive. This needs to be enhanced, but there is no existing behavior that could be in conflict and BSD make is currently the make implementation with the biggest deviations from existing POSIX behavior anyway.

GNU make exists since 31 years and uses a method that could be made compatible with few effort without breaking compatibility to its existing behavior.

(0005079)
psmith (developer)
2020-10-28 13:47

> That's because it writes to standard error when its exit status is 0 and the standard does not allow that. In order to conform it would have to somehow defer writing diagnostic messages about include files until it knows what the exit status is going to be.

To the best of my knowledge, GNU make does not write any messages about include files at all, to anywhere, until it knows whether they can be remade.

  $ cat Makefile
  /tmp/foo.mk: ; touch /tmp/foo.mk
  include /tmp/foo.mk
  all: ; echo hi

  $ rm -f /tmp/foo.mk && make
  touch /tmp/foo.mk
  echo hi
  hi
  $ echo $?

  $ rm -f /tmp/foo.mk && make >/dev/null
  $ echo $?
  0

You may be referring to an old bug, that has since been fixed, that warned about an included makefile first before it was remade. Or you may be referring to an issue I'm not aware of.
(0005080)
geoffclare (manager)
2020-10-28 14:44

Re: Note: 0005079 My statement was based on the GNU make manual.

https://www.gnu.org/software/make/manual/html_node/Include.html [^]

"If an included makefile cannot be found in any of these directories, a warning message is generated, but it is not an immediately fatal error; processing of the makefile containing the include continues."

Is that part of the online manual out of date?
(0005081)
psmith (developer)
2020-10-28 14:49

> GNU make of course could continue to implement its current behavior as an extension. GNU make would however be required to add an attempt to remake an include file just before it is going to be opened.

Clearly that's not going to work.

First, as I already explained above, currently it is not possible for GNU make to transition back from "building targets" to "reading makefiles" and implementing this new behavior would be a very significant amount of work. This is not just a simple little change.

Second, it would mean that for every included makefile GNU make would have to try to build it twice: once before it was included and then again after all makefiles are read in.

Third, it's not possible to be backward compatible. Consider this situation:

  all:

  %.mk : ; echo 'all: ; echo foo' > $@

  include foo.mk

  foo.mk: ; echo 'all: ; echo bar' > foo.mk

In current GNU make the result will be "echo bar".

After the change you suggest, the result will be "echo foo".
(0005082)
psmith (developer)
2020-10-28 14:50

Just to note, while the standard can define that mk1 is rebuilt in this makefile:

  .POSIX:
  mk1: ; echo all: > mk1
  include mk1

it can't be the case that a makefile like this:

  .POSIX:
  include mk1
  mk1: ; echo all: > mk1

is required to fail. The current standard doesn't explicitly define what happens if a file to be included does not exist and the changes made here should not define it either.
(0005083)
psmith (developer)
2020-10-28 14:53

Yes, that part of the manual is out of date. Thanks for pointing it out!
(0005084)
shware_systems (reporter)
2020-10-28 15:08

Re: 5082
Since a file that does not exist has 0 lines, same as a file of size 0, the standard does specify that the attempt to replace the include line with the file contents means the include directive gets ignored. Any other interpretation I see as presuming things not in evidence; it would take an explicit allowance for other behaviors for these to be conforming. Granted, other behaviors are desirable, but that is what these reports are adding.
(0005087)
geoffclare (manager)
2020-10-30 15:56
edited on: 2020-11-05 10:13

Here is an alternative proposal that is intended to allow GNU make's delayed remaking of include files. It also fixes some things we missed in the original resolution:
The wording change was made in text whose context is a relative pathname. It needs to be separated from that context so that it also applies to absolute pathnames.

The -n option is ignored when the command lines in a rule are being processed in order to create or update an include file. GNU make also does the same for -q and -t, but SunPro make doesn't.

I have also proposed a change to c99 -E to ensure it is usable for generating dependencies. The change is the minimum needed for this purpose - a more detailed change could be made instead if desired. Alternatively we could add a -M option.

Page and line numbers are for Issue 8 draft 1.

On page 2888 line 96923 (make -n option), change:
However, lines with a <plus-sign> ('+') prefix shall be executed.
to:
However, lines with a <plus-sign> ('+') prefix and lines being processed in order to create an include file or to bring it up-to-date (see Include Lines in the EXTENDED DESCRIPTION section) shall be executed.

On page 2889 lines 96929-96931 (make -q option), after:
However, a makefile command line (associated with the targets) with a <plus-sign> ('+') prefix shall be executed
add:
and it is unspecified whether command lines that do not have a <plus-sign> prefix and are being processed in order to create an include file or to bring it up-to-date (see Include Lines in the EXTENDED DESCRIPTION section) are executed

On page 2889 lines 96943-96944 (make -t option), after:
However, a command line with a <plus-sign> ('+') prefix shall be executed.
add:
and it is unspecified whether command lines that do not have a <plus-sign> prefix and are being processed in order to create an include file or to bring it up-to-date (see Include Lines in the EXTENDED DESCRIPTION section) are executed

On page 2891 lines 97033-97035 (make extended description), after:
A target shall be considered up-to-date if it exists and is newer than all of its dependencies, or if it has already been made up-to-date by the current invocation of make (regardless of the target's existence or age)
add:
, except that targets that are made up-to-date in order for them to be processed as include line pathnames (see Include Lines below) need not be considered up-to-date during later processing

On page 2892 lines 97097-97102 (make extended description), change:
  • Each pathname that does not begin with a '/' shall be treated as relative to the current working directory of the process, not relative to the directory containing the makefile. If the file does not exist in this location, it is unspecified whether additional directories are searched. If the file cannot be opened, and if the word include was prefixed with a <hyphen-minus> character, the file shall be ignored. Otherwise, if the file cannot be opened an error shall occur.
to:
  • If the pathname does not begin with a '/', it shall be treated as relative to the current working directory of the process, not relative to the directory containing the makefile.

  • The make utility shall use one of the following two methods to attempt to create the file or bring it up-to-date:

    1. The ``immediate remaking'' method

      If make uses this method, any target rules or inference rules for the pathname that were parsed before the include line was parsed shall be used to attempt to create the file or to bring it up-to-date before opening the file.

    2. The ``delayed remaking'' method

      If make uses this method, no attempt shall be made to create the file or bring it up-to-date until after the makefile(s) have been read. During processing of the include line, make shall read the current contents of the file, if it exists, or treat it as an empty file if it does not exist. Once the makefile(s) have been read, make shall use any applicable target rule or inference rule for the pathname, regardless of whether it is parsed before or after the include line, when creating the file or bringing it up-to-date. Additionally in this case, the new contents of the file, if it is successfully created or updated, shall be used when processing rules for the following targets after the makefile(s) have been read:

      • The target_name operands, if any.

      • The first target make encounters that is not a special target or an inference rule, if no target_name operands are specified.

      • All targets that are prerequisites, directly or recursively, of the above targets.


    If the pathname is relative, the file does not exist, and an attempt to create it using a rule has not been made and will not be made, it is unspecified whether additional directories are searched for an existing file of the same relative pathname.

    If, after proceeding as described above, the file still cannot be opened:

    • If the word include was prefixed with a <hyphen-minus> character, the file shall be ignored.

    • Otherwise, an error shall occur.

On page 2904 after line 97617 (make application usage), insert the following new paragraphs:
The requirements relating to dynamic include files (ones that have rules to create or update them) allow two different methods of implementing them, as explained in RATIONALE below. In order to write a set of makefiles and include files (both dynamic and static) that work with both methods, applications need to ensure the following:

  • Rules containing commands for creating dynamic include files (and any prerequisites) precede the include line that will cause the file to be read.

  • Include lines and rules for creating dynamic include files do not depend on the contents of any earlier dynamic include file. For example, defining a macro in a dynamic include file and using that macro in a later include line should be avoided (unless the later include line is itself inside the dynamic include file).

  • One of the rules used in creating a dynamic include file, or a dynamic prerequisite of an include file, contains commands to create its target and does not ignore creation failure.

Note that although the first point above appears at first sight to preclude a dynamic include file that defines its own prerequisites, this can be achieved by having two include lines that name the same file (provided the rules in the file do not contain command lines), as shown in EXAMPLES.

On page 2905 after line 97644 (make examples), insert a new example 4 and renumber the later examples:
The following is an extended version of the previous example that generates make include files containing the prerequisites for each object file. The include file also defines its own prerequisites, and the makefile arranges that these are used by including the file twice. With implementations of make that create include files during parsing, the first time make is run the file does not exist but the use of the <hyphen-minus> prefix on the first include line means it is ignored and is then generated by the rule preceding the second include line. On subsequent runs, if any of the source or header files previously identified has changed, the include file will be updated. There are other ways of updating the include file when headers change, but they involve recursive execution of make.
.POSIX:
.SUFFIXES: .c .d

OFILES = a.o b.o

pgm: $(OFILES)
    c99 $(OFILES) -o pgm
a.o:
    c99 -c a.c
b.o:
    c99 -c b.c

-include $(OFILES:.o=.d)
.c.d:
    +{ \
        set -o pipefail; cfile=$<; ofile=$${cfile%.c}.o; \
        printf '%s %s: %s ' "$$ofile" $@ $<; \
        c99 -E $< | LC_ALL=C sed -n \
            '/^#[[:blank:]]*[[:digit:]]/s/.*"\([^"]*\.h\)".*/\1/p' | \
            LC_ALL=C sort -u | tr '\n' ' '; \
        echo; \
    } > $@
include $(OFILES:.o=.d)

This example does not cope with a header file being removed (make will complain that it does not know how to make it), necessitating that *.d be removed and the make run again. Alternatively, this can be handled by updating the commands which generate the .d file so that they add an empty target rule for each of its prerequisites.

On page 2911 after line 97926 (make rationale), insert the following new paragraphs:
This standard allows two different methods of creating include files or bringing them up-to-date, reflecting established practice in SunPro make and GNU make. The former performs this action during parsing, before the include file is opened. The latter delays performing the action until after all makefiles have been read. Implementors who opt for the ``delayed remaking'' method should be aware of the following potential issues:

  • Diagnostic messages about missing include files must be deferred until the final exit status is known. Note that this is a conformance issue, not just a quality of implementation issue.

  • If the way make handles using updated include file contents is to start over after include files have been made up-to-date, it is possible for a poorly written makefile to cause make to enter a sequence of restarts where nothing changes each time, resulting in the sequence continuing indefinitely unless the situation is detected. Implementors are encouraged to include a mechanism for detecting and reporting this, rather than allowing make to consume an arbitrary amount of system resource until it is forcibly terminated.

  • If make uses this start-over method, makefile contents read from a pipe on standard input or from a FIFO must be copied to a temporary file, and when make starts over it must use this file instead.

  • If make starts over by executing itself using the exec family of functions, the need to replace '-' or the pathnames of FIFOs with the pathnames of temporary files can lead to the exec call failing with an [E2BIG] error if the original execution was close to the {ARG_MAX} limit. Although this is a quality of implementation issue, not a conformance issue (since the general rules for utility errors allow utilities to fail when they encounter a variety of internal errors - see [xref to 1.4 Utility Description Defaults]), implementors are encouraged to explore ways to prevent it, such as passing information via a temporary file instead of on the command line when an [E2BIG] error has occurred. Another solution might be to jump (e.g. using siglongjmp()) back to the start of main() as the way to start over. Making a recursive call to main() is not recommended, as that would run into the stack limit if sufficiently many restarts are needed.

This standard specifies that a non-existent include file is first created if possible, and only if not possible can other directories be searched. Historical versions of GNU make first searched the include directories, then attempted to create the include file. This behavior was not considered suitable for standardization as it means writers of portable applications have to use absolute pathnames for all include files that need to be created via a rule (because they can never be sure what relative pathnames are safe to use, since a file with the same relative pathname might happen to exist in one of the searched directories when installing the application on a new system). Note, however, that this only applies to directories searched by default. If an application uses an extension to specify that one or more directories are searched, this standard does not place any constraints on when the specified directories are searched.

On page 2456 line 80506-80508 (c99 STDOUT), after:
If the -E option is specified, the standard output shall be a text file that represents the results of the preprocessing stage of the language; it may contain extra information appropriate for subsequent compilation passes
add:
and shall contain at least one line with the format:
"# %d \"%s\"\n", <line>, <pathname>
for each file processed as a result of a #include directive, unless no other output generated from that file is present in the output, where <line> is a line number and <pathname> is the pathname used to open the file

On page 2463 after line 80819 (c99 rationale), add a new paragraph:
This standard requires that, when -E is used, lines beginning with '#' are output identifying the files processed as a result of #include directives in order that c99 -E can be used to generate makefile dependencies. See [xref to make]. Usually such lines are output each time the origin of the subsequent lines in the output changes, and the line number after the '#' is the line number in the named file of the line from which the next line in the output was derived.


(0005089)
joerg (reporter)
2020-10-30 18:52
edited on: 2020-10-30 19:33

I did read Note: 0005087 and I am not sure whether I am fully done with my review yet, but we need to change:

- Rules for creating include files do not depend on the contents of any earlier include file. For example, defining a macro in an include file and using that macro in a later include line should be avoided.

into

- Rules for creating include files do not depend on the contents of any earlier non-static include file. For example, defining a macro in an include file and using that macro in a later include line should be avoided. A non-static include file is an include file that has a rule to create or update that include file.

(0005090)
geoffclare (manager)
2020-10-31 09:21

Re: Note: 0005089 Good catch. I've updated the application usage section of Note: 0005087, although not quite in the way you suggested.
(0005094)
psmith (developer)
2020-11-03 15:23

Thank you for all your effort Geoff! I like this version very much. A few notes / clarifying questions:

Regarding the change to page 2891 lines 97033-97035: is this because smake / SunPro make will re-consider and re-build included makefiles if they appear as prerequisites to another target? GNU make never reconsiders a target after it's been considered, regardless of whether or not it's an included file.

Regarding this:

> If a target rule or inference rule for the pathname has been parsed before the include line is parsed, make shall use the rule to attempt to create the file or to bring it up-to-date.

I don't think this is quite correct. In GNU make, anyway, it's not required that an inference rule parsed before the include line is parsed will always be used. If a target rule, or a different inference rule, is defined after the include line then that will be used instead.

For the section "Additionally in this case, the new contents of the file, if it is successfully created or updated, shall be used when processing rules for the following targets after the makefile(s) have been read:" followed by the bullet points, can you give me a hint as to what we are trying to ensure or prevent by this text? By listing these specific things I worry that I will overlook something that we do today that will become invalid as a result of this text. Is there a difference we are trying to surface between how included files normally work, and these rebuilt included files work? Is it not sufficient to say something like, the new contents of the file will be used when processing rules that appear after the included file as if the text had appeared in the makefile?

In the portability text (second bullet), a portable workaround to using a macro defined in one included file in the include line of another, is to put the second include _inside_ the first included file instead of in the outer including makefile. This can be less convenient in some situations but it will work in all situations.

In the portability text I don't understand the need for the third bullet...?

In the make examples text I am not sure what the point of the initial -include line is... why is that needed in this situation? It also seems that each target will list all its prerequisites twice, although of course that's not a problem. You might also consider using the make variable $(CC) in these examples rather than hardcoding the c99 value.

Regarding the issue of searching of include files, I don't agree with this as written. I do understand the problem you're addressing, and I have no problems making some types of changes to GNU make to meet the standard, but the loss of functionality here is too great.

Consider an environment where users want to place their included files in a separate directory "ddir" then use "include $(CFILES:.c=.d)" and adds "ddir" to the include file search path. This won't work as expected with this wording.

One way to square this circle may be to provide a bit more context around included search paths; we may need to make a distinction between "user specified search paths" and "default" or "system" search paths. I'm OK with saying that "default" search paths, if the implementation contains any, are only searched for makefiles if the makefile cannot be remade and even that these are not eligible to be rebuilt. But, I think that user-defined search paths shouldn't preclude those makefiles from being rebuilt.

Regarding the make rationale text specifying potential issues to consider with "delayed remaking", I'm not sure that such advice to implementors really belongs in the spec, even in the rationale section, but I have no problems with it if you want to keep it here. But, perhaps instead of saying "a number of ''gotchas''" perhaps just something like "keep these potential QOI issues in mind" :).
(0005096)
psmith (developer)
2020-11-03 15:31

Apropos of the example given here: I just want to say that in the GNU project we have completely moved away from using rebuilt included makefiles as part of automatic prerequisite generation, as being (a) tricky, (b) impossible to make completely seamless, (c) inefficient, and (d) wholly unnecessary.

I wrote a blog post about this which may be interesting: http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/ [^] This post obviously makes use of GNU make-specific features but I don't believe there's anything there that couldn't be also implemented in standard make.
(0005097)
joerg (reporter)
2020-11-04 16:45

Re: Note: 0005096 The most advanced method has been introduced in January 1986 by SunPro Make.

You put this line:

.KEEP_STATE:

or

.KEEP_STATE: state-file-name

into the Makefile and make then manages the include dependencies in a hidden way by setting the "SUNPRO_DEPENDENCIES=" environment to:

SUNPRO_DEPENDENCIES="some/unique/path <tgt>"

with <tgt> set to $@ for the current command.

The compiler then (as a side-effect) writes the dependencies in make syntax into some/unique/path and SunPro Make collects all such output into the file .make.state or in state-file-name. This is safe and increases overall performance as it avoids a second compiler call.

Together with changes to SunPro Make and the UNIX linker from around 1993, based on the environment SGS_SUPPORT=libmakestate.so.1, a callback interface in the linker delivers all needed information on path names of linked libraries. This completely closes all loopholes on dependencies that before typically have been in hierarchical makefile systems.
(0005098)
joerg (reporter)
2020-11-04 22:53

Re: Note: 0005094 my impression is that you are missunderstanding the intention of Note: 0005087

The intention of that text is to describe what may be done in a portable way. This of course cannot include rules related to an include file that appear after the include statement, since this would neither work with SunPro Make, nor with smake.

The text in Note: 0005087 describes the common denominator of smake, SunPro Make and GNU make. It describes makefile variants that would not work with all implementations as being non-compliant.

Makefiles that make use of rules that appear after the related include statement, depend on a vendor specific enhancement. They are not forbidden, but rather based on non-portable extensions.

If you believe that Note: 0005087 depends on a required feature that is not or cannot be supported by GNU make, this would be a reason for a veto, but from what I can say, this does not apply.
(0005099)
dwheeler (reporter)
2020-11-05 00:54

The ability to remake included files is broadly useful, as demonstrated by the many "make" implementations that support this capability. Here's a specific example of a Makefile fragment using it (this example uses several GNU make capabilities that are missing from POSIX, but the point is to show that remaking included files *is* a capability people use): https://github.com/david-a-wheeler/make-booster [^] .

geoffclare: Thank you for creating this compromise text. I think it is absolutely *vital* that it not conflict with the very-widely-used GNU make implementation.

The POSIX specification of "make" lacks many functionalities that are needed, and thus many people end up creating Makefiles specific to a specific implementation (e.g., GNU make). I hope that the POSIX specification will be extended to handle more situations. This is only one step towards that desirable result, but it seems like a good step forward.
(0005100)
geoffclare (manager)
2020-11-05 09:50
edited on: 2020-11-05 10:14

I have updated Note: 0005087 based on comments here and on the mailing list. The changes are:

Restructured the page 2892 change.

Added "(unless the later include line is itself inside the dynamic include file)" in the second bullet in the page 2904 change.

Improved wording in the page 2911 change (avoiding "gotchas").

Appended a note to the final paragraph of the page 2911 change explaining that it only applies to the default search directories.


- Issue History
Date Modified Username Field Change
2020-02-09 17:17 dmitry_goncharov New Issue
2020-02-09 17:17 dmitry_goncharov Name => Dmitry Goncharov
2020-02-09 17:17 dmitry_goncharov Section => (section number or name, can be interface name)
2020-02-09 18:29 shware_systems Note Added: 0004780
2020-02-10 13:27 joerg Note Added: 0004781
2020-02-10 13:28 joerg Note Edited: 0004781
2020-10-26 15:52 geoffclare Project Online Pubs => Issue 8 drafts
2020-10-26 15:53 geoffclare Note Added: 0005066
2020-10-26 15:54 geoffclare Page Number => (page or range of pages)
2020-10-26 15:54 geoffclare Line Number => (Line or range of lines)
2020-10-26 15:54 geoffclare Final Accepted Text => Note: 0005066
2020-10-26 15:54 geoffclare Status New => Resolved
2020-10-26 15:54 geoffclare Resolution Open => Accepted As Marked
2020-10-26 15:54 geoffclare version => Draft 1
2020-10-26 15:54 geoffclare Tag Attached: issue8
2020-10-26 16:54 nick Relationship added parent of 0001415
2020-10-26 16:57 rhansen Note Added: 0005068
2020-10-26 16:58 rhansen Note Edited: 0005068
2020-10-26 17:00 rhansen Note Edited: 0005068
2020-10-26 17:08 nick Note Added: 0005069
2020-10-26 17:08 nick Resolution Accepted As Marked => Reopened
2020-10-26 22:34 nick Status Resolved => Under Review
2020-10-27 10:09 geoffclare Note Added: 0005070
2020-10-27 13:09 joerg Note Added: 0005071
2020-10-27 13:10 joerg Note Edited: 0005071
2020-10-27 14:04 psmith Note Added: 0005072
2020-10-27 14:36 psmith Note Added: 0005073
2020-10-27 15:24 joerg Note Added: 0005074
2020-10-27 15:26 joerg Note Edited: 0005074
2020-10-27 18:28 psmith Note Added: 0005075
2020-10-27 18:30 psmith Note Added: 0005076
2020-10-28 09:14 geoffclare Note Added: 0005077
2020-10-28 11:23 joerg Note Added: 0005078
2020-10-28 12:16 joerg Note Edited: 0005078
2020-10-28 13:47 psmith Note Added: 0005079
2020-10-28 14:44 geoffclare Note Added: 0005080
2020-10-28 14:49 psmith Note Added: 0005081
2020-10-28 14:50 psmith Note Added: 0005082
2020-10-28 14:53 psmith Note Added: 0005083
2020-10-28 15:08 shware_systems Note Added: 0005084
2020-10-30 15:56 geoffclare Note Added: 0005087
2020-10-30 16:10 geoffclare Note Edited: 0005087
2020-10-30 16:14 geoffclare Note Edited: 0005087
2020-10-30 18:52 joerg Note Added: 0005089
2020-10-30 18:52 joerg Note Edited: 0005089
2020-10-30 18:53 joerg Note Edited: 0005089
2020-10-30 18:54 joerg Note Edited: 0005089
2020-10-30 18:56 joerg Note Edited: 0005089
2020-10-30 19:33 joerg Note Edited: 0005089
2020-10-31 09:13 geoffclare Note Edited: 0005087
2020-10-31 09:16 geoffclare Note Edited: 0005087
2020-10-31 09:21 geoffclare Note Added: 0005090
2020-10-31 09:38 geoffclare Note Edited: 0005087
2020-10-31 09:47 geoffclare Note Edited: 0005087
2020-10-31 09:47 geoffclare Note Edited: 0005087
2020-11-03 15:23 psmith Note Added: 0005094
2020-11-03 15:23 psmith Note Added: 0005095
2020-11-03 15:23 psmith Note Deleted: 0005095
2020-11-03 15:31 psmith Note Added: 0005096
2020-11-04 16:45 joerg Note Added: 0005097
2020-11-04 22:53 joerg Note Added: 0005098
2020-11-05 00:54 dwheeler Note Added: 0005099
2020-11-05 09:41 geoffclare Note Edited: 0005087
2020-11-05 09:41 geoffclare Note Edited: 0005087
2020-11-05 09:50 geoffclare Note Added: 0005100
2020-11-05 10:04 geoffclare Note Edited: 0005087
2020-11-05 10:11 geoffclare Note Edited: 0005087
2020-11-05 10:13 geoffclare Note Edited: 0005087
2020-11-05 10:14 geoffclare Note Edited: 0005100
2020-11-05 16:23 geoffclare Final Accepted Text Note: 0005066 => Note: 0005087
2020-11-05 16:23 geoffclare Status Under Review => Resolved
2020-11-05 16:23 geoffclare Resolution Reopened => Accepted As Marked
2020-12-09 09:56 geoffclare Status Resolved => Applied
2022-10-06 15:59 geoffclare Relationship added has duplicate 0000328


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker