Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0001505 [Issue 8 drafts] Shell and Utilities Comment Omission 2021-08-10 10:14 2022-01-17 11:42
Reporter quinq View Status public  
Assigned To
Priority normal Resolution Reopened  
Status Under Review   Product Version Draft 2
Name Quentin Rameau
Organization
User Reference
Section EXTENDED DESCRIPTION
Page Number 2935
Line Number 98369
Final Accepted Text Note: 0005549
Summary 0001505: Make doesn't seem to specify unset macro expansion behaviour
Description Hello,

I was looking for an actual specification of
what happens when trying to expand an undefined macro.

Intuition would be that it expands to an empty string,
but this doesn't seem to be specified at all,
or maybe I missed that somewhere else.

Could an implementation be allowed
to generate an error on such case (or other behaviour)
without having this defined?
Desired Action Specify that trying to expand an unset macro
results in an empty string.
Tags issue8
Attached Files

- Relationships

-  Notes
(0005432)
geoffclare (manager)
2021-08-10 11:23

I agree the standard doesn't say what happens if make tries to expand an unset macro.

However, I'm not sure adding a requirement that it expands to an empty string would be useful. This is because applications cannot rely on any macro being unset just because they have not set it. Under "Default Rules" the standard says "Implementations may provide additional macros and rules." So any macro name the application tries to use without setting might happen to be the name of a macro the implementation provides, and thus could expand to something unexpected.

It would be better to state that the behaviour is unspecified and warn that applications should explicitly set macros to an empty string if they want them to expand to an empty string.
(0005433)
joerg (reporter)
2021-08-10 12:43

We already have:

If the original value of the variable named by string1 is an empty string, the final result shall be an empty string.

in the current draft of Issue 8.
(0005434)
geoffclare (manager)
2021-08-10 13:30

Re: Note: 0005433 That's an addition related to pattern macro expansions. See https://austingroupbugs.net/view.php?id=519#c1206 [^]

And it has a mistake: "variable" should be "macro"!

We should fix that as part of the resolution of this bug.
(0005435)
joerg (reporter)
2021-08-10 13:44

There is another mistake: it says "word", where np ns with no % in between is ment.
(0005436)
kre (reporter)
2021-08-10 14:28

Re Note: 0005432

   I'm not sure adding a requirement that it expands to an empty string
   would be useful.

From the perspective you're looking at it, you're right, applications
should not depend upon a macro being undefined, and hence necessarily
an empty string when expanded, but that's not the only issue.

   It would be better to state that the behaviour is unspecified

That would be unfortunate, as, as one possible implementation, that could
lead to a situation where an implementation expanded undefined macros as
the name of the macro, rather than as an empty string (or an error).

Then when an application expects a macro to be defined (eg: CC) and does

     $(CC) file.c

in some rule, and the implementation instead of doing the expected invocation
of the C compiler, instead invokes the CC command (which might be anything
at all), that would have potential for very bad things (TM) to happen.

So, I think the standard needs to define how undefined macros are treated
when encountered - either as an error, or an empty string (and I think that
the empty string version is what is actually done in practice, so that one
should win), rather than simply say "unspecified". There can be an
APPLICATION USAGE note telling makefile writers not to depend upon a macro
being undefined, if anyone really believes this is a real problem - I mean,
why could anyone write
   whatever ${FOO} and more
expectig FOO to be undefined, and so to generate
   whatever and more
when they could have just written that in the first place? So I think
this issue most probably isn't real (ie; no-one expects macros to be
undefined) but one often expects them to be defined in situations where
they might in fact not be.
(0005437)
quinq (reporter)
2021-08-11 10:55

Re: 0005432

That is a good point indeed about implementations being authorized to provide additional macros.

The solution of specifying that this is unspecified, and adding a warning note sounds good to me.

Re: 0005436

While I like the idea of having a defined behaviour, the bottom line is “don't do it” so it doesn't really matter there. As you said, it seems that most implementations (?) expand to an empty string, but $(MYTOOL) file could still expand to MYTOOL if an implementation decided to provide that specific macro anyway.

As for the use cases of having empty macros, for example that could happen on variable file inclusions where some macros are defined, or not.
Other cases could happen too, but in the end yes, correct solution is to explicitely set them if they are to be used.
(0005549)
geoffclare (manager)
2021-12-09 17:13

On draft 2.1 page 2943 line 98732 change:
that completely matches the [op]%[os] pattern on the left hand side
to:
that completely matches, in a case-sensitive manner, the [op]%[os] pattern on the left hand side

On draft 2.1 page 2944 line 98750 change:
If the original value of the variable named by string1 is an empty string, the final result shall be an empty string. Matching shall be case-sensitive.
to:
In all forms of macro expansion, if the value of the macro named by string1 is an empty string, the final result shall be an empty string. If the macro named by string1 does not exist, it is unspecified whether the final result is an empty string or make treats it as an error. The application should ensure that all macros used by make exist (and are explicitly set to an empty string if expected to expand to an empty string).
(0005551)
psmith (developer)
2021-12-09 17:37

I don't see any good reason to leave this undefined. Is there any implementation of make anywhere that ever treated this as an error? Why don't we just define the behavior to be what all make programs always do, which is expand to the empty string?

Virtually every makefile in use today expects and relies on the fact that a variable will expand to the empty string if not set; for example, a large number of makefiles have a VERBOSE variable which is not set by default, and if you set it that variable will enable a verbose mode. The most common way this is used is to add something like this to the makefile:

    $(VERBOSE).SILENT:

Now if VERBOSE is not set (the default) you get silent mode; if VERBOSE is set to something silent mode is disabled.

Why would we immediately turn all these into non-conforming makefiles, for no reason?
(0005560)
kre (reporter)
2021-12-17 13:50

This issue needs to be reopened to deal with the issues raised by
Note: 0005551 and continued, at length, on the mailing list.
(0005572)
geoffclare (manager)
2021-12-21 10:45

It was pointed out on the mailing list that the += assignment is another way to use an unset macro, and therefore if expansion is treated as an error, use with += should be an error as well.

I suggest adding this change:

On draft 2.1 page 2949 line 98678, change:
If the macro named by string1 does not exist, this form shall be equivalent to ...
to:
If the macro named by string1 does not exist, make shall treat this as an error if and only if an attempt to expand the macro named by string1 would be treated as an error; if it is not treated as an error, this form shall be equivalent to ...

It has also become clear that some rationale is needed explaining the problem. However, there is no point working on that until a decision has been made on what the normative text should say.

We may also want to add something to application usage saying how ?= can be used to bring in specific macro values from the environment but advising of the risks of doing so.
(0005573)
psmith (developer)
2021-12-21 15:24

As I said on the mailing list, I reject any wording in the standard which allows any POSIX-conforming make implementation to fail when it expands an unset variable. No such change should be accepted into the standard.

I'm fine with changes that allow (but do not require) conforming make implementations to generate diagnostics when they expand an unset variable, but the variable must be expanded to the empty string and make must not be allowed to fail.

On the mailing list I wrote:

The arguments for this change boil down to "we don't think people should do this because it's not reliable" and the arguments against it boil down to "every implementation does this and most every makefile relies on it working this way". Both can be true, but the latter must outweigh the former when deciding what behavior to standardize.

If makefile authors want to ensure that their makefiles are not subject to this kind of unreliability they are free to write their makefiles to avoid it. Whether an implementation assists them in that or not is a QoI issue.
(0005574)
dwheeler (reporter)
2021-12-21 16:26

All make implementations, to my knowledge, expand an unset macro to an empty string. Many makefiles depend on allowing unset macros; it's quite normal to use macros like DESTDIR that are expected to be empty by default but can be set by someone to change the behavior. Treating unset macros by default as "errors" or "unspecified behaviors" or "undefined behaviors" just disconnects the specification from reality. "Unset becomes empty, without warnings", needs to be the default case.

If the committee wants to create an option and/or special target to enable warnings, that would make sense. However, how can the makefile determine if a macro is or is not empty, and do something based on that condition, if any access is an error? Allowing some macros to be empty is a common use case. If the option and/or special target is added, then other functionality (such as a useful if conditional like GNU make's) should be added to support allowing *some* macros to be unset.
(0005575)
psmith (developer)
2021-12-21 19:20

As long as the makefile proceeds and doesn't fail I'm OK with the standard allowing a conforming implementation to generate diagnostics if it wants to (it mustn't be required though). As long as the build doesn't fail so that users don't have to write their makefiles to avoid this to ensure the build succeeds, then whether or not, and how and when, implementations generate diagnostics even in default invocations doesn't need to be restricted by the standard IMO.

Of course that's just my opinion. I won't be changing GNU make to generate diagnostics by default and I doubt any other implementation will do so either so the issue may be moot.
(0005576)
quinq (reporter)
2021-12-21 20:18

I think that's fine if an implementation doesn't want to follow a standard as long as it doesn't claim to do so, independantly of this specific ticket resolution.

I suppose that's partly why POSIX introduced the special .POSIX target in order to allow implementations to behave differently while giving makefile authors a way to ensure compatibility when required (or being notified by the implementation somehow that it doesn't entirely follow POSIX).
(0005616)
geoffclare (manager)
2022-01-17 11:42

In the Jan 13 teleconference it was decided that we should change to a resolution that requires unset macros to expand to an empty string, and deals with the issue of unexpectedly set/unset macros entirely in non-normative text. I took an action to propose new wording and this is my proposal...

All page and line numbers are for Issue 8 draft 2.1.

On page 2943 line 98732 change:
that completely matches the [op]%[os] pattern on the left hand side
to:
that completely matches, in a case-sensitive manner, the [op]%[os] pattern on the left hand side

On page 2944 line 98750 change:
If the original value of the variable named by string1 is an empty string, the final result shall be an empty string. Matching shall be case-sensitive.
to:
In all forms of macro expansion, if the value of the macro named by string1 is an empty string, or if the macro named by string1 does not exist, the final result shall be an empty string.

<small>Note:It is not safe to assume that a macro which has not intentionally been set to a specific value will not exist. See APPLICATION USAGE for more information.</small>

On page 2951 after line 99038, add the following new paragraphs to APPLICATION USAGE:
Although make expands macros that do not exist to an empty string, makefile authors should be aware that it is not safe to assume that a macro which has not intentionally been set to a specific value will expand to an empty string for everyone who uses the makefile. There are two reasons for this:

1. When another user executes make, they might happen to have an environment variable of the same name (which they have set for some unrelated purpose) with a non-empty value.

2. A different implementation of make (or a later version of the same implementation) might have a non-empty value for the macro in its default set.

This is one aspect of a more general problem, which is that any macro that is not one for which this standard requires a default value, and is not explicitly set either in the makefile or on the make command line, can have an unexpected value (or unexpectedly not exist) when the makefile is used by a different user or with a different make implementation.

For this reason, it is recommended that makefile authors do not design makefile schemes in which values for non-standard macros are obtained from the user's environment variables. Safer methods of allowing users to configure macro values include:

* Setting the macros to default values in a make include file where the user can edit the values.

* Executing make from one or more wrapper scripts which set macro values on the command line (and which do not obtain those values from environment variables).

Makefile authors who follow this recommendation may wish to check for any macros they have overlooked by using a make implementation that provides, as an extension, a command-line option that causes make to report attempts to expand (or append to) macros that do not exist. Users of makefiles written by others can also benefit from use of such an option to detect the opposite problem (where the author had a macro being set from an environment variable but the user does not have the variable set). This can avoid misbehaviors that would otherwise be hard to debug, such as a file-processing utility reading from standard input because it was not given any pathnames to process.

Makefile authors who choose not to follow the recommendation can minimize the risk of misbehavior by ensuring all non-standard macros have names that begin with a suitable project-specific prefix.

Use of the -e option is strongly discouraged, as it makes the problem discussed above even more likely by introducing the possibility of unexpected values occurring even for macros set in the makefile. If a specific macro needs a value from the environment to override a value set in the makefile, it is safer to set just that macro on the command line, using for example <tt>make MYPROJ_FOO="$MYPROJ_FOO"</tt>. Alternatively, the makefile can be modified to use the <tt>?=</tt> assignment operator for that macro.

On page 2956 after line 99259 add two new paragraphs to RATIONALE:
Implementations are permitted to include any macro of their choosing in the default set. However, they are encouraged to keep such additions to a minimum in order to reduce the risk of name clashes with user macros.

Implementations are encouraged to provide, as an extension, a command-line option that causes make to report attempts to expand (or append to) macros that do not exist. See APPLICATION USAGE for the intended use cases of such an option.

On page 2960 after line 99479 add a new paragraph to FUTURE DIRECTIONS:
A future version of this standard may add a command-line option that causes make to report attempts to expand (or append to) macros that do not exist.

- Issue History
Date Modified Username Field Change
2021-08-10 10:14 quinq New Issue
2021-08-10 10:14 quinq Name => Quentin Rameau
2021-08-10 10:14 quinq Section => EXTENDED DESCRIPTION
2021-08-10 10:14 quinq Page Number => 2935
2021-08-10 10:14 quinq Line Number => 98369
2021-08-10 10:46 geoffclare Project 2008-TC2 => Issue 8 drafts
2021-08-10 11:23 geoffclare Note Added: 0005432
2021-08-10 12:43 joerg Note Added: 0005433
2021-08-10 13:30 geoffclare Note Added: 0005434
2021-08-10 13:44 joerg Note Added: 0005435
2021-08-10 14:28 kre Note Added: 0005436
2021-08-11 10:55 quinq Note Added: 0005437
2021-12-09 17:13 geoffclare Note Added: 0005549
2021-12-09 17:14 geoffclare Final Accepted Text => Note: 0005549
2021-12-09 17:14 geoffclare Status New => Resolved
2021-12-09 17:14 geoffclare Resolution Open => Accepted As Marked
2021-12-09 17:15 geoffclare Tag Attached: issue8
2021-12-09 17:37 psmith Note Added: 0005551
2021-12-17 13:50 kre Note Added: 0005560
2021-12-21 10:45 geoffclare Note Added: 0005572
2021-12-21 10:45 geoffclare Status Resolved => Under Review
2021-12-21 10:45 geoffclare Resolution Accepted As Marked => Reopened
2021-12-21 15:24 psmith Note Added: 0005573
2021-12-21 16:26 dwheeler Note Added: 0005574
2021-12-21 19:20 psmith Note Added: 0005575
2021-12-21 20:18 quinq Note Added: 0005576
2022-01-17 11:42 geoffclare Note Added: 0005616


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker