Austin Group Defect Tracker

Aardvark Mark III


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0000337 [1003.1(2008)/Issue 7] Shell and Utilities Objection Enhancement Request 2010-10-17 17:05 2013-11-03 21:46
Reporter dwheeler View Status public  
Assigned To ajosey
Priority normal Resolution Open  
Status Under Review  
Name David A. Wheeler
Organization
User Reference
Section make
Page Number 2908-2929
Line Number 95549-95822
Interp Status ---
Final Accepted Text
Summary 0000337: Add standard support for "!=" (macro shell assignment) in make
Description Many "make" implementations make it possible to invoke a shell command with various options (which may themselves be expanded makefile macros), and assign the result to a variable. This single capability also makes many capabilities easy (e.g., implementing wildcards, creating simple makefiles that "compile all .c files in this directory", and so on).

Unfortunately, while there are many non-standard ways to do this, there is no *standard* way to invoke this functionality. So, I propose that some mechanism that is already in *some* implementations be standardized.

Specifically, I propose that the "!=" operator be defined (I'll call this the shell-assignment macro definition). This is ALREADY defined in the "make" of FreeBSD, OpenBSD, and NetBSD:
http://www.freebsd.org/cgi/man.cgi?query=make&sektion=1 [^]
http://www.openbsd.org/cgi-bin/man.cgi?query=make&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html [^]
http://netbsd.gw.com/cgi-bin/man-cgi?make+1+NetBSD-current [^]

Here's the relevant OpenBSD description of this:
   != Expand the value and pass it to the shell for execution and
           assign the result to the variable. Any newlines in the result
           are replaced with spaces (BSD extension).
     Any whitespace before the assigned value is removed...

Thus:
  VAR != echo "$(STUFF)" | sed -e 's/://'
Expands the $macros after !=, then sends that to /bin/sh, replaces newlines with spaces in the result, and IMMEDIATELY assigns that to VAR as a "normal" make variable.

I can't find any documentation about unescaped '#' in this construct; I have a proposal below, but I'd like to know if that should be modified.

*NOT* all makes have this particular syntax, although I suspect many
other makes have some sort of functionality that can do the same thing.
I know that GNU make does *not* implement this particular construct,
but it doesn't use this syntax for anything else and thus it probably
would be easy to add. If you look at GNU make's documentation:
  http://www.gnu.org/software/make/manual/make.html [^]
GNU make can do the same thing with:
  VAR = ${system echo "$(STUFF)" | sed -e 's/://'}

This can be "sort-of" worked around by using a makefile to generate another another makefile, and then including it, but that is an extremely convoluted and hard-to-read way of doing a simple operation. Simple and common operations should be simple, easy to express, and easy to read.

It's also believe it'd be REALLY easy for existing implementations to implement this, if they don't already.

In the proposed "desired action" below I clarified some stuff that isn't clearly documented as far as I can tell.
The "remove all trailing newlines" isn't documented, but I believe that's what current implementations do. The "ignore '#'" is stated so that you can easily include hash symbols in calls to the shell.
I welcome additional comments to improve this proposal.
Desired Action Add:

"The following form defines a shell-assignment macro (replacing any previous definition of the macro named by string1):
  string1 != string2

In a shell-assignment macro, string2 is defined as all characters up to (but not including) an unescaped <newline>, even those an unescaped <#>. Any <blank> characters immediately after the <equals-sign>, or before the <exclamation-point>, shall be ignored.

After a shell-assignment macro definition,
the macro named string1 is defined as having the value after the following process: First, string2 is immediately evaluated. Second, that resulting evaluation is passed to the shell for execution. Third, that result is modified by removing all whitespace at the beginning, removing all trailing newlines, and then replacing all remaining newlines with spaces."

{If comment 330 is accepted, state that
"In an implementation with "single evaluation references", the resulting a is "recursively-expanded".}
 

Tags No tags attached.
Attached Files

- Relationships
related to 0000330Resolvedajosey In make, add support for assignments "::=", "+=", and "?=". 

-  Notes
(0000579)
mirabilos (reporter)
2010-10-18 16:37

Hrm. Existing practice:

tg@blau:~ $ cat x; make -f x
foo!= echo bar#baz
all:
        echo '<${foo}>'
echo '<bar>'
<bar>

I don’t believe OpenBSD changed this since I forked.
MidnightBSD (FreeBSD fork) behaves the same.

https://www.mirbsd.org/man1/make [^] for your convenience,
although it doesn’t differ much from OpenBSD’s…

+1 for the feature though
(0000580)
yaneurabeya (reporter)
2010-10-18 17:34

FreeBSD keeps a pretty good archive of the 2 other primary BSD releases, as well as a few other projects; based on my snooping ( http://www.freebsd.org/cgi/man.cgi?query=make&apropos=0&sektion=0&manpath=OpenBSD+4.7&format=html [^] ), they still have the != operator (and the requirements from the manpage are the same as FreeBSD CURRENT's copy of pmake).
(0000581)
dwheeler (reporter)
2010-10-18 17:41

Okay, based on existing practice, change the "desired action" text so that the comment character '#' immediately ends the string to be processed by the shell. Here's some new proposed "desired action" text:

"The following form defines a shell-assignment macro (replacing any previous definition of the macro named by string1):
  string1 != string2

In a shell-assignment macro, string2 is the set of characters up to (but not including) up to a comment character (’#’) or an unescaped <newline>. Any <blank> characters immediately after the <equals-sign>, or before the <exclamation-point>, shall be ignored.

After a shell-assignment macro definition,
the macro named string1 is defined as having the value after the following process:
First, string2 is immediately evaluated. Second, that resulting evaluation is passed to the shell for execution. Third, that result is modified by removing all whitespace at the beginning, removing all trailing newlines, and then replacing all remaining newlines with spaces."

{If comment 330 is accepted, state that
"In an implementation with "single evaluation references", the resulting a is "recursively-expanded".}


NOTE 1: This is conceptually the same,
but not *QUITE* the same as GNU make's ($system ...).
In fact, I can't find a GNU construct that EXACTLY does this.
In many cases, this GNU make construct will have the same final result:
  VAR = ${system echo "$(STUFF)" | sed -e 's/://'}
but notice that $(VAR) is re-evaluated *EVERY TIME*, creating efficiency problems,
and the change in result may be surprising.
GNU make's ":=" doesn't quite do it either:
  VAR := ${system echo "$(STUFF)" | sed -e 's/://'}
In this case, in GNU make it's executed only once, but then you cannot insert $(VAR2)
as the output of the shell script and have it be re-evaluated.
In most cases using one or the other would be fine, and the $(system ...) function
in GNU clearly shows that there's value in being able to invoke the shell when
evaluating make variable values.
In any case, I think the fact that there's no exact equivalent in GNU make
is a non-problem. I looked at the GNU make source code, and it looks like this
would be pretty easy to add. I don't know if the GNU make maintainers would
accept a patch to do this, but there are some good rationales:
1. Many other "makes" already support this specific syntax and construct
2. The proposed syntax conflicts with nothing already in GNU make.
3. Nothing in GNU make does this exactly... so those wanting this functionality
   will want it added.
In my mind, "it's in the POSIX spec" is a *really* good reason to add this
to GNU make... but that argument only works if it's *in* the POSIX spec :-).


NOTE2: This version of the proposal makes the '#' character a little
harder to enter, but it's not TOO hard. The "!=" command itself can come to the rescue:
# Presume ASCII or UTF-8:
hash != printf '\043'
foo != echo bar$(hash)baz

NOTE3: Thanks for the "+1 for the feature" comment!
(0000582)
dwheeler (reporter)
2010-10-18 19:37

I installed FreeBSD and tried out its "make", to clarify current practice. (The documentation omits important stuff that really needs to be in a standard.) Based on that experimentation, here are additional tweaks to the proposal above, based on current practice.

In short, in the previous proposal, replace:
"Third, that result is modified by removing all whitespace at the beginning, removing all trailing newlines, and then replacing all remaining newlines with spaces."
with this:
"Third, that result is modified removing a single trailing newline (if there is one), and then replacing all remaining newlines with spaces."

I tested and confirmed that in FreeBSD make, "make demo=HELLO" will cause a "demo != ..." to be ignored, just like a "demo = ..." statement. If you specify the macro value on the command line, make will use that preset value instead of calling out to the shell. That makes sense, and I believe that is exactly what the POSIX rules imply in lines 95814-95830 as they are, so I don't think that needs to change.

I can't possibly test on all "make" implementations, but I'd like to know of any makes with different semantics for "!=", or if they have a construct that renders it impossible to add "!=".

Here's my demo input makefile, with the resulting output on FreeBSD release 8.1.

=================================
.POSIX:

demo1!= printf ' 1 2 3\n4\n\n5 \n \n 6\n\n\n\n'
demo2!= printf '7 8\n '
demo3!= printf '$$(demo2)'
foo1!=echo bar#baz
hash != printf '\043'
foo2!= echo "bar$(hash)baz"


all:
    @echo "123456789012345678901234567890"
    @printf '%s\n' "demo1=Z$(demo1)Z"
    @printf '%s\n' "demo2=Z$(demo2)Z"
    @printf '%s\n' "demo3=Z$(demo3)Z"
    @printf '%s\n' "foo1=${foo1}"
    @printf '%s\n' "foo2=$(foo2)"

# OUTPUT ON FREEBSD 8.1 IS:
# 123456789012345678901234567890
# demo1=Z 1 2 3 4 5 6 Z
# demo2=Z7 8 Z
# demo3=Z7 8 Z
# foo1=bar
# foo2=bar#baz
=================================
(0000583)
dwheeler (reporter)
2010-10-19 03:50

Here is some additional (and encouraging) information, suggesting that "!=" is already widely-implemented with standard semantics.

I tried "make" on NetBSD 5.0.2, and "!=" works exactly the same way as FreeBSD 8.1. I suspect they have a common source code ancestor, but no matter, that's encouraging.

The "pmake" program by Adam de Boor appears to have the same semantics, at least as far as this documentation seems to say:
http://www.freebsd.org/doc/en/books/pmake/variables.html [^]
(0000584)
psmith (reporter)
2010-10-19 04:27

David: can you be more clear about the difference between GNU make's := $(shell ...) construct and the proposed != construct? You seem to be just about to get into it above but then veer off and refer to VAR2, which as far as I can see is never mentioned before or since, and I couldn't follow it:

> GNU make's ":=" doesn't quite do it either:
> VAR := ${system echo "$(STUFF)" | sed -e 's/://'}
> In this case, in GNU make it's executed only once, but then you cannot insert $(VAR2)
> as the output of the shell script and have it be re-evaluated.

I don't know what this means...?

Thanks!
(0000586)
dwheeler (reporter)
2010-10-19 17:12

> David: can you be more clear about the difference between GNU make's := $(shell ...) construct and the proposed != construct?

I'll try. It's subtle.

Like all makes, in GNU make when you say this:
 VAR1 = $(VAR2)
The value of VAR1 is just "$(VAR2)" - it is NOT evaluated before it is stored. (I'm sure you know this, but bear with me.) This behavior is also true for functions like $(shell ...), so:
 VAR1 = $(shell date)
is NOT evaluated when VAR1 one is set. Instead, it is evaluated every time you later evaluate VAR1, e.g., when you evaluate $(VAR1), it pulls in $(shell ...) and evaluates that. Note that the function $(shell ...) is executed, the result of the shell command is the result; the shell's result is not RE-evaluated. This is very inefficient, and can be a big problem if re-executing the shell command will be a problem.

The typical GNU make user's way to avoid re-evaluation is typically:
 VAR1 := $(shell date)
this works by completely evaluating the text after "shell", then sending that result to shell, and then setting VAR1. But in GNU make, the ":=" setting guarantees that VAR1 will *NEVER* be re-evaluated again (this is different than the BSD behavior) - when VAR1 is evaluated, it ONLY returns its first-evaluated result. So if the shell returns a value like $(VAR3), and you reference $(VAR1), then then the $(VAR1) will just return the characters $, (, V, A, R, 3, ). It will NOT recurse and evaluate $(VAR3) as part of evaluating $(VAR1) in this case.

In contrast, in the BSD "!=" command, the line is evaluated, sent to the shell, and the result (with a few tweaks) is stored in the variable. But the BSD makes don't have two different "flavors" of variable; variables are ALWAYS re-evaluated when referenced. So, if the shell returns a make value like $(VAR), and != has that stored, then the $(VAR) will be evaluated itself in the BSDs. The only way to do that in GNU make is to basically do self-generating makefiles that are then included (!) - which are tricky.

To demonstrate this, I have posted two files:
* http://www.dwheeler.com/misc/makefile.bsd [^]
  This demonstrates the != semantics from BSD.
* http://www.dwheeler.com/misc/makefile.gnu [^]
  This demonstrates the $(shell ... ) semantics of GNU make.

In the BSD version you say:
XYZZY = fiddle-dee-dee
dollar = $$
VAR3 != printf '%s' '$(dollar)(XYZZY)'
all:
    @printf '%s\n' 'VAR3=<$(VAR3)>'

This prints "VAR3=<fiddle-dee-dee>". That's because VAR3 was set to "$(XYZZY)"
by the "!=" command, and when $(VAR3) is evaluated, the $(XYZZY) is recursively resolved.


In contrast, in the GNU version you might say:
XYZZY = fiddle-dee-dee
dollar = $$
VAR3 = $(shell printf '%s' '$(dollar)XYZZY')
all:
        @echo 'VAR3=<$(VAR3)>'

This prints "VAR3=<$XYZZY>", and NOT "VAR3=<fiddle-dee-dee>".
Note that VAR3 is *NOT* recursively evaluated in the GNU make case.

There are advantages to the various semantics (GNU vs. BSD).
I'm trying to find some semantic that *EVERYONE* could support
(either now, or one they could plausibly add).
The BSD semantics for "!=" are supported by many, and appear
to be relatively easy to add by everyone else. So, let's add that one.
This capability adds a *LOT* of power to makefiles, because with this,
you can run arbitrary shell commands and bring the results into makefile variables.


P.S. Here's a larger GNU makefile and a repeat of the BSD makefile,
demonstrating GNU make and BSD make semantics (per the URLs listed above):
======================== GNU make ===============================================
# This won't make any difference:
.POSIX:

# This increments the value in "count" each it is evaluated:
VAR1 = $(shell if [ ! -f count1 ]; then echo 0 > count1; else expr `cat count1` + 1 > count1; fi ; cat count1 )

# This is evaluated only once, here:
VAR2 := $(shell if [ ! -f count2 ]; then echo 0 > count2; else expr `cat count2` + 2 > count2; fi ; cat count2 )

# In GNU make, when $(shell...) returns $(XYZZY), it will NOT be
# re-evaluated later to produce 'xyzzy' from $(XYZZY) using EITHER "=" OR ":=":
XYZZY = fiddle-dee-dee
dollar = $$
VAR3 = $(shell printf '%s' '$(dollar)XYZZY')
VAR4 := $(shell printf '%s' '$(dollar)XYZZY')

all:
        @rm -f count1 count2
        @echo 'VAR1=<$(VAR1)>'
        @echo 'VAR1=<$(VAR1)>'
        @echo 'VAR2=<$(VAR2)>'
        @echo 'VAR2=<$(VAR2)>'
        @echo 'VAR3=<$(VAR3)>'
        @echo 'VAR4=<$(VAR4)>'

# GNU make outputs:
# VAR1=<0>
# VAR1=<1>
# VAR2=<0>
# VAR2=<0>
# VAR3=<$XYZZY>
# VAR4=<$XYZZY>


======================== BSD make ===============================================

# This demonstrates the semantics of '!=' on many BSD makes (and similar).
# Your make must implement '!=' for this to work!

.POSIX:

# G
demo1!= printf ' 1 2 3\n4\n\n5 \n \n 6\n\n\n\n'
demo2!= printf '7 8\n '
demo3!= printf '$$(demo2)'
foo1!=echo bar#baz
hash != printf '\043'
foo2!= echo "bar$(hash)baz"

# Note that variables are re-evaluated later, so the shell can output
# a value like $(XYZZY) as part of !=. The $(XYZZY) will be EVALUATED
# when the value containing it is evaluated. On the negative side, this
# means if you don't want this, you need to escape dollar signs as $$.
# On the positive side, it means that shell programs can output macros
# that are then evaluated as they are traditionally evaluated.. and that
# you can use traditional macro evaluation semantics to implement !=.
XYZZY = fiddle-dee-dee
dollar = $$
VAR3 != printf '%s' '$(dollar)(XYZZY)'


all:
    @echo " 123456789012345678901234567890"
    @printf '%s\n' "demo1=<$(demo1)>"
    @printf '%s\n' "demo2=<$(demo2)>"
    @printf '%s\n' "demo3=<$(demo3)>"
    @printf '%s\n' "foo1=<${foo1}>"
    @printf '%s\n' "foo2=<$(foo2)>"
    @printf '%s\n' 'VAR3=<$(VAR3)>'

# OUTPUT ON FREEBSD 8.1 IS:
# 123456789012345678901234567890
# demo1=< 1 2 3 4 5 6 >
# demo2=<7 8 >
# demo3=<7 8 >
# foo1=<bar>
# foo2=<bar#baz>
# VAR3=<fiddle-dee-dee>
(0000629)
eblake (manager)
2010-12-02 16:53

This issue was discussed, but consensus was that issue 0000330 must be resolved
first before this issue can be resolved.

Glenn Fowler wrote in sequence 14841:
if we don't step back and take a abstract view of the current make discussion
we will die a death of 1K features from your-favorite make -- each one worth
about 100 messages on the list

we also need make clarify/respect the difference between "easy to implement"
vs. damage done to abstract/semantic/syntactic models

for example

variable assignment/expansion flavors are a problem between make implementations
the differences are exposed by combinations of the = and := assignment operators

the proposal for != is problematic because it introduces a new variable expansion
semantic for some of the current implementations, and it is a syntax-specific solution
to a problem that has been solved in a general way in some implementations

3 assignment operators capture the differing semantics

    var-expanded-later = value-not-expanded-now
    var-not-expanded-later <now>= value-expanded-now
    var-expanded-later <later-now>= value-expanded-now

so, instead of the specific !=, propose the 3 assignment ops and nail
down the syntax/semantics once and for all

then add generalized $(<function> ...) expansions
and specifically propose the $(shell ...) expansion function

the result will be 3 assignment operators that will work the same in all
compliant implementations and provide a portable mechanism for emulating
the union of { = := != $(shell ...) }

to avoid confusion the <now>= and <later-now>= assigment ops can be chosen
from ones not in use by current make implementations
(guessing ~= @= %= ^= &= *= -= |= ;= <= >= /= )

current implementations can continue to support { := != }
this provides a migration path from implementation-specific to portable constructs
(0001012)
dwheeler (reporter)
2011-11-15 17:45

GNU make's trunk has added support for the "!=" operator, with *exactly* the same semantics as the *BSD makes as described above. Briefly, the command is executed right then, and its result is stored in the make variable; the variable is then always re-expanded later on use, like any "normal" make variable.

I think this is a good argument for adding "!=", *regardless* of bug #330. The "!=" is orthogonal from whatever other "generalized" operators are added to make, it is widely implemented, and it only requires the existence of variables that are expanded on every use (something all compliant makes *already* do).
(0001391)
dwheeler (reporter)
2012-10-05 20:47

Per comment 629, there was a consensus that issue #330 had to be resolved first. Issue 330 has been resolved (it was accepted).

Therefore, it is time to return to this issue.

The objections in sequence 14841 have, for the most part, been overtaken by events. That comment objected to the creation of flavors, but the notion of flavors has been discussed and accepted in issue 330, per the justifications given in that issue. The "!=" is already widely-implemented in the *BSD makes. Both the "!=" operator, and the newly-approved "::=" operator, have already been added to the trunk version of GNU make as well (see http://cvs.savannah.gnu.org/viewvc/make/variable.c?root=make&view=log). [^] In short, sequence 14841 proposes a whole new set of operators that are not implemented in any existing "make". In contrast, the "!=" is *already* widely implemented, and for many, does the job.

I do agree that adding macro functions to make would be valuable; I have already proposed them in issue 512. But for the simple and useful case of "invoke a shell command and stuff its result into a make variable", it is still convenient to have a shorthand, even if they are also accepted. This issue proposes that shorthand, one which is already widely implemented.
(0001870)
dwheeler (reporter)
2013-10-10 03:49

FYI, the official release of GNU Make 4.0 now also supports "!=" (in addition to "::=" of #330). Details here: http://lists.gnu.org/archive/html/make-w32/2013-10/msg00021.html [^]

That means that the *BSD makes _and_ GNU make, at least, support "!=". I think that's even an even stronger argument for adding "!=" to POSIX.
(0001955)
shware_systems (reporter)
2013-11-03 21:46

I was looking over still Opens, and this appears mostly ready to go, whether #629 or #1391 are the basis for resolution, with one concern and a suggestion, and I do have a couple concerns about #0000330 that can be said to relate. That can't have notes added to it any longer so I'm just adding it here rather than start a new ERN. As it's an Issue 8 target, I think it can just be reopened and further discussion and proposed text can be edited there if this seems warranted.

Side Note: GNUs docs for make are off. #330 wasn't added to the standard in 2012, its addition to Issue 8 was decided then, and so ?=, +=, and ::= are still extensions, not normative.

For here, I think it should be explicit that if string1 is already defined, the flavor used to create string1 should apply to how the command output after string2 is executed is evaluated. When undefined the default of treating it as a 'string1 = command output' alias is ok. To change flavor an undefine or direct assignment is needed anyways. This would be consistent with the expectations of '+=' now, but a replace rather than append and a separate operator isn't needed to permit an effective 'string1 ::= command output'. It may be useful to consider '!!=' as an additional operator that does appends in a way consistent with below.

For #330, The Accepted Text for '+=' has:
* If the macro named by string1 exists and is an immediate-expansion
macro, then a <space> or <tab> character followed by the evaluation
of string2 shall be appended to the value currently assigned to the
macro named by string1.
* If the macro named by string1 exists and is a delayed-expansion
macro, then a <space> or <tab> character followed by the unevaluated
string2 shall be appended to the value currently assigned to the
macro named by string1.

My concern is that requiring the addition of the <space> or <tab> may prevent this from being used to build macros that represent filenames or paths or other values where <blanks> aren't allowed. I understand having the separator added can be useful to build multiple argument trailers to a command, so I'm not against that. I think the other usage shouldn't be blocked, however. I suggest that if the form "string2" is used that, after a conversion of escape sequences, evaluations take place or not as stated above of string2 but neither the separator or quotes are appended. Leading separators inside the quotes I'd consider to be intended as part of string2, not trimmed off. This would prevent makefiles from needing to use string1 = string1string2, or ::=, to get a similar effect.

Also, "<space> or <tab>" is ambiguous. I'd be more comfortable seeing just <space> used as <tab> may be converted to spaces in some contexts, or a settable option of which <separator> from a set of choices should be used. The latter could include '-', ':', ';', '.' or ',' as choices along with <space> and <tab>, with <space> being the default. It would allow the makefile to set the most often used separator as the default for the current format, and the quoted format would allow other characters needed less often as separators to be the first character after the opening double quote.

- Issue History
Date Modified Username Field Change
2010-10-17 17:05 dwheeler New Issue
2010-10-17 17:05 dwheeler Status New => Under Review
2010-10-17 17:05 dwheeler Assigned To => ajosey
2010-10-17 17:05 dwheeler Name => David A. Wheeler
2010-10-17 17:05 dwheeler Section => make
2010-10-17 17:05 dwheeler Page Number => 2908-2929
2010-10-17 17:05 dwheeler Line Number => 95549-95822
2010-10-18 16:37 mirabilos Note Added: 0000579
2010-10-18 17:34 yaneurabeya Note Added: 0000580
2010-10-18 17:41 dwheeler Note Added: 0000581
2010-10-18 19:37 dwheeler Note Added: 0000582
2010-10-19 03:50 dwheeler Note Added: 0000583
2010-10-19 04:27 psmith Note Added: 0000584
2010-10-19 17:12 dwheeler Note Added: 0000586
2010-12-02 16:53 eblake Note Added: 0000629
2011-11-07 05:45 ajosey Relationship added related to 0000330
2011-11-15 17:45 dwheeler Note Added: 0001012
2012-10-05 20:47 dwheeler Note Added: 0001391
2013-10-10 03:49 dwheeler Note Added: 0001870
2013-11-03 21:46 shware_systems Note Added: 0001955


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker