Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0000336 [1003.1(2008)/Issue 7] Shell and Utilities Objection Enhancement Request 2010-10-16 18:13 2022-11-01 14:58
Reporter dwheeler View Status public  
Assigned To ajosey
Priority normal Resolution Accepted As Marked  
Status Applied  
Name David A. Wheeler
Organization
User Reference
Section make
Page Number 2915
Line Number 95801-95808
Interp Status ---
Final Accepted Text Note: 0005988
Summary 0000336: Add recursive variable indirection in makefiles
Description Recursive variable indirection in makefiles, such as $(var1$(var2)), is not officially standardized, but is widely supported and useful. For more, see:
http://article.gmane.org/gmane.comp.standards.posix.austin.general/3119 [^]
http://lists.gnu.org/archive/html/automake/2009-04/msg00040 [^]

Eric Blake reports in the first message that "GNU Automake would really like to standardize double variable indirection". Intriguingly, he reports that "all 'make' implementations that I could get hold of *do* accept this construct correctly; that includes IRIX 6.5, AIX 4.3.3, Tru64 4.0D, Solaris 2.6, HP-UX 10.20, NetBSD, FreeBSD, OpenBSD, and of course GNU".

Perhaps even more tellingly, "Also, we tried an alternative implementation
that uses only constructs specified in POSIX, but that turned out to be
*a* *lot* *less* portable in practice."

This construct is useful in the same way that associative arrays are useful in many other languages (e.g., allowing for easy selection between sets of values, such as selecting different file sets for different platforms). It is also very easy to implement, if there is a make implementation that doesn't already support it. Since it *appears* that this construct is *more* portable than those constructs already in POSIX, this construct should be added.

[This part of a set of comments aimed at trying to increase the scope of functionality in *standard* makefiles. I believe the current standard lacks important functionality.]
Desired Action Lines 95801-95802 and 95807-95808 both contain:
"If string1 in a macro expansion contains a macro
expansion, the results are unspecified."
In both cases, replace them with:
"If string1 in a macro expansion contains a macro
expansion, that inner macro expansion is recursively expanded before use."
Tags issue8
Attached Files

- Relationships

-  Notes
(0000630)
nick (manager)
2010-12-02 17:02
edited on: 2022-10-06 15:50

OLD Interpretation response
------------------------
The standard clearly states that the results are unspecified, and conforming applications must conform to this.

Rationale:
-------------
All known implementations have implemented the proposed action, and therefore this interpretation is being considered for TC1.

Notes to the Editor (not part of this interpretation):
-------------------------------------------------------
See desired action. If this interpretation is approved without objection, the editor is encouraged to include this action in the next published Technical Corrigendum.

(0000634)
eblake (manager)
2010-12-09 17:01

During the 9 Dec 2010 meeting, the following points were raised:

HP-UX make has a nesting limit of 100, IRIX make has a nesting limit of 1 (that is, IRIX did not support two levels of recursion). IRIX is no longer supported by SGI, so that does not influence the standard, but wording to this effect is needed:

In a macro definition, behavior is unspecified if string2 recursively refers the expansion of string1, or if string2 has more than <X> levels of nested macros requiring expansion.

where <X> needs some rationale and needs to be somewhere between 1 and 100.

Also to be explored is whether all implementations support ${A} and $(A) synonymously (previous testing only focused on nesting via $(A)); and whether nesting via subst1 and subst2 in ${A:subst1=subst2} is universally supported.
(0000671)
dwheeler (reporter)
2011-02-08 20:06

During the 9 Dec 2010 meeting, it was noted that HP-UX has a nesting limit of 100. No other nesting limits have been identified to date in widely-used implementations (there may be one, but I don't know of one, and no one else has identified any other than the now-abandoned IRIX). Thus, I propose adding this text to deal with limited levels of nesting, with a limit based on the largest limit of a currently-supported implementation:

"In a macro definition, behavior is unspecified if string2 recursively refers the expansion of string1, or if string2 has more than 100 levels of nested macros requiring expansion."

The previous note asked for some additional information. It said, "Also to be explored is whether all implementations support ${A} and $(A) synonymously (previous testing only focused on nesting via $(A)); and whether nesting via subst1 and subst2 in ${A:subst1=subst2} is universally supported". I don't know about "all implementations" and "universally supported", but I did test GNU make and FreeBSD's make, and in both cases (1) ${A} is synonymous with $(A), and (2) substitutions work as expected. So I don't think we need to say anything special about this at all. FYI, the test case I used was as follows:
x=y
y=z
oldsuffix=.o
newsuffix=.c
mylist=a.o b.o
whichlist=$(mylist)
all:
    # should print 4 lines of "z":
    @echo ${${x}}
    @echo $($(x))
    @echo ${$(x)}
    @echo $(${x})
    # Check substitution, should print 2 lines of "a.c b.c"
    @echo ${whichlist:${oldsuffix}=${newsuffix}}
    @echo $(whichlist:$(oldsuffix)=$(newsuffix))


Again, this worked on both GNU make and FreeBSD make. That's not all "makes" but it's encouraging.
(0000999)
eblake (manager)
2011-11-01 14:05

An interesting side note: NeXTSTEP 3.3 (now long obsolete) supports nesting
only if the outer and inner macros alternate in bracketing operators (that
is, ${$()} or $(${}) work, but $($()) fails):
https://lists.gnu.org/archive/html/bug-automake/2011-11/msg00000.html [^]
But since NeXTSTEP is no longer actively maintained and is not striving
for POSIX compliance, this should not be used as a reason to block any
proposed solution to this bug report.
(0001021)
dwheeler (reporter)
2011-11-16 18:40

Given the comments above, here is the revised proposal:

Lines 95801-95802 and 95807-95808 both currently contain:
"If string1 in a macro expansion contains a macro
expansion, the results are unspecified."
In both cases, replace them with:
"If string1 in a macro expansion contains a macro
expansion, that inner macro expansion is recursively expanded before use. Systems shall support at least 100 levels of indirection."

Rationale:
This construct is useful in the same way that associative arrays are useful in many other languages (e.g., allowing for easy selection between sets of values, such as selecting different file sets for different platforms). It is also widely implemented.

Regarding the limit of 100, the only system that might strive for future POSIX compliance, and has a limit at all, is HP-UX. Its maximum is 100. There is no particular rationale for a smaller number, and small values could impede portability in complex situations. Therefore, 100 is used here. The NeXTSTEP 3.3 and SGI systems will not meet this new requirement, but both are obsolete and not striving for POSIX compliance anyway.

"100 ought to be enough for anyone." :-).
(0001873)
dwheeler (reporter)
2013-10-10 16:40

Note that this construct can also implement simple portable conditionals for macro/variable values (e.g., see http://gallium.inria.fr/blog/portable-conditionals-in-makefiles/). [^] It can't replace conditionals for all cases, but for some cases it's adequate.

Note that this is in *addition* to its utility in implementing associative arrays, and the fact that most makes *already* do this.

So, that's yet *another* reason to add recursive variable indirection to the standard's specification of make.
(0004698)
Love4Boobies (reporter)
2019-12-22 15:12

Is there an update on this ticket? It has been under review for almost 10 years. Comment #0001021 seems to provide a sensible solution.
(0004699)
shware_systems (reporter)
2019-12-23 01:02

While Note 1021 provides a direction aimed at taking current practice into account, a limit that may vary between platforms, such as 100 or larger, requires additional support to be considered portable, that I see; and no current platform implements additions in any form, that might let an application adjust to a specific limit, so a proper proposal for Issue 8 can be written. Even with such a proposal, other platforms that would need modifying haven't indicated they'd be ok with adopting those changes. Unti this happens this will probably stay in limbo.
(0004700)
Love4Boobies (reporter)
2019-12-23 12:38

It seems to me like leaving what happens past 100 levels of indirection as unspecified behavior is acceptable for two reasons:

1. A mechanism by which makefiles can detect the supported levels of indirection can be added at a later point, if required.

2. No one in the real world actually uses so many indirections. They might use a few. In the end, it looks like a non-problem blocking an actual problem from being solved.
(0004701)
shware_systems (reporter)
2019-12-23 14:23

1. Various mechanisms already exist; this support simply hasn't been added to any of them by any platform. Because it is simple enough, to add it or invent something equally valid, it is not unreasonable to require it.

2. That's no one you know of actually uses that many. I'm not going to preclude use cases that might benefit from larger values, if the environment is capable of supporting them, just because I haven't thought of one yet. I can look up a number of reports where recursive algorithms fail because a compiler author presumed X amount of stack space was sufficient for all applications, and use cases happened where this presumption proved false.
(0004702)
Love4Boobies (reporter)
2019-12-23 15:33

What would such a mechanism look like? An environment variable? It can't be something internal to make because, without adding a preprocessor to it (which is undoubtedly silly and overcomplicated), there is no way the makefile containing something like $($($(...))) could decide to do something else instead.
(0004706)
psmith (developer)
2020-01-06 20:16

Note #1021 is fine and should be adopted. As mentioned it has virtually universal support among existing implementations.

I disagree with both note #4699 and note #4701. First, the mere fact that "this support hasn't been added to any of them by any platform" is sufficient by itself to make any such extra feature ineligible for standardization and I see no point in delaying this capability until such a mechanism is invented and some number of existing implementations support it.

Second, I don't see any point in such a mechanism: even if it were available who would use it, for what purpose, and how?

There are no "test" operators in make, outside of those provided in a recipe. In what situations would someone want to write a recipe that would detect this limit? If you write a makefile with >100 recursive expansions and you run it on a system that doesn't support it, it will fail; how is that different than writing an explicit test? POSIX make doesn't have any way to dynamically decide how deep a recursion should go so there's no way to "work around" this limit in a compliant makefile.
(0005988)
geoffclare (manager)
2022-10-06 15:20

On D2.1 page 2943 line 98700 section make, change:
Macro expansions in string2 of macro definition lines shall be performed according to the form used to define the macro. Immediate-expansion forms (including appending to an existing immediate-expansion macro) shall be expanded in the macro definition line, and shall not be re-expanded when used in a rule or command. Delayed-expansion forms (including appending to an existing delayed-expansion macro, and conditional assignment to a macro not previously existing) shall be expanded when the macro identified by string1 is expanded in a rule or command, and the result of the expansion shall be scanned for further macros.
to:
Macro expansions in string2 of macro definition lines shall be performed according to the form of macro definition used. In immediate-expansion forms (including appending to an existing immediate-expansion macro), they shall be expanded in the macro definition line and the result of the expansion shall not be scanned for further macros when the macro identified by string1 is expanded. In delayed-expansion forms (including appending to an existing delayed-expansion macro, and conditional assignment to a macro not previously existing), they shall not be expanded in the macro definition line; they shall be expanded when the macro identified by string1 is expanded, and the result of the expansion shall be scanned for further macros. Implementations shall support at least 100 levels of indirection.

On D2.1 page 2943 line 98720, 98726 section make, change:
If string1 in a macro expansion contains a macro expansion, the results are unspecified.
to:
If string1 in a macro expansion contains a macro expansion, that inner macro expansion shall be performed as described above and the result substituted into string1 to produce the macro name used for the outer macro expansion.

- Issue History
Date Modified Username Field Change
2010-10-16 18:13 dwheeler New Issue
2010-10-16 18:13 dwheeler Status New => Under Review
2010-10-16 18:13 dwheeler Assigned To => ajosey
2010-10-16 18:13 dwheeler Name => David A. Wheeler
2010-10-16 18:13 dwheeler Section => make
2010-10-16 18:13 dwheeler Page Number => 2915
2010-10-16 18:13 dwheeler Line Number => 95801-95808
2010-12-02 17:02 nick Note Added: 0000630
2010-12-02 17:11 nick Interp Status => Pending
2010-12-02 17:11 nick Final Accepted Text => See Note: 0000630
2010-12-02 17:11 nick Status Under Review => Interpretation Required
2010-12-02 17:11 nick Resolution Open => Accepted
2010-12-09 17:01 eblake Note Added: 0000634
2010-12-09 17:03 eblake Interp Status Pending => ---
2010-12-09 17:03 eblake Status Interpretation Required => Under Review
2010-12-09 17:03 eblake Resolution Accepted => Reopened
2010-12-09 17:03 eblake Final Accepted Text See Note: 0000630 =>
2011-02-08 20:06 dwheeler Note Added: 0000671
2011-11-01 14:05 eblake Note Added: 0000999
2011-11-16 18:40 dwheeler Note Added: 0001021
2013-10-10 16:40 dwheeler Note Added: 0001873
2019-12-22 15:12 Love4Boobies Note Added: 0004698
2019-12-23 01:02 shware_systems Note Added: 0004699
2019-12-23 12:38 Love4Boobies Note Added: 0004700
2019-12-23 14:23 shware_systems Note Added: 0004701
2019-12-23 15:33 Love4Boobies Note Added: 0004702
2020-01-06 20:16 psmith Note Added: 0004706
2020-01-15 08:13 Love4Boobies Issue Monitored: Love4Boobies
2022-10-06 15:20 geoffclare Note Added: 0005988
2022-10-06 15:21 geoffclare Final Accepted Text => Note: 0005988
2022-10-06 15:21 geoffclare Status Under Review => Resolved
2022-10-06 15:21 geoffclare Resolution Reopened => Accepted As Marked
2022-10-06 15:21 geoffclare Tag Attached: issue8
2022-10-06 15:50 geoffclare Note Edited: 0000630
2022-11-01 14:58 geoffclare Status Resolved => Applied


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker