0001797: strftime "%s" should be able to examine tm_gmtoff

Notes
(0006624) steffen (reporter) 2024-01-16 01:42	Wait, i see what is meant. I delete my note.

(0006651) eblake (manager) 2024-02-12 16:11	Feedback from Paul Eggert via email: On 2024-02-05 08:15, Eric Blake wrote: > Did you consider the effect of the change on applications that > populate struct tm directly (and don't currently set tm_gmtoff, except > perhaps by zeroing the structure)? Yes. Very few apps do that. (I looked for some in the GNU code I help maintain, and found none.) They are greatly outnumbered by the applications that call localtime/localtime_r/mktime/gmtime/gmtime_r/etc. and pass the result to strftime, which is what this bug report is about. > Does the latest tzdata code only use tm_gmtoff in the rare cases when > it is necessary for disambiguation, or is it always used (overriding > the timezone data)? The bug description implies the former, but the > desired action would allow the latter. The former. That is, TZDB 2024a strftime looks only at tm_gmtoff, tm_year, tm_mon, tm_day, tm_hour, tm_min, and tm_sec to determine %s, because that's all you need. The desired action allows either the TZDB behavior, or the glibc behavior which if I recall consults tm_gmtoff only when tm_isdst is ambiguous. The TZDB behavior is technically better than the glibc behavior for three reasons: (1) it removes a multithreading bottleneck, (2) even in a single-threaded platform it's faster because mktime is slower than using tm_gmtoff, and (3) when user code mistakenly calls gmtime and then strftime then %s does what the user expects. The bug report that caused TZDB to behave this way was about (3), but (1) and (2) also play a part.

(0006677) kre (reporter) 2024-02-25 06:50 edited on: 2024-02-25 06:54	I do not like this proposed change - as strftime(%s) is specified in the drafts for Issue 8 is just fine - it and mktime() always produce the same value, given the same (in range) values in the struct tm passed to them. Never anything different. The change that was made to the tzcode implementation to use tm_gmtoff (which despite what was said in Note: 0006651, is always used if it exists - in that implementation what TZ is set to is immaterial to strftime(%s)) was based upon a user hope for what would happen (to get back the time_t that was used to produce a particular struct tm) completely ignoring that no such invocation is required to have occurred. strftime() simply formats as text values from the struct tm which the format string requests - regardless of how those values were inserted. The "very few apps" which "do that" must not be broken by a change like this. Applications which call gmtime() and then strftime(%s) upon the result, and expect the (formatted as a string) value that was used as gmtime()'s arg to be produced are simply broken. (That usage was exactly what led to the "bug" report that was "fixed" by the tzcode change - nonsense.) Simply reject this "bug". I'd also note that while NetBSD has the 2024a version of tzcode (in HEAD anyway) the change that was made to strftime(%s) in that release was reverted, we still do things the old way, using TZ, not using tm_gmtoff.

(0006688) eggert (reporter) 2024-02-26 18:28	Unfortunately draft 4 for Issue 8 has significant problems here, and these problems need to be fixed somehow. The draft does not tell users that strftime %s might inspect tm_gmtoff - something that implementations other than TZDB already do - and it is not entirely clear what the draft means by values "outside the normal range" (line 69773 in draft 4) - for example, is tm_day outside the normal range when it equals 30, and tm_mon happens to be 1? We agree that applications that call gmtime and then strftime %s are broken (unless localtime happens to be UTC). The desired action doesn't change that. These apps will still not necessarily generate the output that the app authors wanted. The desired action does need improvement, along the lines of saying that struct tm values are "in range" if they are values that localtime could have set. Expect a revised proposal soon.

(0006689) eblake (manager) 2024-02-26 19:32	Based on discussions through the 26 Feb 2024 meeting with feedback from Paul Eggert, the following wording is the Austin Group's starting point for an interpretations request to be started after Issue 8 is released. Note that we still need to flesh out the interpretation Rationale (currently a placeholder in the paste below). Among others: the fact that the standard required gmtime_r() (but not gmtime()) to set tm_zone to "UTC" is inconsistent with existing practice where some implementations set it to "GMT". We may also need to add more to the Rationale section of gmtime() and/or other functions on why "UTC" is preferred but why "GMT" is permitted for historical compatibility. Interpretation response ------------------------ The standard states that the strftime() %s conversion calculates the number of seconds since the Epoch as described for mktime(), and conforming implementations must conform to this. However, concerns have been raised about this which are being referred to the sponsor. Rationale: ------------- <Place Rationale, if any, here.> Notes to the Editor (not part of this interpretation): ------------------------------------------------------- Using draft 4.0 line numbering: On page 452 lines 15762-15763 section XBD <time.h>, add CX shading to the lines for tm_gmtoff and tm_zone. On page 1211 line 41381 section gmtime() DESCRIPTION, change (already in CX shading): ...where the names in the structure and in the expression correspond. to: ...where the names in the structure and in the expression correspond; additionally, the tm_gmtoff field of the struct tm shall be set to 0, and the tm_zone field shall be set to a pointer to an implementation-defined string set to "UTC" or "GMT", which shall have static storage duration. On page 1211 line 41397 section gmtime() RETURN VALUE, delete: The structure’s tm_zone member shall be set to a pointer to the string "UTC", which shall have static storage duration. On page 1428 line 47958 section mktime(), change: shall calculate the time since the Epoch value using either the offset in effect before the change or the offset in effect after the change. to: shall calculate the time since the Epoch value using either the offset in effect before the change or the offset in effect after the change; mktime( ) may use the value of tm_gmtoff to decide which of these two results is the more appropriate to return. On page 2135 line 69773 section strftime(), change: If any of the specified values are outside the normal range, the characters stored are unspecified. to: If any of the specified values are outside the normal range, as if set by localtime( ), the characters stored are unspecified. On page 2135 line 69777 section strftime(), change: Local timezone information shall be set as though strftime( ) called tzset( ). to: It shall be implementation-defined whether local timezone information is set as though strftime( ) called tzset( ). On page 2136 line 69836 section strftime(), change: Replaced by the number of seconds since the Epoch as a decimal number, calculated as described for mktime( ). [tm_year, tm_mon, tm_mday, tm_hour, tm_min, tm_sec, tm_isdst] to: Replaced by the number of seconds since the Epoch as a decimal number, calculated as described for mktime( ), except that if the tm_gmtoff value does not match the local timezone information, it is implementation-defined whether the number of seconds since the Epoch is calculated using the offset specified by tm_gmtoff instead of the offset derived from the local timezone information. Note that, unlike mktime( ), the tm structure member values passed to strftime( ) need to be within the normal range, as if set by localtime( ), to avoid unspecified output. [tm_year, tm_mon, tm_mday, tm_hour, tm_min, tm_sec, tm_isdst, tm_gmtoff] After page 2140 line 70007 section strftime(), add a new paragraph to APPLICATION USAGE: The %s conversion should not be used with a tm structure populated solely using strptime( ), since strptime( ) does not set the tm_gmtoff field and the behavior of the strftime( ) %s conversion may depend on this value. Additionally, a tm structure populated by gmtime( ) or gmtime_r( ) is generally not suitable for portable use with the %s conversion of strftime( ); although some implementations of strftime may consult the tm_gmtoff field and reproduce the same value that was provided on input to the gmtime family of functions, this conversion is not required across all implementations.

(0006690) eblake (manager) 2024-02-26 20:02	We may also need to tweak wording on page 2310 line 75181 tzset() about the relation between tzset() and strftime(), if our intent is to allow implementations that do not update or consult the local timezone information for strftime() (independently of whether we accept 0001816 about obsoleting the 3 global variables aspect of tzset()

(0006691) geoffclare (manager) 2024-02-29 12:10	The following is an alternative resolution to be considered, which retains the requirement for strftime() %s to give the same result as mktime() (for in-range struct tm field values). It does allow strftime() %s - and mktime() - to make use of tm_gmtoff, but only as a means to choose between two valid results. It also includes fixes for the various other related issues raised during the discussion to far. Interpretation response ------------------------ The standard clearly states that the strftime() %s conversion calculates the number of seconds since the Epoch as described for mktime(), and conforming implementations must conform to this. Rationale: ------------- There is no requirement for applications to set the tm_gmtoff member and therefore implementations cannot rely on it having been set. However, in situations where there are two valid results of the conversion, an implementation may choose to make use of tm_gmtoff to decide between these two values (provided it can do so safely if tm_gmtoff is uninitialized), even though the strftime() %s conversion is not specified as making use of tm_gmtoff, since returning either of the values is allowed by the standard and the method used to choose between them is internal implementation detail that does not affect applications. Notes to the Editor (not part of this interpretation): ------------------------------------------------------- Using draft 4.0 line numbering: On page 452 lines 15762-15763 section XBD <time.h>, add CX shading to the lines for tm_gmtoff and tm_zone. On page 1211 line 41381 section gmtime() DESCRIPTION, change (already in CX shading): ...where the names in the structure and in the expression correspond. to: ...where the names in the structure and in the expression correspond; additionally, the tm_gmtoff field of the struct tm shall be set to 0, and the tm_zone field shall be set to a pointer to an implementation-defined string set to "UTC" or "GMT", which shall have static storage duration. On page 1211 line 41397 section gmtime() RETURN VALUE, delete: The structure’s tm_zone member shall be set to a pointer to the string "UTC", which shall have static storage duration. On page 1428 line 47958 section mktime(), change: shall calculate the time since the Epoch value using either the offset in effect before the change or the offset in effect after the change. to: shall calculate the time since the Epoch value using either the offset in effect before the change or the offset in effect after the change; mktime() may use the value of tm_gmtoff to decide which of these two results is the more appropriate to return, provided it can do so safely if tm_gmtoff is uninitialized. On page 2135 line 69773 section strftime(), change: If any of the specified values are outside the normal range, the characters stored are unspecified. to: If any of the specified values are outside the normal range, as if set by localtime(), the characters stored are unspecified. On page 2135 line 69777 section strftime(), change: Local timezone information shall be set as though strftime() called tzset(). to: It shall be implementation-defined whether local timezone information is set as though strftime() called tzset(). On page 2136 line 69836 section strftime(), change: Replaced by the number of seconds since the Epoch as a decimal number, calculated as described for mktime(). [tm_year, tm_mon, tm_mday, tm_hour, tm_min, tm_sec, tm_isdst] to: Replaced by the number of seconds since the Epoch as a decimal number, calculated as described for mktime(). Note that, unlike mktime(), the tm structure member values used by this conversion, other than tm_gmtoff, need to be within the normal range, as if set by localtime(), to avoid unspecified output; the value of tm_gmtoff can be uninitialized (see [xref to mktime()]). [tm_year, tm_mon, tm_mday, tm_hour, tm_min, tm_sec, tm_isdst, tm_gmtoff] After page 2140 line 70007 section strftime(), add a new paragraph to APPLICATION USAGE: Using a tm structure populated by gmtime() or gmtime_r() with the %s conversion of strftime() should be avoided, as the calculated seconds since the Epoch value will only correspond to the specified broken-down time if the offset from UTC of the local timezone happens to be zero at that time.

(0006692) geoffclare (manager) 2024-02-29 12:15	I would prefer the resolution in Note: 0006691 over the one in Note: 0006689 (i.e. I think POSIX should not change to allow the new TZDB behaviour). I agree with the points kre made in Note: 0006677 and in particular with the statement that we must not break apps which use strftime() %s without setting tm_gmtoff. The counter-argument that there are very few such apps does not hold water, for two reasons: 1. It is only possible to survey apps for which we have access to the source. We cannot know how many closed-source apps would be affected. 2. We should not consider only apps which use %s explicitly. There are also apps which use user-specified or configurable time format strings. Although a lot of these would be passing in a struct tm obtained from localtime() (as with, for example, GNU ls -l --time-style=+%s), passing one that is populated a different way is also a reasonable thing for an app to do. For example, it could store broken-down times in a standard format and want to present them to the user in their preferred format; to do this it could use strptime() followed by strftime(). Another point in favour of disallowing "blind" use of tm_gmtoff is to consider what system implementers will do when faced with the choice of how to handle a change in POSIX to allow it. In order to continue to support apps written for POSIX.1-2017 and earlier (which did not have tm_gmtoff and therefore conforming apps cannot set it directly) and also have the TZDB behaviour for apps written for POSIX.1-2024 and later, they would need to provide two different versions of strftime(), selected according to _POSIX_C_SOURCE or _XOPEN_SOURCE. It seems highly unlikely that any system implementers would bother to do that (it has been done in the past, but only when there was no choice); the new TZDB-like behaviour would therefore be rare and become a portability gotcha.

(0006693) shware_systems (reporter) 2024-02-29 16:19	Additionally, while the text above references how gmtime() should set tm_gmtoff, there is no equivalent text in localtime() for how that field should be set. Do current implementations just set it to zero, leave it as malloc() garbage, compute something if second TZ format is used? Also, it's questionable whether localtime_r() should behave as if tzset() is called, as the tm struct provided may have value set and unmodified that may conflict with the values in the TZ variable.

(0006698) geoffclare (manager) 2024-03-01 10:09	> Additionally, while the text above references how gmtime() should set tm_gmtoff, there is no equivalent text in localtime() for how that field should be set. Good point. My take on this observation is that the addition in gmtime() is unnecessary. Both gmtime() and localtime() are required to set all of the fields in struct tm, and the description of tm_gmtoff in <time.h> ("Seconds east of UTC") is sufficient to require what value it is set to: in the case of gmtime(), the number of seconds east of UTC is zero because gmtime() returns the broken-down time as UTC. The reason an explicit statement is needed about tm_zone for gmtime() is because its description in <time.h> ("Timezone abbreviation") is not sufficiently clear what string this should be for gmtime().

(0006700) eggert (reporter) 2024-03-02 09:02	> 1. It is only possible to survey apps for which we have access to the source. We cannot know how many closed-source apps would be affected. Although that’s true, we can use our best judgment about which programs will benefit. We know that there are programs that benefit from the resolution in Note: 0006689, because we have a bug report to that effect and it’s plausible there are more programs like that. Although these programs may not follow the strict reading of the standard, they’re plausible uses of the standard functions. In contrast, we don’t know of programs that would benefit from the resolution in Note: 0006691, and these programs are less plausible. > 2.... There are also apps which use user-specified or configurable time format strings. Although a lot of these would be passing in a struct tm obtained from localtime() (as with, for example, GNU ls -l --time-style=+%s), passing one that is populated a different way is also a reasonable thing for an app to do. Unfortunately it’s not reasonable. These apps are broken regardless of which resolution is taken, because the issue comes up with formats other than %s. Consider the following program, which simulates a user-specified format string containing "%Z" used to format a struct tm set by hand or by strptime: #include <stdio.h> #include <time.h> #include <string.h> int main (void) { char buf[1000]; struct tm tm; /* Make TM "uninitialized". */ memset (&tm, 0x81, sizeof tm); tm.tm_isdst = 0; strftime (buf, sizeof buf, "%Z", &tm); puts (buf); } This program dumps core on GNU/Linux, on FreeBSD, and I expect on pretty much any other implementation that supports TZDB-style struct tm, because it uses a bad pointer in tm.tm_zone. And yet the program conforms to POSIX.1-2017, which says that the call to strftime sets the buffer to some string. In other words, programmers can’t trust user-specified strftime formats on anything other than completely filled-in struct tm values, and that’s true regardless of what we do about %s. I suppose we could say that GNU/Linux, FreeBSD, and pretty much everyone else is broken. But it’s better to be realistic and tell users that when using strftime’s %z, %Z and %s formats, tm_zone and tm_gmtoff must be set to in-range values. Insisting on a pedantic reading based on older POSIX would not serve users well because it would give users the wrong impression of what code will actually work. > consider what system implementers will do We implementers will figure it out. It’s the user code that matters here anyway.

(0006701) kre (reporter) 2024-03-03 01:20	Re Note: 0006700 We know that there are programs that benefit from the resolution in Note: 0006689, because we have a bug report to that effect Incorrect bug reports should not be used to justify changes to anything (including tzcode). The code in that report was simply wrong, even with the wording in Note: 0006689 it would still be broken. Attempting to make broken code work is absurd. If I sent in a bug report claiming that my code, with a divide by 0, wasn't working as I expected it to, would you go fix the hardware/OS/something to generate the answer I demanded? In contrast, we don’t know of programs that would benefit from the resolution in Note: 0006691, and these programs are less plausible. Here's one, with all of the error handling, #includes, etc, removed to save space here (they exist in the real thing, naturally): main(int argc, char *argv) { struct tm tm; char p; int i; char buf[64]; p = strptime(argv[1], "%Y-%m-%d %H:%M:%S", &tm); tm.tm_isdst = -1; for (i = 2; i < argc; i++) { setenv("TZ", argv[i], 1); tzset(); strftime(buf, sizeof buf, "%s", &tm); printf("%s %s\n", buf, argv[i]); } exit(0); } It can be run like prog 2024-03-03T07:23:46 UTC Asia/Bangkok Australia/Melbourne \ America/New_York Europe/Berlin Africa/Cairo Africa/Windhoek \ America/St_Johns America/Buenos_Aires America/Anchorage \| sort -n to discover the east->west order of a set of random zones -- which in this case gives: 1709411026 Australia/Melbourne 1709425426 Asia/Bangkok 1709443426 Africa/Cairo 1709443426 Africa/Windhoek 1709447026 Europe/Berlin 1709450626 UTC 1709461426 America/Buenos_Aires 1709463226 America/St_Johns 1709468626 America/New_York 1709483026 America/Anchorage or it can be used to observe differences in summer time behaviour when used like: prog 2024-03-03T07:23:46 Australia/Darwin Australia/Adelaide \ Australia/Brisbane Australia/Sydney \| sort -n 1709411026 Australia/Sydney 1709412826 Australia/Adelaide 1709414626 Australia/Brisbane 1709416426 Australia/Darwin prog 2024-06-03T07:23:46 Australia/Darwin Australia/Adelaide \ Australia/Brisbane Australia/Sydney \| sort -n 1717363426 Australia/Brisbane 1717363426 Australia/Sydney 1717365226 Australia/Darwin 1717365226 Australia/Adelaide from which one observes that when summer time is not in effect, Brisbane and Sydney are the same, as are Darwin and Adelaide - with the former pair of zones achieving a wallclock time before the latter, but when summer time is in effect, all 4 are different, and Adelaide precedes Brisbane, rather than following it. There's no compelling reason to break this. I'm not sure I even really support allowing tm_gmtoff to be used (ever) to handle the very weird cases where nothing else is available to disambiguate the very odd cases, like the America/Caracas example given in the description, though if there's any good, and safe, way an implementation can truly: provided it can do so safely if tm_gmtoff is uninitialized. then it really all that important, provided the code given in this note always works as it should (where tm_gmtoff is entirely nonsense, one could init it to ~0 if desired - or leave it to generate a trap if referenced on hardware that supports that kind of thing). Also, I don't think that this: Note that, unlike mktime(), the tm structure member values used by this conversion, other than tm_gmtoff, need to be within the normal range, as if set by localtime(), to avoid unspecified output; is needed either. The reason that strftime() needs in range values in general, is not because of %s (which did not exist when that requirement was added) but to avoid problems with conversions like %a and %b if tm_wday or tm_mon are set to way out of range values (what is the name of the 29th day of the week?). But for that, all that is needed is that the values are within the ranges specified by <time.h> - there's no issue at all with tm.tm_mon = 1; /* Feb */ tm.tm_mday = 31; and then using strftime("%d %b"); The result might not make sense, but it is perfectly well defined. For %s we don't really need the "within range ..." restriction at all, because of the "calcluated as described by mktime()" - but nor do they do any harm really, as the only reason mktime() permits out of range values is to allow time arithmetic (which given the POSIX definition of time_t isn't really required by POSIX apps) and one thing I can't really imagine is anyone using strftime("%s") for that purpose. But there's no reason to make the ranges required for the %s conversion any tighter than for any other conversion that strftime() offers, (and in any case mktime() is a very poor, in fact, unworkable, interface for that feature, it works OK with gmtime()/timegm(), but it would actually be far better to create (not here, but by the implementations) a companion to difftime() (perhaps: time_t addtime(time_t, double); or something like that) to allow values to be added/subtracted from a time_t. (Then if there are implementations where time_t is not a simple integer count of seconds, they can do whatever is needed to make that work). The wording in Note: 0006691 looks OK to me (though I'd prefer the two changes just mentioned - that is, not to make the two changes referred to just above in this note).

(0006706) geoffclare (manager) 2024-03-07 10:44	>> consider what system implementers will do > We implementers will figure it out. It’s the user code that matters here anyway. I think you missed my point. I expect almost all system implementers will "figure it out" by taking the easy option and sticking with Issue 7 compatible behaviour. (If they have TZDB as the upstream for this stuff, they will revert the strftime() tm_gmtoff change, as NetBSD has done.) The existence of any system where the strftime() implementation behaves like TZDB then creates a portability problem for user code written on any of the other systems; it will work fine on most systems but misbehave if/when it is ever ported to this unusual and rare system.

(0006707) geoffclare (manager) 2024-03-07 11:04	> This program dumps core on GNU/Linux, on FreeBSD, and I expect on pretty much any other implementation that supports TZDB-style struct tm, because it uses a bad pointer in tm.tm_zone. And yet the program conforms to POSIX.1-2017, which says that the call to strftime sets the buffer to some string. The program conforms to POSIX.1-2017, and your results show that GNU/Linux and FreeBSD apparently do not; these implementations must be using tm_zone "blindly" and do not conform because POSIX.1-2017 requires that the only struct tm field used by %Z is tm_isdst. (This doesn't mean implementations can't use non-standard fields such as tm_zone, but if they are used, it must be in such a way that a conforming application can't tell the difference, i.e. it is just internal implementation detail. In the case of tm_zone this would mean checking the pointer is equal to a known valid value for the current timezone before using it.)

(0006718) geoffclare (manager) 2024-03-11 09:55	> I suppose we could say that GNU/Linux, FreeBSD, and pretty much everyone else is broken. But it’s better to be realistic and tell users that when using strftime’s %z, %Z and %s formats, tm_zone and tm_gmtoff must be set to in-range values. For %z and %Z, requiring applications to set tm_zone and tm_gmtoff would create a conflict with the C standard. C17 says that the only member used by %z and %Z is tm_isdst. Of course, if an application explicitly sets tm_zone or tm_gmtoff then it is using an extension to C17 and this could alter the behaviour, and if the structure was populated by localtime() et al then use of tm_zone or tm_gmtoff can be an internal implementation detail, as per my previous note. However, the crucial case is when the application populates the structure in a way that does not set tm_zone or tm_gmtoff; for this case C17 requires that the conversion is done based on tm_isdst.

(0006793) eggert (reporter) 2024-05-23 08:04	Re Note: 0006701 > The code in that report was simply wrong, even > with the wording in Note: 0006689 it would still be broken. That code is unportable, not “wrong” or “broken”. The wording in Note: 0006689 would let implementations do what the code’s author intended. For this use case, that’s better than requiring implementations to not follow the intent. I continue to be skeptical that typical developers would expect strftime(buf, size, "%F %T %Z", gmtime(&t)) to do anything other than output a string representing UTC. > In contrast, we don’t know of programs that would benefit from the > resolution in Note: 0006691, and these programs are less plausible. > > Here's one, with all of the error handling, #includes, etc, removed to > save space here (they exist in the real thing, naturally): That example doesn’t use gmtime, so it is not completely on point. I accept that one can contrive programs that prefer the other interpretation. However, I continue to maintain that the vast majority of programs assume the interpretations used by TZDB for %z and %Z. To check this, I did a GitHub code search today with the query “strftime gmtime path:*.c”. Most matches were irrelevant, because the output of gmtime was not passed to strftime, or because the strftime format string did not use %s, %z, or %Z. In a few matches it was hard to see what’s going on, so I ignored them. However, all the relevant matches I found (listed below) preferred the proposed interpretation: that is, they expected strftime to format the gmtime output as UTC, not as local time. In many cases this was because they let the user specify an arbitrary strftime format to be applied to the result of gmtime, and users would naturally expect gmtime to output GMT aka UTC. Here are the matches I found: https://github.com/davidmoreno/onion/blob/de8ea938342b36c28024fd8393ebc27b8442a161/src/onion/shortcuts.c#L350 [^] https://github.com/erlang/otp/blob/945c940f6bc6c0bcb026cdc6ae8f3ce358e859bb/erts/etc/unix/run_erl.c#L583 [^] https://github.com/esnet/iperf/blob/93c60bf2ea8aa52e98399223ec04a95f65f00f7b/src/iperf_api.c#L948 [^] https://github.com/jonas/tig/blob/587a59abab3f2ebe7612bc69f44c22df570b2ee3/src/util.c#L208 [^] https://github.com/openwall/john/blob/9f913a734055aae2606c68851a5a16b0d50a2060/src/logger.c#L324 [^] https://github.com/RediSearch/RediSearch/blob/10261d2f93e77a3d1118ccded01fb4fa0e1c9cca/src/aggregate/functions/date.c#L36 [^] https://github.com/RMerl/asuswrt-merlin.ng/blob/466f5aee9e274971480a19497573c4264f865019/release/src/router/aws-iot/src/time_util.c#L56 [^] https://github.com/RMerl/asuswrt-merlin.ng/blob/466f5aee9e274971480a19497573c4264f865019/release/src/router/rc/conn_diag_log.c#L334 [^] https://github.com/root-project/root/blob/333f042a16525e746da4ec56c3a5a3ed79ceb677/tutorials/graphs/timeonaxis3.C#L25 [^] https://github.com/cheat-engine/cheat-engine/blob/54f90a22bfc9d4981f414a7917d70d3d17f190e5/lua/src/loslib.c#L129 [^] > The reason that strftime() needs in range values in > general, is not because of %s (which did not exist when that requirement > was added) but to avoid problems with conversions like %a and %b It’s true that the in-range requirement was not put in because of %s, because (as you mention) %s was not in the standard back then. However, the in-range requirement is needed for %s regardless of the resolution of this issue. For example, on a host with 32-bit time_t and 32-bit int, a tm_year value of INT_MAX is out of range for localtime/gmtime, and would cause strftime to overflow when calculating %s. So we cannot wave away the in-range requirement - it’s essential for multiple reasons. > The existence of any system where the strftime() implementation behaves like TZDB then creates a portability problem Users will have portability problems no matter what. We already have discrepancies between how strftime behaves for %s/%z/%Z on FreeBSD, z/OS, TZDB, AIX, etc. Our job is to document discrepancies and (if we can get consensus) to suggest a preferred behavior. > The program conforms to POSIX.1-2017, and your results show that GNU/Linux and FreeBSD apparently do not It’s not just just GNU/Linux and FreeBSD; it’s also macOS and I expect every other other system that has tm_zone. It would be counterproductive to say that all these systems are wrong and that POSIX is right. This is an area where if POSIX disagrees with common practice, then POSIX needs to be fixed. > For %z and %Z, requiring applications to set tm_zone and tm_gmtoff would create a conflict with the C standard. C17 says that the only member used by %z and %Z is tm_isdst. There’s no conflict here. C17 does not specify the method used to deduce the time zone used for %z and %Z. An implementation is free to use whatever screwy method it likes. It can even indulge in undefined behavior, since C17 does not define the method (that’s what “undefined behavior” means, after all...). C17’s language cannot mean that %z and %Z’s output must be completely determined by tm_isdst and the locale. If that were true, it would prohibit even POSIX.1-2017’s requirement that global state set by tzset() - which is not part of the locale - can also play a role. Luckily it is not true, as C17 gives wide latitude as to how the implementation can determine the timezone. POSIX has already exploited that latitude, and tm_zone and tm_gmtoff are yet another way to exploit that latitude. While we’re on the topic of %Z, I noticed one other thing: IBM z/OS uses the proposed interpretation for %Z. That is, the IBM documentation <https://www.ibm.com/docs/en/zos/3.1.0?topic=functions-strftime-convert-formatted-time> [^] states that strftime generates an abbreviation appropriate for UTC (not for localtime) if the struct tm was produced by gmtime. TZDB, GNU/Linux, FreeBSD, macOS and z/OS all disagree one way or another with the idea that strftime should not look at tm_zone or tm_gmtoff when computing %s, %z, and %Z. POSIX should not try to impose a requirement that would invalidate all these systems. There’s one other potential issue with requiring strftime to do the equivalent of inspecting the tzname and timezone global variables: these variables do not contain well-defined value when TZ is set to something typical like TZ='Europe/London', due to the resolution of Issue 0001816 “daylight, timezone, tzname do not work with location-based TZ”. This suggests that %s, %z, and %Z could expand to anything if one uses a location-based TZ. Although POSIX.1-2017 allows that behavior, the intent of the next POSIX, as I understand it, is to nail this down more precisely. In my next comment I’ll try to reword the latest proposed resolution to address these issues.

(0006794) eggert (reporter) 2024-05-23 08:07	Although the resolution proposed in Note: 0006691 has a lot to like, we need to do better with %s, %z, and %Z, as discussed in my previous comment. One thing that’s not clearly nailed down is that the implementation should be consistent in this area. That is, whatever it does for %s should be consistent with what it does for %z, and likewise for %Z. This is a “should” not a “must” because not every implementation is consistent, but intuitively it makes sense that the three formats should expand to consistent output. Interpretation response ------------------------ The standard states that when the input is in the normal range the strftime() %s conversion calculates the number of seconds since the Epoch as described for mktime(), and conforming implementations must conform to this. However, concerns have been raised about this which are being referred to the sponsor. Rationale: ------------- In previous editions of POSIX, struct tm lacked tm_gmtoff and tm_zone members, so strftime needed to inspect global state to process %z and %Z. The current version allows strftime to instead inspect tm_gmtoff for %z, and tm_zone for %Z, as is common practice. For consistency it also allows strftime to inspect tm_gmtoff to process %s, a newly-required format. A strftime that inspects tm_gmtoff and tm_zone works better with applications that use geographical timezones, or that fill in struct tm members by calling gmtime or equivalent. Conversely, a strftime that inspects global state is more compatible with previous editions of POSIX when programs use the second TZ format in XBD Section 8.3 and fill in struct tm members individually without calling localtime or equivalent. A strftime implementation can do both: that is, it can inspect global state when the second TZ format is used, and use tm_gmtoff and tm_zone when a geographical timezone is used. This edition of the standard does not say whether the abovementioned global state consists of the contents of the variables tzname and timezone, or some other part of the system state that user code cannot modify directly. Modifying tzname and timezone therefore might (but might not) affect the global state that in turn might affect strftime’s behavior with %s, %z, and %Z, or might not if strftime inspects tm_gmtoff and tm_zone instead. Regardless of implementation method, implementations should process the conversion specifications %s, %z, and %Z consistently. For example, if strftime uses tm_gmtoff to process %z, it should also use tm_gmtoff to process %s and should use tm_zone to process %Z. Conversely, if strftime uses global state to determine one of the the three conversion specifications, it should use the same global state to determine the other two. Notes to the Editor (not part of this interpretation): ------------------------------------------------------- Using draft 4.0 line numbering: On page 452 lines 15762-15763 section XBD <time.h>, add CX shading to the lines for tm_gmtoff and tm_zone. On page 1211 line 41381 section gmtime() DESCRIPTION, change (already in CX shading): ...where the names in the structure and in the expression correspond. to: ...where the names in the structure and in the expression correspond; additionally, the tm_gmtoff field of the struct tm shall be set to 0, and the tm_zone field shall be set to a pointer to an implementation-defined string set to "UTC" or "GMT", which shall have static storage duration. On page 1211 line 41397 section gmtime() RETURN VALUE, delete: The structure’s tm_zone member shall be set to a pointer to the string "UTC", which shall have static storage duration. On page 1428 line 47958 section mktime(), change: shall calculate the time since the Epoch value using either the offset in effect before the change or the offset in effect after the change. to: shall calculate the time since the Epoch value using either the offset in effect before the change or the offset in effect after the change; mktime() may use the value of tm_gmtoff to decide which of these two results is the more appropriate to return, provided it can do so safely if tm_gmtoff is uninitialized. On page 2135 line 69773 section strftime(), change: If any of the specified values are outside the normal range, the characters stored are unspecified. to: If any of the specified values are outside the normal range, as if set by localtime(), the characters stored are unspecified. On page 2135 line 69777 section strftime(), change: Local timezone information shall be set as though strftime() called tzset(). to: It shall be implementation-defined whether local timezone information is set as though strftime() called tzset(). On page 2136 line 69836 section strftime(), change: Replaced by the number of seconds since the Epoch as a decimal number, calculated as described for mktime(). [tm_year, tm_mon, tm_mday, tm_hour, tm_min, tm_sec, tm_isdst] to: Replaced by the number of seconds since the Epoch as a decimal number, calculated as described for mktime(). Note that, unlike mktime(), the tm structure member values used by this conversion need to be within the normal range, as if set by localtime(), to avoid unspecified output. [tm_year, tm_mon, tm_mday, tm_hour, tm_min, tm_sec, tm_isdst, tm_gmtoff] After page 2140 line 70007 section strftime(), add a new paragraph to APPLICATION USAGE: Using a tm structure populated by gmtime() or gmtime_r() with the %s, %z, and %Z conversion of strftime() is not portable unless local time is UTC, as the implementation might process the conversions either by inspecting the structure’s tm_gmtoff and tm_zone members, or by inspecting global state.

(0006795) geoffclare (manager) 2024-05-23 10:47	>> For %z and %Z, requiring applications to set tm_zone and tm_gmtoff would create a conflict with the C standard. C17 says that the only member used by %z and %Z is tm_isdst. >There’s no conflict here. C17 does not specify the method used to deduce the time zone used for %z and %Z. An implementation is free to use whatever screwy method it likes. Only the C committee can give a definitive answer on what C17 requires here. We cannot finalise new wording for %z and %Z until we liaise with them. Therefore our next step on this bug should be to prepare a paper for Nick to submit for consideration by the C committee.

(0006797) kre (reporter) 2024-05-23 16:57	Re Note: 0006793 and Note: 0006794 I really believe that those notes show a lack of understanding of the issue here. The following comment in Note: 0006793 demonstrates that: > Here's one, with all of the error handling, #includes, etc, removed to > save space here (they exist in the real thing, naturally): That example doesn’t use gmtime, so it is not completely on point. But it is exactly on point. How is the implementation of strftime() intended to learn whether or not its caller used gmtime(), or localtime() or neither of those. Let me give that example again, with one more minor addition, to make the underlying difficulty more obvious: main(int argc, char *argv) { time_t now; struct tm tm; char p; int i; char buf[64]; (void)time(&now); /* or fill in "now" however you like / tm = gmtime(&now); /* do something with the tm, doesn't matter what */ p = strptime(argv[1], "%Y-%m-%d %H:%M:%S", &tm); tm.tm_isdst = -1; for (i = 2; i < argc; i++) { setenv("TZ", argv[i], 1); tzset(); strftime(buf, sizeof buf, "%s", &tm); printf("%s %s\n", buf, argv[i]); } exit(0); } nb: same caveat about missing error handling, #includes, etc, as when the earlier version of this was in Note: 0006701 Now the program has called gmtime() - that call just happens to be 100% irrelevant to the calls of strftime() (using a struct tm which had the results from gmtime() copied into it, some time in the past). How does your implementation propose to generate sensible results here. Even without the call to gmtime() there, how does your implementation propose to tell whether there was one or not? It doesn't matter that this might be an unusual thing to do, the fields that the strptime() call filled in (along with tm_isdst which is set separately) are all that is required for mktime() to work on the data, and I think it was previously agreed that strftime(%s) and mktime() should produce the same result (one as a string, the other as a time_t but that's immaterial). That's certainly how it has always been specified. Applications are entitled to operate that way. Similarly for %z and %Z - both C and POSIX have promised application writers that all they need do is set tm_isdst, in an otherwise completely randomly filled (or unfilled) struct tm, and results will be produced. Certainly the standards do not, and should not, tell the implementers how to generate the results, but those implementers need to make things work when the application does the bare minimum required of it by the standards, regardless of how many other applications do things differently. We simply cannot ignore what has been promised before, and change the state of the world, requiring applications to update from what has been promised as being acceptable, and which has previously worked in conforming applications. There is simply no way to fix this with the current API. You'd need entirely new functions, which no existing application can be calling (obviously) with a different set of rules. Re Note: 0006795 -- I don't think simply passing this off to the C committee is really the right thing to do. Certainly, we can let them be the "bad guys" so we can throw up our hands and just say "we need to conform to the C standards" and make it look as if it isn't our problem. The right thing to do here is simply to reject this bug, and not waste everyone's time on it any more than it already has.

(0006798) eggert (reporter) 2024-05-24 18:59	Re Note 0006797: > That example doesn’t use gmtime, so it is not completely on point. > > But it is exactly on point. I wrote “on point” because I thought the example was a program in the wild. But it now appears that it was a test program written to address this issue. Although I’ve cited several real-world programs using gmtime + strftime + %s/%z/%Z, we haven’t seen a similar set of programs using strptime + tm_isdst=-1 + strftime + %s/%z/%Z, so the real-world programs we’ve seen suggest that strftime should consult tm_gmtoff+tm_zone rather than global state. > How is the implementation of strftime() intended to learn whether or > not its caller used gmtime(), or localtime() or neither of those. Note 0006794 suggests that strftime use tm_gmtoff (for %s and %z) and tm_zone (for %Z). > It doesn't matter that this might be an unusual thing to do Unfortunately it does matter, because so many real-world implementations and programs disagree with a reading of the standard that requires strftime to ignore tm_gmtoff/tm_zone when processing %s/%z/%Z. When the standard isn’t clear on a matter it’s better to clear it up; and when clearing it up, it’s better to minimize disruption to common practice; and to do that, we need to know what common practice is. Although a test app that we write might be useful for other things, it does not help us discern common practice in apps. > it was previously agreed that strftime(%s) and mktime() > should produce the same result Yes, when the struct tm values are inside the normal range (that is, when they could have been produced by localtime). But we are talking about what strftime should do with struct tm values outside the normal range. > Similarly for %z and %Z - both C and POSIX have promised application > writers that all they need do is set tm_isdst, in an otherwise completely > randomly filled (or unfilled) struct tm, and results will be produced. I don’t see that written down. The C standard says that strftime can consult tm_isdst for %Z. It does not say that the output of %Z is completely determined by tm_isdst’s value. Implementations can also look elsewhere when generating %Z output, and they can impose additional requirements on the “elsewhere”. This is true even for POSIX.1-2017, which says that if TZNAME_MAX is 6 and you set TZ='ABCDEFG0' in the environment, strftime with %Z is allowed to dump core because the time zone abbreviation is too long. This provision of POSIX.1-2017 does not violate the C standard, even though it is an additional requirement above and beyond strftime’s caller setting tm_isdst. And requiring tm_zone to have sensible contents is not a violation either: tm_zone is an extension to the C standard and the C standard has nothing to say about what tm_zone’s contents can or should be.

(0006799) kre (reporter) 2024-05-24 21:32	Re Note: 0006798 But we are talking about what strftime should do with struct tm values outside the normal range. No, we aren't. strftime() is clear the results are unspecified if the relevant fields aren't within the normal range(s). You can do what you like in that case (but only the relevant fields, when converting %a for example, it isirrelevant what tn_mday or tm_mon or ... are set to, just tm_wday). The same goes for all of the conversions. The C standard says that strftime can consult tm_isdst for %Z. I don't have that one in front of me, but I doubt it says anything much different than posix says, which is: The appropriate characters are determined using the LC_TIME category of the current locale and by the values of zero or more members of the broken-down time structure pointed to by timeptr, as specified in brackets in the description. What that means is that the specified fields of the struct tm (the "broken-down time structure") are all that the application needs to ensure are set in order to perform a specific conversion. While you're certainly correct that the implementation can look at other data, that other data cannot come from the struct, as there's no expectation that any of that will have been set by the application, and there's no field in the struct (which would need to be some kind of checksum to detect changes to other fields made by the application) by which it is possible to determine that other fields have consistent values with the fields of interest to the conversion. But yes, you can certainly use what is in TZ, or anything obtained from its use, and anything you can get from the LC_TIME locale. And while: tm_zone is an extension to the C standard and the C standard has nothing to say about what tm_zone’s contents can or should be. is certainly true, this conclusion: And requiring tm_zone to have sensible contents is not a violation either: is incorrect. A conforming C application cannot set tm_zone as no such thing exists. Hence an implementation which is supposed to support conforming C applications cannot use data from that field, as it has no way to know if anything is there or not (or anything relevant for sure). The implementation is certainly allowed to set that field (as in localtime() for example), a conforming application will never look there, and so will never care if it is set or not. POSIX is intended to support conforming C applications. And finally: When the standard isn’t clear on a matter it’s better to clear it up; That's right, but in this case it is clear. Always has been (even if the wording was a bit bizarre in some places). Furthermore, since this affects C programs, it should be fixed, if any fixing is needed, by the C standard first, not by some end-run performed here.

(0006801) eggert (reporter) 2024-05-24 22:56	> A conforming C application cannot set tm_zone as no such thing exists. No, there are many ways a conforming C application can set tm_zone. The several real-world applications I cited in Note: 0006793 all set tm_zone by calling gmtime, and these apps work fine on GNU/Linux, macOS, z/OS, etc. This is common practice. Obviously we disagree about how to interpret POSIX. I find the overly-strict interpretation unconvincing because (a) significant real-world implementations have disagreed with that interpretation for decades, (b) it's easy to find real-world applications that rely on these implementations' behavior, and (c) it's hard to find real-world applications that rely on the overly-strict interpretation. This is not a new issue. What's new is that this lack of clarity or glitch in the standards is being brought to the standardization bodies' attention.

(0006802) kre (reporter) 2024-05-25 17:12	Re Note: 0006801 there are many ways a conforming C application can set tm_zone. There cannot be, as C does not have such a field, any attempt to reference it, in any way, directly or indirectly, would be non-conforming. [...] I cited in Note: 0006793 all set tm_zone by calling gmtime That's not the application setting tm_zone (etc) it is the implementation. That's allowed, as an extension to the standard. And while the application certainly knows it called gmtime() there is no way (assuming it is a conforming C application) that it can deduce from doing so that tm_zone (or tm_gmtoff which is more relevant for %s) was set by that call. Such a program should be able to be compiled and run on a system in which gmtime() does not set those fields, which is perfectly allowable for implementations of the C standard (it will not be for POSIX applications after the new standard appears, at least for implementations claiming conformance). On such a system the (conforming) program should run and produce the same results as on a system that happens to set the new fields. Furthermore, strftime() needs to work correctly when the struct tm passed to it has never seen the results of any previous implementation provided function, and also when it has, but that was in some previous incarnation, whose use has been completed, and the struct has now been repurposed to be used just for strftime, with values entirely unrelated to the one it was previously used for, and only the required values for the format string used assigned (new) values. Unless you can provide me with text in the C or POSIX standard which says that gmtime() localtime() or something equivalent must have generated (or updated, perhaps via mktime()) any struct tm which is to be passed to strftime() then the implementation of strftime() simply must work when that has not happened. "work" means producing the correct result, which then is obviously also required to be the result if struct tm with the same required field values had been produced by localtime() or gmtime() - two different results cannot both be correct (normally anyway). It makes no difference how common it might be for strftime() to be called with a struct tm returned from localtime() or gmtime(). Unless there is a requirement somewhere making that mandatory for applications, then we cannot assume that it has happened. Nor can we require that tm_gmtoff or tm_zone be set by the application, as conforming C applications cannot do that, and strftime() is a C standard interface (it would be different were it one added by POSIX.) It is irrelevant how many non-conforming, extended, or simply buggy, implementations, or applications which depend upon them, you can find. And I'd note, that until earlier this year, your implementation (or ado's or whoever it was who added strftime()), which I suspect is, or is the basis of, most real world implementation (certainly of ones which include the tm_gmtoff and tm_zone fields) was not one of the ones which abused strftime(%s) the way you are now proposing.

Issue History
Date Modified	Username	Field	Change
2024-01-15 23:56	eggert	New Issue
2024-01-15 23:56	eggert	Name	=> Paul Eggert
2024-01-15 23:56	eggert	Organization	=> UCLA Computer Science Dept.
2024-01-15 23:56	eggert	User Reference	=> strftime-%s
2024-01-15 23:56	eggert	Section	=> strftime
2024-01-15 23:56	eggert	Page Number	=> 2136
2024-01-15 23:56	eggert	Line Number	=> 69836-69837
2024-01-16 01:29	steffen	Note Added: 0006623
2024-01-16 01:42	steffen	Note Added: 0006624
2024-01-16 01:42	steffen	Note Deleted: 0006623
2024-02-01 16:42	nick	Relationship added	related to 0001533
2024-02-12 16:11	eblake	Note Added: 0006651
2024-02-25 06:50	kre	Note Added: 0006677
2024-02-25 06:54	kre	Note Edited: 0006677
2024-02-26 18:28	eggert	Note Added: 0006688
2024-02-26 19:23	eblake	Relationship added	related to 0001816
2024-02-26 19:32	eblake	Note Added: 0006689
2024-02-26 19:52	eblake	Relationship added	child of 0001612
2024-02-26 19:55	eblake	Relationship added	child of 0000169
2024-02-26 20:02	eblake	Note Added: 0006690
2024-02-29 12:10	geoffclare	Note Added: 0006691
2024-02-29 12:15	geoffclare	Note Added: 0006692
2024-02-29 16:19	shware_systems	Note Added: 0006693
2024-03-01 10:09	geoffclare	Note Added: 0006698
2024-03-02 09:02	eggert	Note Added: 0006700
2024-03-03 01:20	kre	Note Added: 0006701
2024-03-07 10:44	geoffclare	Note Added: 0006706
2024-03-07 11:04	geoffclare	Note Added: 0006707
2024-03-11 09:55	geoffclare	Note Added: 0006718
2024-05-23 08:04	eggert	Note Added: 0006793
2024-05-23 08:07	eggert	Note Added: 0006794
2024-05-23 10:47	geoffclare	Note Added: 0006795
2024-05-23 16:57	kre	Note Added: 0006797
2024-05-24 18:59	eggert	Note Added: 0006798
2024-05-24 21:32	kre	Note Added: 0006799
2024-05-24 22:56	eggert	Note Added: 0006801
2024-05-25 17:12	kre	Note Added: 0006802

Aardvark Mark IV