0001797: strftime "%s" should be able to examine tm_gmtoff

Notes
(0006624) steffen (reporter) 2024-01-16 01:42	Wait, i see what is meant. I delete my note.

(0006651) eblake (manager) 2024-02-12 16:11	Feedback from Paul Eggert via email: On 2024-02-05 08:15, Eric Blake wrote: > Did you consider the effect of the change on applications that > populate struct tm directly (and don't currently set tm_gmtoff, except > perhaps by zeroing the structure)? Yes. Very few apps do that. (I looked for some in the GNU code I help maintain, and found none.) They are greatly outnumbered by the applications that call localtime/localtime_r/mktime/gmtime/gmtime_r/etc. and pass the result to strftime, which is what this bug report is about. > Does the latest tzdata code only use tm_gmtoff in the rare cases when > it is necessary for disambiguation, or is it always used (overriding > the timezone data)? The bug description implies the former, but the > desired action would allow the latter. The former. That is, TZDB 2024a strftime looks only at tm_gmtoff, tm_year, tm_mon, tm_day, tm_hour, tm_min, and tm_sec to determine %s, because that's all you need. The desired action allows either the TZDB behavior, or the glibc behavior which if I recall consults tm_gmtoff only when tm_isdst is ambiguous. The TZDB behavior is technically better than the glibc behavior for three reasons: (1) it removes a multithreading bottleneck, (2) even in a single-threaded platform it's faster because mktime is slower than using tm_gmtoff, and (3) when user code mistakenly calls gmtime and then strftime then %s does what the user expects. The bug report that caused TZDB to behave this way was about (3), but (1) and (2) also play a part.

(0006677) kre (reporter) 2024-02-25 06:50 edited on: 2024-02-25 06:54	I do not like this proposed change - as strftime(%s) is specified in the drafts for Issue 8 is just fine - it and mktime() always produce the same value, given the same (in range) values in the struct tm passed to them. Never anything different. The change that was made to the tzcode implementation to use tm_gmtoff (which despite what was said in Note: 0006651, is always used if it exists - in that implementation what TZ is set to is immaterial to strftime(%s)) was based upon a user hope for what would happen (to get back the time_t that was used to produce a particular struct tm) completely ignoring that no such invocation is required to have occurred. strftime() simply formats as text values from the struct tm which the format string requests - regardless of how those values were inserted. The "very few apps" which "do that" must not be broken by a change like this. Applications which call gmtime() and then strftime(%s) upon the result, and expect the (formatted as a string) value that was used as gmtime()'s arg to be produced are simply broken. (That usage was exactly what led to the "bug" report that was "fixed" by the tzcode change - nonsense.) Simply reject this "bug". I'd also note that while NetBSD has the 2024a version of tzcode (in HEAD anyway) the change that was made to strftime(%s) in that release was reverted, we still do things the old way, using TZ, not using tm_gmtoff.

(0006688) eggert (reporter) 2024-02-26 18:28	Unfortunately draft 4 for Issue 8 has significant problems here, and these problems need to be fixed somehow. The draft does not tell users that strftime %s might inspect tm_gmtoff - something that implementations other than TZDB already do - and it is not entirely clear what the draft means by values "outside the normal range" (line 69773 in draft 4) - for example, is tm_day outside the normal range when it equals 30, and tm_mon happens to be 1? We agree that applications that call gmtime and then strftime %s are broken (unless localtime happens to be UTC). The desired action doesn't change that. These apps will still not necessarily generate the output that the app authors wanted. The desired action does need improvement, along the lines of saying that struct tm values are "in range" if they are values that localtime could have set. Expect a revised proposal soon.

(0006689) eblake (manager) 2024-02-26 19:32	Based on discussions through the 26 Feb 2024 meeting with feedback from Paul Eggert, the following wording is the Austin Group's starting point for an interpretations request to be started after Issue 8 is released. Note that we still need to flesh out the interpretation Rationale (currently a placeholder in the paste below). Among others: the fact that the standard required gmtime_r() (but not gmtime()) to set tm_zone to "UTC" is inconsistent with existing practice where some implementations set it to "GMT". We may also need to add more to the Rationale section of gmtime() and/or other functions on why "UTC" is preferred but why "GMT" is permitted for historical compatibility. Interpretation response ------------------------ The standard states that the strftime() %s conversion calculates the number of seconds since the Epoch as described for mktime(), and conforming implementations must conform to this. However, concerns have been raised about this which are being referred to the sponsor. Rationale: ------------- <Place Rationale, if any, here.> Notes to the Editor (not part of this interpretation): ------------------------------------------------------- Using draft 4.0 line numbering: On page 452 lines 15762-15763 section XBD <time.h>, add CX shading to the lines for tm_gmtoff and tm_zone. On page 1211 line 41381 section gmtime() DESCRIPTION, change (already in CX shading): ...where the names in the structure and in the expression correspond. to: ...where the names in the structure and in the expression correspond; additionally, the tm_gmtoff field of the struct tm shall be set to 0, and the tm_zone field shall be set to a pointer to an implementation-defined string set to "UTC" or "GMT", which shall have static storage duration. On page 1211 line 41397 section gmtime() RETURN VALUE, delete: The structure’s tm_zone member shall be set to a pointer to the string "UTC", which shall have static storage duration. On page 1428 line 47958 section mktime(), change: shall calculate the time since the Epoch value using either the offset in effect before the change or the offset in effect after the change. to: shall calculate the time since the Epoch value using either the offset in effect before the change or the offset in effect after the change; mktime( ) may use the value of tm_gmtoff to decide which of these two results is the more appropriate to return. On page 2135 line 69773 section strftime(), change: If any of the specified values are outside the normal range, the characters stored are unspecified. to: If any of the specified values are outside the normal range, as if set by localtime( ), the characters stored are unspecified. On page 2135 line 69777 section strftime(), change: Local timezone information shall be set as though strftime( ) called tzset( ). to: It shall be implementation-defined whether local timezone information is set as though strftime( ) called tzset( ). On page 2136 line 69836 section strftime(), change: Replaced by the number of seconds since the Epoch as a decimal number, calculated as described for mktime( ). [tm_year, tm_mon, tm_mday, tm_hour, tm_min, tm_sec, tm_isdst] to: Replaced by the number of seconds since the Epoch as a decimal number, calculated as described for mktime( ), except that if the tm_gmtoff value does not match the local timezone information, it is implementation-defined whether the number of seconds since the Epoch is calculated using the offset specified by tm_gmtoff instead of the offset derived from the local timezone information. Note that, unlike mktime( ), the tm structure member values passed to strftime( ) need to be within the normal range, as if set by localtime( ), to avoid unspecified output. [tm_year, tm_mon, tm_mday, tm_hour, tm_min, tm_sec, tm_isdst, tm_gmtoff] After page 2140 line 70007 section strftime(), add a new paragraph to APPLICATION USAGE: The %s conversion should not be used with a tm structure populated solely using strptime( ), since strptime( ) does not set the tm_gmtoff field and the behavior of the strftime( ) %s conversion may depend on this value. Additionally, a tm structure populated by gmtime( ) or gmtime_r( ) is generally not suitable for portable use with the %s conversion of strftime( ); although some implementations of strftime may consult the tm_gmtoff field and reproduce the same value that was provided on input to the gmtime family of functions, this conversion is not required across all implementations.

(0006690) eblake (manager) 2024-02-26 20:02	We may also need to tweak wording on page 2310 line 75181 tzset() about the relation between tzset() and strftime(), if our intent is to allow implementations that do not update or consult the local timezone information for strftime() (independently of whether we accept 0001816 about obsoleting the 3 global variables aspect of tzset()

(0006691) geoffclare (manager) 2024-02-29 12:10	The following is an alternative resolution to be considered, which retains the requirement for strftime() %s to give the same result as mktime() (for in-range struct tm field values). It does allow strftime() %s - and mktime() - to make use of tm_gmtoff, but only as a means to choose between two valid results. It also includes fixes for the various other related issues raised during the discussion to far. Interpretation response ------------------------ The standard clearly states that the strftime() %s conversion calculates the number of seconds since the Epoch as described for mktime(), and conforming implementations must conform to this. Rationale: ------------- There is no requirement for applications to set the tm_gmtoff member and therefore implementations cannot rely on it having been set. However, in situations where there are two valid results of the conversion, an implementation may choose to make use of tm_gmtoff to decide between these two values (provided it can do so safely if tm_gmtoff is uninitialized), even though the strftime() %s conversion is not specified as making use of tm_gmtoff, since returning either of the values is allowed by the standard and the method used to choose between them is internal implementation detail that does not affect applications. Notes to the Editor (not part of this interpretation): ------------------------------------------------------- Using draft 4.0 line numbering: On page 452 lines 15762-15763 section XBD <time.h>, add CX shading to the lines for tm_gmtoff and tm_zone. On page 1211 line 41381 section gmtime() DESCRIPTION, change (already in CX shading): ...where the names in the structure and in the expression correspond. to: ...where the names in the structure and in the expression correspond; additionally, the tm_gmtoff field of the struct tm shall be set to 0, and the tm_zone field shall be set to a pointer to an implementation-defined string set to "UTC" or "GMT", which shall have static storage duration. On page 1211 line 41397 section gmtime() RETURN VALUE, delete: The structure’s tm_zone member shall be set to a pointer to the string "UTC", which shall have static storage duration. On page 1428 line 47958 section mktime(), change: shall calculate the time since the Epoch value using either the offset in effect before the change or the offset in effect after the change. to: shall calculate the time since the Epoch value using either the offset in effect before the change or the offset in effect after the change; mktime() may use the value of tm_gmtoff to decide which of these two results is the more appropriate to return, provided it can do so safely if tm_gmtoff is uninitialized. On page 2135 line 69773 section strftime(), change: If any of the specified values are outside the normal range, the characters stored are unspecified. to: If any of the specified values are outside the normal range, as if set by localtime(), the characters stored are unspecified. On page 2135 line 69777 section strftime(), change: Local timezone information shall be set as though strftime() called tzset(). to: It shall be implementation-defined whether local timezone information is set as though strftime() called tzset(). On page 2136 line 69836 section strftime(), change: Replaced by the number of seconds since the Epoch as a decimal number, calculated as described for mktime(). [tm_year, tm_mon, tm_mday, tm_hour, tm_min, tm_sec, tm_isdst] to: Replaced by the number of seconds since the Epoch as a decimal number, calculated as described for mktime(). Note that, unlike mktime(), the tm structure member values used by this conversion, other than tm_gmtoff, need to be within the normal range, as if set by localtime(), to avoid unspecified output; the value of tm_gmtoff can be uninitialized (see [xref to mktime()]). [tm_year, tm_mon, tm_mday, tm_hour, tm_min, tm_sec, tm_isdst, tm_gmtoff] After page 2140 line 70007 section strftime(), add a new paragraph to APPLICATION USAGE: Using a tm structure populated by gmtime() or gmtime_r() with the %s conversion of strftime() should be avoided, as the calculated seconds since the Epoch value will only correspond to the specified broken-down time if the offset from UTC of the local timezone happens to be zero at that time.

(0006692) geoffclare (manager) 2024-02-29 12:15	I would prefer the resolution in Note: 0006691 over the one in Note: 0006689 (i.e. I think POSIX should not change to allow the new TZDB behaviour). I agree with the points kre made in Note: 0006677 and in particular with the statement that we must not break apps which use strftime() %s without setting tm_gmtoff. The counter-argument that there are very few such apps does not hold water, for two reasons: 1. It is only possible to survey apps for which we have access to the source. We cannot know how many closed-source apps would be affected. 2. We should not consider only apps which use %s explicitly. There are also apps which use user-specified or configurable time format strings. Although a lot of these would be passing in a struct tm obtained from localtime() (as with, for example, GNU ls -l --time-style=+%s), passing one that is populated a different way is also a reasonable thing for an app to do. For example, it could store broken-down times in a standard format and want to present them to the user in their preferred format; to do this it could use strptime() followed by strftime(). Another point in favour of disallowing "blind" use of tm_gmtoff is to consider what system implementers will do when faced with the choice of how to handle a change in POSIX to allow it. In order to continue to support apps written for POSIX.1-2017 and earlier (which did not have tm_gmtoff and therefore conforming apps cannot set it directly) and also have the TZDB behaviour for apps written for POSIX.1-2024 and later, they would need to provide two different versions of strftime(), selected according to _POSIX_C_SOURCE or _XOPEN_SOURCE. It seems highly unlikely that any system implementers would bother to do that (it has been done in the past, but only when there was no choice); the new TZDB-like behaviour would therefore be rare and become a portability gotcha.

(0006693) shware_systems (reporter) 2024-02-29 16:19	Additionally, while the text above references how gmtime() should set tm_gmtoff, there is no equivalent text in localtime() for how that field should be set. Do current implementations just set it to zero, leave it as malloc() garbage, compute something if second TZ format is used? Also, it's questionable whether localtime_r() should behave as if tzset() is called, as the tm struct provided may have value set and unmodified that may conflict with the values in the TZ variable.

(0006698) geoffclare (manager) 2024-03-01 10:09	> Additionally, while the text above references how gmtime() should set tm_gmtoff, there is no equivalent text in localtime() for how that field should be set. Good point. My take on this observation is that the addition in gmtime() is unnecessary. Both gmtime() and localtime() are required to set all of the fields in struct tm, and the description of tm_gmtoff in <time.h> ("Seconds east of UTC") is sufficient to require what value it is set to: in the case of gmtime(), the number of seconds east of UTC is zero because gmtime() returns the broken-down time as UTC. The reason an explicit statement is needed about tm_zone for gmtime() is because its description in <time.h> ("Timezone abbreviation") is not sufficiently clear what string this should be for gmtime().

(0006700) eggert (reporter) 2024-03-02 09:02	> 1. It is only possible to survey apps for which we have access to the source. We cannot know how many closed-source apps would be affected. Although that’s true, we can use our best judgment about which programs will benefit. We know that there are programs that benefit from the resolution in Note: 0006689, because we have a bug report to that effect and it’s plausible there are more programs like that. Although these programs may not follow the strict reading of the standard, they’re plausible uses of the standard functions. In contrast, we don’t know of programs that would benefit from the resolution in Note: 0006691, and these programs are less plausible. > 2.... There are also apps which use user-specified or configurable time format strings. Although a lot of these would be passing in a struct tm obtained from localtime() (as with, for example, GNU ls -l --time-style=+%s), passing one that is populated a different way is also a reasonable thing for an app to do. Unfortunately it’s not reasonable. These apps are broken regardless of which resolution is taken, because the issue comes up with formats other than %s. Consider the following program, which simulates a user-specified format string containing "%Z" used to format a struct tm set by hand or by strptime: #include <stdio.h> #include <time.h> #include <string.h> int main (void) { char buf[1000]; struct tm tm; /* Make TM "uninitialized". */ memset (&tm, 0x81, sizeof tm); tm.tm_isdst = 0; strftime (buf, sizeof buf, "%Z", &tm); puts (buf); } This program dumps core on GNU/Linux, on FreeBSD, and I expect on pretty much any other implementation that supports TZDB-style struct tm, because it uses a bad pointer in tm.tm_zone. And yet the program conforms to POSIX.1-2017, which says that the call to strftime sets the buffer to some string. In other words, programmers can’t trust user-specified strftime formats on anything other than completely filled-in struct tm values, and that’s true regardless of what we do about %s. I suppose we could say that GNU/Linux, FreeBSD, and pretty much everyone else is broken. But it’s better to be realistic and tell users that when using strftime’s %z, %Z and %s formats, tm_zone and tm_gmtoff must be set to in-range values. Insisting on a pedantic reading based on older POSIX would not serve users well because it would give users the wrong impression of what code will actually work. > consider what system implementers will do We implementers will figure it out. It’s the user code that matters here anyway.

(0006701) kre (reporter) 2024-03-03 01:20	Re Note: 0006700 We know that there are programs that benefit from the resolution in Note: 0006689, because we have a bug report to that effect Incorrect bug reports should not be used to justify changes to anything (including tzcode). The code in that report was simply wrong, even with the wording in Note: 0006689 it would still be broken. Attempting to make broken code work is absurd. If I sent in a bug report claiming that my code, with a divide by 0, wasn't working as I expected it to, would you go fix the hardware/OS/something to generate the answer I demanded? In contrast, we don’t know of programs that would benefit from the resolution in Note: 0006691, and these programs are less plausible. Here's one, with all of the error handling, #includes, etc, removed to save space here (they exist in the real thing, naturally): main(int argc, char *argv) { struct tm tm; char p; int i; char buf[64]; p = strptime(argv[1], "%Y-%m-%d %H:%M:%S", &tm); tm.tm_isdst = -1; for (i = 2; i < argc; i++) { setenv("TZ", argv[i], 1); tzset(); strftime(buf, sizeof buf, "%s", &tm); printf("%s %s\n", buf, argv[i]); } exit(0); } It can be run like prog 2024-03-03T07:23:46 UTC Asia/Bangkok Australia/Melbourne \ America/New_York Europe/Berlin Africa/Cairo Africa/Windhoek \ America/St_Johns America/Buenos_Aires America/Anchorage \| sort -n to discover the east->west order of a set of random zones -- which in this case gives: 1709411026 Australia/Melbourne 1709425426 Asia/Bangkok 1709443426 Africa/Cairo 1709443426 Africa/Windhoek 1709447026 Europe/Berlin 1709450626 UTC 1709461426 America/Buenos_Aires 1709463226 America/St_Johns 1709468626 America/New_York 1709483026 America/Anchorage or it can be used to observe differences in summer time behaviour when used like: prog 2024-03-03T07:23:46 Australia/Darwin Australia/Adelaide \ Australia/Brisbane Australia/Sydney \| sort -n 1709411026 Australia/Sydney 1709412826 Australia/Adelaide 1709414626 Australia/Brisbane 1709416426 Australia/Darwin prog 2024-06-03T07:23:46 Australia/Darwin Australia/Adelaide \ Australia/Brisbane Australia/Sydney \| sort -n 1717363426 Australia/Brisbane 1717363426 Australia/Sydney 1717365226 Australia/Darwin 1717365226 Australia/Adelaide from which one observes that when summer time is not in effect, Brisbane and Sydney are the same, as are Darwin and Adelaide - with the former pair of zones achieving a wallclock time before the latter, but when summer time is in effect, all 4 are different, and Adelaide precedes Brisbane, rather than following it. There's no compelling reason to break this. I'm not sure I even really support allowing tm_gmtoff to be used (ever) to handle the very weird cases where nothing else is available to disambiguate the very odd cases, like the America/Caracas example given in the description, though if there's any good, and safe, way an implementation can truly: provided it can do so safely if tm_gmtoff is uninitialized. then it really all that important, provided the code given in this note always works as it should (where tm_gmtoff is entirely nonsense, one could init it to ~0 if desired - or leave it to generate a trap if referenced on hardware that supports that kind of thing). Also, I don't think that this: Note that, unlike mktime(), the tm structure member values used by this conversion, other than tm_gmtoff, need to be within the normal range, as if set by localtime(), to avoid unspecified output; is needed either. The reason that strftime() needs in range values in general, is not because of %s (which did not exist when that requirement was added) but to avoid problems with conversions like %a and %b if tm_wday or tm_mon are set to way out of range values (what is the name of the 29th day of the week?). But for that, all that is needed is that the values are within the ranges specified by <time.h> - there's no issue at all with tm.tm_mon = 1; /* Feb */ tm.tm_mday = 31; and then using strftime("%d %b"); The result might not make sense, but it is perfectly well defined. For %s we don't really need the "within range ..." restriction at all, because of the "calcluated as described by mktime()" - but nor do they do any harm really, as the only reason mktime() permits out of range values is to allow time arithmetic (which given the POSIX definition of time_t isn't really required by POSIX apps) and one thing I can't really imagine is anyone using strftime("%s") for that purpose. But there's no reason to make the ranges required for the %s conversion any tighter than for any other conversion that strftime() offers, (and in any case mktime() is a very poor, in fact, unworkable, interface for that feature, it works OK with gmtime()/timegm(), but it would actually be far better to create (not here, but by the implementations) a companion to difftime() (perhaps: time_t addtime(time_t, double); or something like that) to allow values to be added/subtracted from a time_t. (Then if there are implementations where time_t is not a simple integer count of seconds, they can do whatever is needed to make that work). The wording in Note: 0006691 looks OK to me (though I'd prefer the two changes just mentioned - that is, not to make the two changes referred to just above in this note).

(0006706) geoffclare (manager) 2024-03-07 10:44	>> consider what system implementers will do > We implementers will figure it out. It’s the user code that matters here anyway. I think you missed my point. I expect almost all system implementers will "figure it out" by taking the easy option and sticking with Issue 7 compatible behaviour. (If they have TZDB as the upstream for this stuff, they will revert the strftime() tm_gmtoff change, as NetBSD has done.) The existence of any system where the strftime() implementation behaves like TZDB then creates a portability problem for user code written on any of the other systems; it will work fine on most systems but misbehave if/when it is ever ported to this unusual and rare system.

(0006707) geoffclare (manager) 2024-03-07 11:04	> This program dumps core on GNU/Linux, on FreeBSD, and I expect on pretty much any other implementation that supports TZDB-style struct tm, because it uses a bad pointer in tm.tm_zone. And yet the program conforms to POSIX.1-2017, which says that the call to strftime sets the buffer to some string. The program conforms to POSIX.1-2017, and your results show that GNU/Linux and FreeBSD apparently do not; these implementations must be using tm_zone "blindly" and do not conform because POSIX.1-2017 requires that the only struct tm field used by %Z is tm_isdst. (This doesn't mean implementations can't use non-standard fields such as tm_zone, but if they are used, it must be in such a way that a conforming application can't tell the difference, i.e. it is just internal implementation detail. In the case of tm_zone this would mean checking the pointer is equal to a known valid value for the current timezone before using it.)

(0006718) geoffclare (manager) 2024-03-11 09:55	> I suppose we could say that GNU/Linux, FreeBSD, and pretty much everyone else is broken. But it’s better to be realistic and tell users that when using strftime’s %z, %Z and %s formats, tm_zone and tm_gmtoff must be set to in-range values. For %z and %Z, requiring applications to set tm_zone and tm_gmtoff would create a conflict with the C standard. C17 says that the only member used by %z and %Z is tm_isdst. Of course, if an application explicitly sets tm_zone or tm_gmtoff then it is using an extension to C17 and this could alter the behaviour, and if the structure was populated by localtime() et al then use of tm_zone or tm_gmtoff can be an internal implementation detail, as per my previous note. However, the crucial case is when the application populates the structure in a way that does not set tm_zone or tm_gmtoff; for this case C17 requires that the conversion is done based on tm_isdst.

Issue History
Date Modified	Username	Field	Change
2024-01-15 23:56	eggert	New Issue
2024-01-15 23:56	eggert	Name	=> Paul Eggert
2024-01-15 23:56	eggert	Organization	=> UCLA Computer Science Dept.
2024-01-15 23:56	eggert	User Reference	=> strftime-%s
2024-01-15 23:56	eggert	Section	=> strftime
2024-01-15 23:56	eggert	Page Number	=> 2136
2024-01-15 23:56	eggert	Line Number	=> 69836-69837
2024-01-16 01:29	steffen	Note Added: 0006623
2024-01-16 01:42	steffen	Note Added: 0006624
2024-01-16 01:42	steffen	Note Deleted: 0006623
2024-02-01 16:42	nick	Relationship added	related to 0001533
2024-02-12 16:11	eblake	Note Added: 0006651
2024-02-25 06:50	kre	Note Added: 0006677
2024-02-25 06:54	kre	Note Edited: 0006677
2024-02-26 18:28	eggert	Note Added: 0006688
2024-02-26 19:23	eblake	Relationship added	related to 0001816
2024-02-26 19:32	eblake	Note Added: 0006689
2024-02-26 19:52	eblake	Relationship added	child of 0001612
2024-02-26 19:55	eblake	Relationship added	child of 0000169
2024-02-26 20:02	eblake	Note Added: 0006690
2024-02-29 12:10	geoffclare	Note Added: 0006691
2024-02-29 12:15	geoffclare	Note Added: 0006692
2024-02-29 16:19	shware_systems	Note Added: 0006693
2024-03-01 10:09	geoffclare	Note Added: 0006698
2024-03-02 09:02	eggert	Note Added: 0006700
2024-03-03 01:20	kre	Note Added: 0006701
2024-03-07 10:44	geoffclare	Note Added: 0006706
2024-03-07 11:04	geoffclare	Note Added: 0006707
2024-03-11 09:55	geoffclare	Note Added: 0006718

Aardvark Mark IV