 Notes |
(0006412)
Don Cragun (manager)
2023-07-31 16:13
|
We looked at this issue again during the 2023-07-31 conference call. We are still waiting for a note with suggested changes from kre. |
|
(0006459)
kre (reporter)
2023-09-07 14:14
|
I am about to attach a series of notes - making them separate rather than one
huge combined note might make it easier to follow. When I am done, I will
append one more note, which will just be an index to the previous notes,
so you can more easily jump to whatever might be of interest.
The following note will be my proposed replacement for section 2.6.5.
As foreshadowed in the Description above "This whole section needs
(yet another) complete rewrite." that is exactly what I have done.
As always with my text, the words in the next note are not intended to
remain untouched - by all means remove the redundant noise I tend to
include, make it more "standards like" (correct terminology, formatting,
etc), and whatever else. But I believe that what is here is the simplest
most concise description I can give of how all this really operates (or should).
As always of course, all that is required is an "as if" behaviour, the
shell I maintain certainly doesn't do things in the slow baroque way described
here, but the results are the same.
Notes following will show my test program (rather bigger than the earlier one
and easier to follow, as the text names show exactly what the input was, see
the relevant note) and some discussion of what doesn't work, and as best I
can tell why, in the one (two if one counts zsh - I am not counting at all, or
even testing any more, the ancient broken PD ksh that is NetBSD's /bin/ksh)
shell which doesn't produce the same results. Some of that might be argued
to be following an interpretation of the standard as it has been, but other
differences seem to simply be broken. |
|
(0006460)
kre (reporter)
2023-09-07 14:15
|
2.6.5 Field Splitting
After parameter expansion (Section 2.6.2), command substitution
(Section 2.6.3), and arithmetic expansion (Section 2.6.4), and
if the shell variable IFS is set, and its value is not empty, or
if IFS is unset, the shell shall scan each field containing results of
expansions and substitutions that did not occur in double-quotes for
field splitting and multiple fields can result.
If IFS is set with an empty string as its value, no field splitting occurs,
however if an input field which contained expansions or substitutions is
entirely empty, it shall be removed. Note that this occurs before quote
removal, any input field that contains any quoting characters can never
be empty at this point.
Fields which contain no unquoted results from any of the expansions or
substitutions shall not be affected by field splitting, and shall remain
unaltered.
In the following description, it is assumed that there is in the
field at least one unquoted expansion or substitution result present, this
assumption will not be restated. Field splitting only considers altering
those parts of the field.
For the purposes of this section, the term "IFS white space" shall mean
any of the characters <space> <tab> or <newline> which are present in
the value of the IFS variable. It is implementation defined whether
other White Space characters which appear in the value of IFS are also
considered as "IFS white space". If the IFS variable is unset, then for
the purposes of this section, but without altering the value of the
variable, its value shall be considered to contain the three characters
<space> <tab> and <newline>, all of which are IFS white space characters.
The three characters above defined as IFS white space characters are always
IFS white space, when they occur in the value of IFS, for other locale
specific White Space characters allowed by the implementation it is
unspecified whether the character is considered as IFS white space if it
is White Space at the time it is assigned to the IFS variable, or if it
is White Space at the time field splitting occurs (the locale may have
altered between those events).
The shell shall treat non-empty sequences formed from the characters in
the IFS value, in any order, provided that no sequence contains more than
one character that is not IFS white space, and use the delimiters as
field terminators to split the results of parameter expansion, command
substitution, and arithmetic expansion into separate fields, as described
below. Note that these delimiters terminate a field, they do not, of
themselves, cause a new one to start, subsequent non IFS whitespace
characters in the field are required for a new field to begin.
If results of the algorithm are that no fields are delimited, that is,
if the input is wholly empty or consists entirely of IFS white space,
the result shall be zero fields (rather than an empty field).
For the purposes of this section, when a field is said to be delimited,
then the characters from the input from after the delimiter which terminated
the previous field, or from the beginning of the input if this is the first,
up to the point immediately preceding the delimiter, shall be considered as
a candidate for forming an output field. If that candidate is not empty,
or if both the previous output field (if any), and the current field
being delimited were delimited a delimiter that includes a character which is
not IFS white space, then the candidate shall become an output field. If
the current candidate output field is empty, and either it, or the previous
output field were delimited by only IFS white space, then the empty candidate
shall be removed, and shall not form an output field.
Each expansion, or substitution shall be processed in order as follows,
examining characters in the input field, left to right, and beginning with
an empty candidate field:
While the input is not empty...
When instructed to perform the next iteration, start
again here, with the input as modified.
Consider the first remaining character of the input. If it is:
a. A character that did not result from an unquoted expansion
or substitution:
Append this character to the candidate, and remove it
from the input. Next iteration of the loop.
b. A character in the input that is not a character in IFS:
Append this character to the candidate, and remove it
from the input. Next iteration.
c. An IFS white space character:
Remove that character from the input, advance
to the next input character, and repeat this step.
d. Another IFS character, not IFS white space:
Remove that character from the input, but note it was
observed. Remove any immediately following IFS white
space characters from the input.
At this point, if the candidate is not empty, or if a non IFS white space
character was seen at step d, then the candidate becomes an output field.
In either case, empty the candidate, and perform the next iteration.
When the input is empty, if the candidate is not empty, it becomes an
output field.
The set of output fields so produced, which may be none, replaces the
input field. |
|
(0006462)
kre (reporter)
2023-09-07 14:30
edited on: 2023-09-07 14:43
|
This is an extract from my test program for the shells, I won't include all
of it here, as most of the end is simply lots and lots of weird test cases.
In all of them, the test case names tell you what the input was, each
character of the name represents something in the input. 'p' is a
prefix - characters that are part of the field, that are not part of an
unquoted expansion or substitution. 'q' is the suffix, the same as 'p'
but following the interesting bit. There is always an interesting bit,
fields that contain nothing to split at all are boring, so we have none of
those. Because no field splitting happens in the prefix or suffix, in the
tests those contain no chars that could be confused with IFS characters.
This simplifies things for my sh strawman implementation of the algorithm
in the previous note (which will appear below).
'W' is a word - a sequence of chars which contain no IFS chars. In the tests
the first word is always "abc" and the second "def" (no tests currently have
more than two, but you can guess what the third would be...)
S and C are the IFS chars. In these tests IFS is always ' ,' (space and
comma). One IFS white space character, and one other. This simplifies the
testing, without limiting the generality of the algorithm (my strawman
implementation only handles those, a real implementation would simply treat
any IFS white space the way I treat space, and any other IFS character the
way I treat ',').
That's it... The order of the characters in the test name shows the order
in which those elements appear in the input field, to be split. In this
test we use the shell running the test to perform the field splitting.
The NetBSD and FreeBSD shells, bash, bosh, dash, ksh93, yash, all produce
identical results for all the tests. zsh seems to think a terminating
non-whitespace IFS character starts a new field (in --emulate sh mode, otherwise
the results are nothing like this at all). mksh will be discussed in a
later note.
Here is the test code, and some of the initial tests. I will attempt to
upload the entire thing as an attached file, later.
args()
{
name=$1; shift
printf '%s:\t%d:\t' "$name" "$#"
printf '[%s]' "$@"
printf '\n'
}
IFS=' ,'
S=' '
C=','
W='abc'
SW=' abc'
WS='abc '
# many more omitted here
# The tests:
args W $W
args SW $SW
args WS $WS
args SWS $SWS
args CW $CW
args WC $WC
[ and many nore]. The next note will show the complete set of results.
|
|
(0006464)
kre (reporter)
2023-09-07 14:45
|
W: 1: [abc]
SW: 1: [abc]
WS: 1: [abc]
SWS: 1: [abc]
CW: 2: [][abc]
WC: 1: [abc]
CWC: 2: [][abc]
WSW: 2: [abc][def]
WSSW: 2: [abd][def]
WCW: 2: [abc][def]
WCCW: 3: [abc][][def]
WSCW: 2: [abc][def]
WCSW: 2: [abc][def]
WSCSW: 2: [abc][def]
WSCSCSW: 3: [abc][][def]
WSCSCSWS: 3: [abc][][def]
WSCSCSWC: 3: [abc][][def]
S: 0: []
C: 1: []
SS: 0: []
SSS: 0: []
CC: 2: [][]
CCC: 3: [][][]
SC: 1: []
CS: 1: []
SCCS: 2: [][]
CSSC: 2: [][]
SCWSCWCS: 3: [][abc][def]
SSCSSCSSCSS: 3: [][][]
pW: 1: [pabc]
pSW: 2: [p][abc]
pWS: 1: [pabc]
pSWS: 2: [p][abc]
pCW: 2: [p][abc]
pWC: 1: [pabc]
pCWC: 2: [p][abc]
pWSW: 2: [pabc][def]
pWSSW: 2: [pabd][def]
pWCW: 2: [pabc][def]
pWCCW: 3: [pabc][][def]
pWSCW: 2: [pabc][def]
pWCSW: 2: [pabc][def]
pWSCSW: 2: [pabc][def]
pWSCSCSW: 3: [pabc][][def]
pWSCSCSWS: 3: [pabc][][def]
pWSCSCSWC: 3: [pabc][][def]
pS: 1: [p]
pC: 1: [p]
pSS: 1: [p]
pSSS: 1: [p]
pSC: 1: [p]
pCS: 1: [p]
pCSSC: 2: [p][]
pSSS: 1: [p]
pCCC: 3: [p][][]
pSCCS: 2: [p][]
pSCWSCWCS: 3: [p][abc][def]
pSSCSSCSSCSS: 3: [p][][]
Wq: 1: [abcq]
SWq: 1: [abcq]
WSq: 2: [abc][q]
SWSq: 2: [abc][q]
CWq: 2: [][abcq]
WCq: 2: [abc][q]
CWCq: 3: [][abc][q]
WSWq: 2: [abc][defq]
WSSWq: 2: [abd][defq]
WCWq: 2: [abc][defq]
WCCWq: 3: [abc][][defq]
WSCWq: 2: [abc][defq]
WCSWq: 2: [abc][defq]
WSCSWq: 2: [abc][defq]
WSCSCSWq: 3: [abc][][defq]
WSCSCSWSq: 4: [abc][][def][q]
WSCSCSWCq: 4: [abc][][def][q]
Sq: 1: [q]
Cq: 2: [][q]
SSq: 1: [q]
SSSq: 1: [q]
SCq: 2: [][q]
CSq: 2: [][q]
CSSCq: 3: [][][q]
SSSq: 1: [q]
CCCq: 4: [][][][q]
SCCSq: 3: [][][q]
SCWSCWCSq: 4: [][abc][def][q]
SSCSSCSSCSSq: 4: [][][][q]
pWq: 1: [pabcq]
pSWq: 2: [p][abcq]
pWSq: 2: [pabc][q]
pSWSq: 3: [p][abc][q]
pCWq: 2: [p][abcq]
pWCq: 2: [pabc][q]
pCWCq: 3: [p][abc][q]
pWSWq: 2: [pabc][defq]
pWSSWq: 2: [pabd][defq]
pWCWq: 2: [pabc][defq]
pWCCWq: 3: [pabc][][defq]
pWSCWq: 2: [pabc][defq]
pWCSWq: 2: [pabc][defq]
pWSCSWq: 2: [pabc][defq]
pWSCSCSWq: 3: [pabc][][defq]
pWSCSCSWSq: 4: [pabc][][def][q]
pWSCSCSWCq: 4: [pabc][][def][q]
pSq: 2: [p][q]
pCq: 2: [p][q]
pSSq: 2: [p][q]
pSSSq: 2: [p][q]
pSCq: 2: [p][q]
pCSq: 2: [p][q]
pCSSCq: 3: [p][][q]
pSSSq: 2: [p][q]
pCCCq: 4: [p][][][q]
pSCCSq: 3: [p][][q]
pSCWSCWCSq: 4: [p][abc][def][q]
pSSCSSCSSCSSq: 4: [p][][][q] |
|
(0006465)
kre (reporter)
2023-09-07 14:54
|
Apologies for the mess with the original version of the results, those
reading this via the mailing list will note that I was using angle brackets
as the field delimiter characters, to show what is in each field, and totally
forgot that mantis would interpret those, so I have just done a quick switch
to use square brackets, which works for the note, but is much uglier to look at.
Anyway, here is the (now using []) shell strawman implementation of the
algorithm in Note: 0006460 .
Again, this is truncated, most of the actual test cases are omitted, though
you can deduce what they are from the results in Note: 0006464
This test does (or did before I fiddled with the "args()" function which
prints the results just now, produce identical results to the version run
by the shell, in Note: 0006464 - so I am not going to include those again.
I have run this with every reaosnable shell I have (not pdksh, and not zsh,
as I don't really understand its differences). All of them (mksh included)
produce the same results here, so I believe the code is portable enough.
# This is a dummy implementation of the proposed field splitting
# algorithm (witten in sh, so hopefully sh people can follow it)
# to demonstrate that the algorithm as presented generates the
# expected output (that generated by almost every shell).
# This code knows that in the tests IFS=' ,' (space and comma)
# and rather than handling that generically, which would be possible,
# but messy, simply builds those two characters (literally) into the
# implementation (space, as a IFS white space char, and comma as an
# IFS char that is not white space).
# Similarly the code "knows" that if there is a prefix in the field
# (chars not to be treated as generated by an expansion, and hence
# exepmt fmom splitting) that will be simply a single 'p' always, and
# siumilarly a suffix will be 'q' - because of that we do not need to
# have any method to indicate what part of the field is to be subject
# to field splitting
# In the following comments that start '##' are text lifted directly
# from my proposed section 2.6.5 ("Field Splitting") text, which might
# allow readers to match this algorithm with what is described there.
# The results from this test match exactly the results from all shells
# considered to operate correctly (the same output routine is used, and
# the results compared with diff - with zero differences).
S=' '
C=','
field_split() {
ARG=$1 # the field that needs to be split
set -- # the set of output fields, initially empty
# IFS is defined (IFS=' ,') and not empty, IFS white space is ' '
# We simply know that!
# C is our candidate field,
# CD indicates the delimiter that terminated the candidate field
# ' ' indicates the delimiter was IFS white space alone
# ',' indicates the delimuter was a ',' (perhaps with white space)
# '' indicates there has been no delimiter
C= CD=
## Each expansion, or substitution shall be processed in order
## as follows [...]
## While the input is not empty...
while test -n "${ARG}"
do
## Consider the first remaining character of the input.
## If it is:
## a. A character that did not result from an unquoted
## expansion or substitution:
## b. A character in the input that is not a character in IFS:
# since we know exactly what the IFS chars are, and that
# chars that did not result from an expandion (etc) are not
# IFS chars (our test cases ensure that) we don't need to
# treat those two differently, just skip forward until we
# get to an IFS char, or we run out, appending the non-IFS
# chars to the candidate and removing them from the input.
# here we only care about the current first char in ${ARG}
while case "${ARG}" in
'') break 2 # the end of the input, done
;;
[\ ,]*) false # delimiter located, exit loop
;;
*) TAIL=${ARG#?} # something else
C=${C}${ARG%"${TAIL}"} # appended to candidate
ARG=${TAIL} # removed from input
;;
esac
do
:
done
# Now we are at the start of a delimiter in ARG, and the
# candidate field is C
# which kind of delimiter do we have?
## c. An IFS white space character:
# assume the delim will be just IFS white space (case 'c')
CD=' '
# and then skip any of that we find (repeating 'c' over & over)
while case "${ARG}" in
' '*) ARG=${ARG#* };;
*) false;;
esac
do :; done
## d. Another IFS character, not IFS white space:
# Next if we have a non white space IFS char,
# then it is the other kind of delimiter (case 'd' in the algo)
case "${ARG}" in
,*) CD=, ; ARG=${ARG#,} # Remember we saw it, then remove
# and skip any following IFS white space
while case "${ARG}" in
' '*) ARG=${ARG#* };;
*) false;;
esac
do :; done
;;
esac
# now a field has been delimited so we are subject to:
## At this point, if the candidate is not empty, or if a
## non IFS white space character was seen at step d, then
## the candidate becomes an output field.
## In either case, empty the candidate, and perform the
## next iteration.
if test -n "${C}" # candicate is not empty (or...) => output
then
## if the candidate is not empty
## then the candidate becomes an output field.
set -- "$@" "'${C}'"
# otherwise The candidate is empty, if it was delimited
# by only IFS white space, then candidate is dropped
elif test "${CD}" != ' '
then
## or if a non IFS white space character was seen
## then the candidate becomes an output field.
set -- "$@" "''" # no need for $C, it is ""
fi
## In either case, empty the candidate, and perform
## the next iteration.
CD=
C=
done
## When the input is empty, if the candidate is not empty, it
## becomes an output field.
if test -n "${C}"
then
# not an empty field after last delim, so it is included
set -- "$@" "'${C}'"
fi
# return the split field, as a list of quoted words (to become fields)
printf %s "$*"
}
args()
{
name=$1; shift
printf '%s:\t%d:\t' "$name" "$#"
printf '[%s]' "$@"
printf '\n'
}
tst()
{
N=$1
eval set -- $(field_split "$2")
args "$N" "$@"
}
W='abc'
SW=' abc'
WS='abc '
SWS=' abc '
CW=',abc'
WC='abc,'
CWC=',abc,'
WSW='abc def'
WSSW='abd def'
# and many more definitions like that
# followed by the actual test invocations
tst W "$W"
tst SW "$SW"
tst WS "$WS"
tst SWS "$SWS"
tst CW "$CW"
tst WC "$WC"
tst CWC "$CWC"
tst WSW "$WSW"
tst WSSW "$WSSW"
tst WCW "$WCW"
tst WCCW "$WCCW"
tst WSCW "$WSCW"
tst WCSW "$WCSW"
tst WSCSW "$WSCW"
tst WSCSCSW "$WSCSCSW"
# and many more. |
|
(0006466)
kre (reporter)
2023-09-07 15:01
|
Here, for the record, are the differences between the intended results
(produced by almost all shells) and those of zsh
diff /tmp/good /tmp/zsh
6,7c6,7
< WC: 1: [abc]
< CWC: 2: [][abc]
---
> WC: 2: [abc][]
> CWC: 3: [][abc][]
17c17
< WSCSCSWC: 3: [abc][][def]
---
> WSCSCSWC: 4: [abc][][def][]
19c19
< C: 1: []
---
> C: 2: [][]
22,29c22,29
< CC: 2: [][]
< CCC: 3: [][][]
< SC: 1: []
< CS: 1: []
< SCCS: 2: [][]
< CSSC: 2: [][]
< SCWSCWCS: 3: [][abc][def]
< SSCSSCSSCSS: 3: [][][]
---
> CC: 3: [][][]
> CCC: 4: [][][][]
> SC: 2: [][]
> CS: 2: [][]
> SCCS: 3: [][][]
> CSSC: 3: [][][]
> SCWSCWCS: 4: [][abc][def][]
> SSCSSCSSCSS: 4: [][][][]
35,36c35,36
< pWC: 1: [pabc]
< pCWC: 2: [p][abc]
---
> pWC: 2: [pabc][]
> pCWC: 3: [p][abc][]
46c46
< pWSCSCSWC: 3: [pabc][][def]
---
> pWSCSCSWC: 4: [pabc][][def][]
48c48
< pC: 1: [p]
---
> pC: 2: [p][]
51,53c51,53
< pSC: 1: [p]
< pCS: 1: [p]
< pCSSC: 2: [p][]
---
> pSC: 2: [p][]
> pCS: 2: [p][]
> pCSSC: 3: [p][][]
55,58c55,58
< pCCC: 3: [p][][]
< pSCCS: 2: [p][]
< pSCWSCWCS: 3: [p][abc][def]
< pSSCSSCSSCSS: 3: [p][][]
---
> pCCC: 4: [p][][][]
> pSCCS: 3: [p][][]
> pSCWSCWCS: 4: [p][abc][def][]
> pSSCSSCSSCSS: 4: [p][][][]
I believe, that all of these differences occur when there is a trailing
'C' in the input, with no other following non IFS-whitespace (but any amount
of that surrounding the C). zsh is appending an additional empty field.
I believe that is the only difference, but haven't examined all of those in
detail. |
|
(0006467)
kre (reporter)
2023-09-07 15:03
|
And the differences produced by mksh:
51c51
< pSC: 1: [p]
---
> pSC: 2: [p][]
56,58c56,58
< pSCCS: 2: [p][]
< pSCWSCWCS: 3: [p][abc][def]
< pSSCSSCSSCSS: 3: [p][][]
---
> pSCCS: 3: [p][][]
> pSCWSCWCS: 4: [p][][abc][def]
> pSSCSSCSSCSS: 4: [p][][][]
109c109
< pSCq: 2: [p][q]
---
> pSCq: 3: [p][][q]
114,116c114,116
< pSCCSq: 3: [p][][q]
< pSCWSCWCSq: 4: [p][abc][def][q]
< pSSCSSCSSCSSq: 4: [p][][][q]
---
> pSCCSq: 4: [p][][][q]
> pSCWSCWCSq: 5: [p][][abc][def][q]
> pSSCSSCSSCSSq: 5: [p][][][][q]
That's a much smaller set, but they're much harder to characterise,
so for now, I won't attempt that. |
|
(0006468)
kre (reporter)
2023-09-07 15:15
|
I have just attached 3 new files (to join the original "ifs" which was the
test used when this report was originally filed).
IFS-test is the test to run with a shell, just $SHELL IFS-test >results
That is what is abbreviated in Note: 0006462 but I uploaded the version which
retains the angle bracket delimiters, those are much nicer to read the results.
POSIX-bug-1649-impl.sh is the complete strawman implementation and test, as
in Note: 0006465 - again the angle bracket version.
Expected-Results is what you should expect to see from those, just the same
as what is in Note: 0006464 but with angle brackets instead of square ones. |
|
(0006469)
kre (reporter)
2023-09-07 15:21
edited on: 2023-09-11 03:39
|
Note: 0006459 - a brief introductuction to the proposed solution
Note: 0006460 - a very much draft replacement for XCU 2.6.5
Note: 0006478 - the first revision of the text in Note: 0006460 (a still very much draft replacement for XCU 2.6.5)
Note: 0006462 - a description of the shell test (IFS-test), and its useful code
Note: 0006464 - the expected results (the square bracket version)
Note: 0006465 - an intro to, and the essential code of, a strawman implementation (as sh code) of the algorithm in Note: 0006460
Note: 0006466 - the differences (one difference, repeated) from zsh
Note: 0006467 - the differences in the output from mksh (perhaps someone can explain them)
Note: 0006468 - a note about the attached files that have been uploaded
Note: 0006469 - this one, which is the last of this set.
|
|
(0006471)
kre (reporter)
2023-09-07 16:12
|
Oops, I just realised that the args() function I managed to leave in
those tests, is missing a test, the middle of the 3 printf statements
(the one that contains the problematic brackets, so I won't include
all of it) should look like:
test $# -gt 0 && printf ...
(with the args to printf being the same as they are now). This applies
to both test programs (that function was just copied from one to the other).
That explains why when the results say 0 fields, it prints one anyway (the 0
is correct), the empty field output is nonsense. Sorry about that!
This makes no difference when comparing the results (as the same issue
affected everything), but if you're trying to look at an affected result,
it can be confusing (it was to me when I saw it first).
I know I had that extra "test" in there in some earlier version, I
must have lost that somewhere. |
|
(0006472)
geoffclare (manager)
2023-09-07 16:40
|
The problem description includes mention of character encoding and LC_CTYPE, but the proposed wording in Note: 0006460 does not address this. Instead it describes everything in terms of characters, effectively reversing the changes made by bug 0001560. Without those changes it is unclear how field splitting works on command substitutions (or pathname expansions) where the expansion results contain byte sequences that do not form characters.
Changes along the lines of those in bug 1560 need to be made to the proposed new wording. The character encoding issue could perhaps be partially addressed by extending the requirement that <period>, <slash>, <newline>, and <carriage-return> are invariant across all locales (see XBD 6.1) to include at least <space> and <tab> as well. How the handling of IFS characters with non-invariant encodings should be specified (i.e. "which LC_CTYPE") will depend on what shells currently do.
(Please note that after posting this I will be unavailable for three weeks, so will not be able to respond to any follow up comments in a timely manner.) |
|
(0006473)
kre (reporter)
2023-09-07 17:06
edited on: 2023-09-07 17:09
|
Re Note: 0006472
The following section in Note: 0006460 was intended to cover as much as I
considered a problem with LC_CTYPE (though it doesn't mention that by
name):
It is implementation defined whether
other White Space characters which appear in the value of IFS are also
considered as "IFS white space".
(and a little bit more explanation a bit later in the same paragraph).
It also explicitly says that space tab and newline (if included in IFS
obviously) are always IFS white space characters, regardless of locale.
I had no intent to change anything related to whatever bug:1560 addresses,
one I didn't know existed (I suppose I saw it once, but have forgotten...)
I will take a look at that, but not tonight, and see if I can work out
what to do to reconcile the two - but I did note that I knew the wording
would need changes, handling that kind of thing is (partly) what I meant.
I wanted to (hopefully finally, and forever, if I got it right - and everyone
who knows an implementation, please do check to verify that) get the
algorithm that is used actually specified correctly, finally. Beyond that
was for some other time (or someone else) - but I will take a look, later.
|
|
(0006477)
kre (reporter)
2023-09-11 03:35
|
I am about to add a new note (to follow this one) with revised text
from that in Note: 0006460 to (hopefully) address some of the concerns
in Note: 0006472
The 7th paragraph (not counting the section title as a paragraph)
is intended to contain all that I know what is relevant about issues
relating to LC_CTYPE, and what happens if it changes during sh execution.
Note that I know essentially zero about how to handle multi-byte
characters (that is, how such things are intended to be processed by
shells) and consequently this might be inadequate. But someone else who
does understand (if there is anyone, since this all appears to be black magic
to me) will need to provide better or additional wording.
I have made revisions to the wording to hopefully make it fall more into
line with what bug:1560 was attempting to achieve in this area (though I
am not at all convinced that what it changed would have been nearly enough
to make a coherent result even with the largely wishy-washy specification of
field splitting that we currently have (up to and including Draft 3 of Issue 8).
I also made some corrections to issues that were largely the result of my
being in a hurry (and being rather late in my day) to get something posted
here before last Thursday's Austin Group meeting, just in case this was to be
considered again then. There was text that remained there which was from
earlier ideas, which had been half deleted, but not completely, and poor
descriptions, like considering the fields as a set (which would be an
unordered collection) and even worse describing it as a "set which may
be none" (I don't even know what that means) rather than "may be an empty set".
I have also simplified things somewhat, to make it easier to read, though
the actual algorithm remains unchanged - at least when only considering cases
where all characters are one byte, which is all I really know anything about.
I have tried to make the algorithm handle the case where that isn't true,
with multi-byte characters, but I really have no idea whether what I wrote
makes any sense or not (I didn't remember bug:1560 as it is about that kind
of issue, and I simply switch off whenever those issues are being discussed,
I know I know nothing of the issues involved, so have nothing to contribute).
But it is clear from that, and from the definition of IFS in XCU 2.5.3, that
IFS is supposed to contain characters, and input fields just byte sequences,
and that somehow field splitting is supposed to reconcile that. Since nothing
(no shell) I'm aware of actually attempts to fo that (at least as far as I
know, I'm not really capable of testing it) I have no idea whether what I have
written here is anything like what is intended to happen, or simply fantasy.
Someone who understands what should happen (even better, if there's a shell
which actually does it, who understands what does happen) needs to look very
carefully and make corrections as appropriate (to the wording, or the
algorithm).
As well as including it in the following note, I will upload the same text
as an attached file, "Revised-bug-1649-suggestion", to make it easier for
someone else to download and correct (rather than needing to cut and paste
from one of these notes, or from an e-mail message).
Note that I have not altered the tests (though my local copy has one small
section removed, to bring it in line with the slightly simplified algorithm
specified now - none of the test results alter - the previous text was the
way it was just because it had always been specified like that, there really
was no good reason for that). The ## comments in the prototype impl (in sh)
which contain lines from the spec are still from the Note: 0006460 version, not
this new one - but as the basic algorithm is unaltered, it should be easy
enough to find them there, and correlate with the newer version. |
|
(0006478)
kre (reporter)
2023-09-11 03:36
|
2.6.5 Field Splitting
After parameter expansion (Section 2.6.2), command substitution
(Section 2.6.3), and arithmetic expansion (Section 2.6.4), and
if the shell variable IFS [xref XCU 2.5.3] is set and its value is
not empty, or if the IFS variable is unset, the shell shall scan each
field containing results of expansions and substitutions that did not
occur in double-quotes for field splitting; zero, one or multiple fields
can result.
For the remainder of this section, any reference to the results of an
expansion, or results of expansions, shall be interpreted to mean the
results from one or more unquoted variable or arithmetic expansions,
or unquoted command substitutions.
If the IFS variable is set and has an empty string as its value, no field
splitting occurs. However if an input field which contained the results of
an expansion is entirely empty, it shall be removed. Note that this occurs
before quote removal, any input field that contains any quoting characters
can never be empty at this point. After from the removal of any such
fields from the input, the possibly modified input field list becomes
the output.
Each input field is considered in sequence, first to last, with the
results of the algorithm described in this section causing output fields
to be generated, which remain in the same order as the input fields
from which they originated.
Fields which contain no results from expansions shall not be affected by
field splitting, and shall remain unaltered, simply moving from the
list of input fields to be next in the list of output fields.
In the remainder of this description, it is assumed that there is present
in the field at least one expansion result, this assumption will not be
restated. Field splitting only ever alters those parts of the field.
For the purposes of this section, the term "IFS white space" shall mean
any of the single byte characters <space>, <tab> or <newline> from the
Portable Character Set [xref XBD 6.1] which are present in the value of
the IFS variable, and perhaps other White Space characters. It is
implementation defined whether other White Space characters which appear
in the value of IFS are also considered as "IFS white space". The three
characters above specified as IFS white space characters are always IFS
white space, when they occur in the value of IFS, regardless of whether
they are White Space characters in any relevant locale. For other locale
specific White Space characters allowed by the implementation it is
unspecified whether the character is considered as IFS white space if it
is White Space at the time it is assigned to the IFS variable, or if it
is White Space at the time field splitting occurs (the locale may have
altered between those events).
If the IFS variable is unset, then for the purposes of this section, but
without altering the value of the variable, its value shall be considered
to contain the three single byte characters <space>, <tab> and <newline>
from the Portable Character Set, all of which are IFS white space characters.
The shell shall use the byte sequences that form the characters in the
value of the IFS variable as delimiters. Each of the characters <space>
<tab> and <newline> which appears in the value of IFS shall be a single
byte delimiter. The shell shall use these delimiters as field terminators
to split the results of expansions, along with other adjacent bytes, into
separate fields, as described below. Note that these delimiters terminate
a field, they do not, of themselves, cause a new field to start, subsequent
data bytes that are not from the results of an expansion, or that do not
form IFS whitespace characters are required for a new field to begin.
Note that the shell processes arbitrary bytes from the input fields, there
is no requirement that those bytes form valid characters.
If results of the algorithm are that no fields are delimited, that is,
if the input field is wholly empty or consists entirely of IFS white space,
the result shall be zero fields (rather than an empty field).
For the purposes of this section, when a field is said to be delimited,
the the candidate field, as generated below shall become an output field.
When the algorithm transforms a candidate into an output field it shall be
appended to the current list of output fields.
Each field containing the results from an expansion shall be processed
in order, intermixed with fields not containing the results of expansions,
processed as described above, as if as follows, examining bytes in the input
field, left to right:
Begin with an empty candidate field.
While the input is not empty...
When instructed to perform the next iteration of the loop, start
again with the test here, with the input as modified below.
Consider the leading remaining byte or byte sequence of the input.
No such byte sequence shall contain data such that some bytes in the
sequence resulted from an expansion, and others did not. If the byte
or sequence of bytes is:
a. A byte (or sequence of bytes) in the input that did not result
from an expansion:
Append this byte (or sequence) to the candidate, and remove
it from the input. Next iteration of the loop.
b. A byte sequence in the input which resulted from an expansion
that does not form a character in IFS:
Append the first byte of the sequence to the candidate,
and remove that byte from the input. Next iteration of
the loop.
c. A byte sequence in the input which resulted from an expansion
that forms an IFS white space character:
Remove that byte sequence from the input, consider
the new leading input byte sequence, and repeat this step.
d. A byte sequence in the input that resulted from an expansion
that forms an IFS character, which is not IFS white space:
Remove that byte sequence from the input, but note it was
observed.
At this point, if the candidate is not empty, or if a sequence of bytes
representing an IFS character that is not IFS white space was seen at
step d, then a field is said to have been delimited, and the candidate
becomes an output field.
Empty (clear) the candidate, and perform the next iteration of the loop.
When the input is empty, if, and only if, the candidate is not empty, it
becomes an output field.
The ordered list of output fields so produced, which may be empty, replaces
the list of input fields. |
|
(0006479)
kre (reporter)
2023-09-11 03:59
|
Now, that's done, something I have no idea about whatever (as in one
byte character locales the answers to these are irrelevant).
Do the results of each expansion or substitution get field split
separately, or is the field constructed from all the substitutions
and expansions first, then those parts which resulted from expansions
(whichever expansion the resulting bytes came from) interpreted for
field splitting
That is: assume we have a local with 2 byte characters (if you like
assume we're using UTF-8 and and a char set which is encoded as a 2 byte
sequence, or just some other char set with 2 byte encoding).
Then assume we have
X=$'\xXX'
Y-$'\xYY'
where neither XX nor YY as a byte is a valid character, but where the
two byte sequence XXYY is a valid character in the encoding. Let's
call that character 'C' (obviously it won't be, think more like alpha
or a-umlaut or something encoded as 2 bytes in UTF-8, or anything you
like in some other encoding).
Now if we have
IFS='...C...'
(what else is in IFS is irrelevant, except that as $X and $Y are not
characters, things would be unspecified if they were there, alone, so
assume they are not.)
If a field contains
foo${X}${Y}bar
what is the result supposed to be? If the results from the 2 expansions
are considered separately, then nothing will change, as there are no IFS
characters in either of those (there cannot be, as they are not characters
and IFS can't contain anything which isn't).
On the other hand, we we simply consider the result to be
foo$'\sXX\xYY'bar
with the part shown there inside the $'...' being the (unquoted) results of
the expansions, that is
fooCbar
and treat that for field splitting, then C, the results of the (2) expansions
is in IFS, so the result will be 2 fields, "foo" and "bar".
Which is supposed to happen? My algorithm simply finesses this point
(says nothing about it at all) which isn't good, but I have no idea what
answer we should be producing, I have no idea if any shell even allows or
attempts to deal with this kind if thing, so ....
Please be clear in any reply whether you are just saying what you believe
should happen (and in that case, please say why), or if you are indicating
what some implementation actually does with this (in that case, say which,
and ideally show a real worked example).
If we cannot produce some consensus on this, I'll simply add text to the
(draft proposed) spec saying that whether the byte sequences considered in
the loop must all come from the results of a single expansion, or can be
from the results of several expansions, is unspecified. |
|
(0006480)
kre (reporter)
2023-09-11 04:18
edited on: 2023-09-11 04:20
|
And finally, for today, my first impression of an answer to
the question (my question) in Note: 0006479 is that individual
expansions should be field split separately, and never combined
(so we never treat the example field as being fooCbar until after
field splitting has happened).
I suspect that's the best solution, as one can always do
NEW="${X}${Y}"
(where the quotes aren't really needed, as no field splitting or
pathname expansions happen there anyway) and then
foo${NEW}bar
if one is deliberately trying to compose characters, which then might
be a field splitting delimiter.
On the other hand, if we just allow adjacent expansions to run together,
then people would be continually needing to write
foo${X}''${Y}bar
just in case (as it would in this example) the results of the two
substitutions might just accidentally take the trailing bytes from $X
and the leading bytes from $Y and manage to concoct a character encoding
from them which just happens to be in IFS (all accidental).
While sticking the '' in that example would work (though is very ugly
and cumbersome) in this example to solve that, if the expansion were
just
${X}${Y}
it wouldn't be safe, as if both $X and $Y happen to be empty ('') the
results of that should be nothing (field deleted), whereas the results
of
${X}''${Y}
(after quote removal) would be an empty field. Without inventing new
mechanism, I'm unaware of any way of solving that one.
If we were to agree that this is correct (and by some chance it
happened that some shell actually works this way) then I would
change the current wording in the draft in Note: 0006478 from:
No such byte sequence shall contain data such that some bytes in the
sequence resulted from an expansion, and others did not.
to
No such byte sequence shall contain data such that some bytes in the
sequence resulted from an expansion, and others did not, or which
contains bytes resulting from the results of more that one expansion.
|
|
(0006483)
Don Cragun (manager)
2023-09-21 16:48
edited on: 2023-09-25 09:47
|
Replace XCU section 2.6.5 on issue 8 draft 3 P2476, L80478-80504 with:
After parameter expansion (Section 2.6.2), command substitution
(Section 2.6.3), and arithmetic expansion (Section 2.6.4), and
if the shell variable IFS [xref XCU 2.5.3] is set and its value is
not empty, or if the IFS variable is unset, the shell shall scan each
field containing results of expansions and substitutions that did not
occur in double-quotes for field splitting; zero, one or multiple fields
can result.
For the remainder of this section, any reference to the results of an
expansion, or results of expansions, shall be interpreted to mean the
results from one or more unquoted variable or arithmetic expansions,
or unquoted command substitutions.
If the IFS variable is set and has an empty string as its value, no field
splitting occurs. However if an input field which contained the results of
an expansion is entirely empty, it shall be removed. Note that this occurs
before quote removal, any input field that contains any quoting characters
can never be empty at this point. After the removal of any such
fields from the input, the possibly modified input field list becomes
the output.
Each input field is considered in sequence, first to last, with the
results of the algorithm described in this section causing output fields
to be generated, which remain in the same order as the input fields
from which they originated.
Fields which contain no results from expansions shall not be affected by
field splitting, and shall remain unaltered, simply moving from the
list of input fields to be next in the list of output fields.
In the remainder of this description, it is assumed that there is present
in the field at least one expansion result, this assumption will not be
restated. Field splitting only ever alters those parts of the field.
For the purposes of this section, the term "IFS white space" shall mean
any of the single byte characters <space>, <tab> or <newline> from the
Portable Character Set [xref XBD 6.1] which are present in the value of
the IFS variable, and perhaps other White Space characters. It is
implementation defined whether other White Space characters which appear
in the value of IFS are also considered as "IFS white space". The three
characters above specified as IFS white space characters are always IFS
white space, when they occur in the value of IFS, regardless of whether
they are White Space characters in any relevant locale. For other locale
specific White Space characters allowed by the implementation it is
unspecified whether the character is considered as IFS white space if it
is White Space at the time it is assigned to the IFS variable, or if it
is White Space at the time field splitting occurs (the locale may have
altered between those events).
If the IFS variable is unset, then for the purposes of this section, but
without altering the value of the variable, its value shall be considered
to contain the three single byte characters <space>, <tab> and <newline>
from the Portable Character Set, all of which are IFS white space characters.
The shell shall use the byte sequences that form the characters in the
value of the IFS variable as delimiters. Each of the characters <space>
<tab> and <newline> which appears in the value of IFS shall be a single
byte delimiter. The shell shall use these delimiters as field terminators
to split the results of expansions, along with other adjacent bytes, into
separate fields, as described below. Note that these delimiters terminate
a field, they do not, of themselves, cause a new field to start, subsequent
data bytes that are not from the results of an expansion, or that do not
form IFS white space characters are required for a new field to begin.
Note that the shell processes arbitrary bytes from the input fields, there is
no requirement that those bytes form valid characters.
If results of the algorithm are that no fields are delimited, that is,
if the input field is wholly empty or consists entirely of IFS white space,
the result shall be zero fields (rather than an empty field).
For the purposes of this section, when a field is said to be delimited,
the the candidate field, as generated below shall become an output field.
When the algorithm transforms a candidate into an output field it shall be
appended to the current list of output fields.
Each field containing the results from an expansion shall be processed
in order, intermixed with fields not containing the results of expansions,
processed as described above, as if as follows, examining bytes in the input
field, beginning to end:
Begin with an empty candidate field.
While the input is not empty...
When instructed to perform the next iteration of the loop, start
again with the test here, with the input as modified below.
Consider the leading remaining byte or byte sequence of the input.
No such byte sequence shall contain data such that some bytes in the
sequence resulted from an expansion, and others did not. If the byte
or sequence of bytes is:
<ol type="a">A byte (or sequence of bytes) in the input that did not result from an expansion:Append this byte (or sequence) to the candidate, and remove it from the input. Next iteration of the loop.
A byte sequence in the input which resulted from an expansion that does not form a character in IFS:Append the first byte of the sequence to the candidate, and remove that byte from the input. Next iteration of the loop.
A byte sequence in the input which resulted from an expansion that forms an IFS white space character:Remove that byte sequence from the input, consider the new leading input byte sequence, and repeat this step.
A byte sequence in the input that resulted from an expansion that forms an IFS character, which is not IFS white space:
Remove that byte sequence from the input, but note it was observed.
At this point, if the candidate is not empty, or if a sequence of bytes
representing an IFS character that is not IFS white space was seen at the final step, then a field is said to have been delimited, and the candidate
becomes an output field.
Empty (clear) the candidate, and perform the next iteration of the loop. Once the input is empty, the candidate becomes an output field if and only if it is not empty.
The ordered list of output fields so produced, which may be empty, replaces
the list of input fields.
|
|
(0006485)
kre (reporter)
2023-09-21 19:28
|
Aside from the obviously unintended formatting directive (ol type=...) that
appears where there almost certainly should be just a blank line, Note: 0006483
looks OK to me. I'm slightly surprised by the decision regarding the issue
in Note: 0006479 but that one was always so far outside my area of competence
that I can only trust that the correct decision was made.
Upon reflection, I'd probably add an xref to XBD 3.412 just after the (first)
use of "White Space" (soon after the xref to XBD 6.1) but that's not all that
important.
It would also be better, if it can be made to happen with the formatting tools
available, to (at least) eliminate the blank line separating the "While the
input is not empty..." and "When instructed..." paragraphs, so the "this test"
reference in the latter doesn't look to be referring to nothing, and also if
possible, indent that "When instructed..." bit further, so it doesn't align
itself with the steps to perform that follow, it isn't one of those, it is
just a "how to interpret what follows" directive, so I think would be better
if visually different from what follows. |
|
(0006487)
Don Cragun (manager)
2023-09-25 14:21
|
As noted in Note: 0006485 there are some parts of this resolution that need to be looked at again. There are also several other editorial issues that should be fixed. |
|
(0006488)
Don Cragun (manager)
2023-09-25 15:43
edited on: 2023-10-02 17:49
|
Replace XCU section 2.6.5 on issue 8 draft 3 P2476, L80478-80504 with:
After parameter expansion (Section 2.6.2), command substitution (Section 2.6.3), and arithmetic expansion (Section 2.6.4), and if the shell variable IFS [xref XCU 2.5.3] is set and its value is not empty, or if the IFS variable is unset, the shell shall scan each field containing results of expansions and substitutions that did not occur in double-quotes for field splitting; zero, one or multiple fields can result.
For the remainder of this section, any reference to the results of an expansion, or results of expansions, shall be interpreted to mean the results from one or more unquoted variable or arithmetic expansions, or unquoted command substitutions.
If the IFS variable is set and has an empty string as its value, no field splitting occurs. However if an input field which contained the results of an expansion is entirely empty, it shall be removed. Note that this occurs before quote removal; any input field that contains any quoting characters can never be empty at this point. After the removal of any such fields from the input, the possibly modified input field list becomes the output.
Each input field is considered in sequence, first to last, with the results of the algorithm described in this section causing output fields to be generated, which remain in the same order as the input fields from which they originated.
Fields which contain no results from expansions shall not be affected by field splitting, and shall remain unaltered, simply moving from the list of input fields to be next in the list of output fields.
In the remainder of this description, it is assumed that there is present in the field at least one expansion result; this assumption will not be restated. Field splitting only ever alters those parts of the field.
For the purposes of this section, the term "IFS white space" shall mean any of the white-space bytes [xref to XBD 3.412, 3.413, and 3.414] <space>, <tab> or <newline> from the Portable Character Set [xref XBD 6.1] which are present in the value of the IFS variable, and perhaps other white-space characters. It is implementation defined whether other white-space characters which appear in the value of IFS are also considered as "IFS white space". The three characters above specified as IFS white-space bytes are always IFS white space, when they occur in the value of IFS, regardless of whether they are white-space characters in any relevant locale. For other locale specific white-space characters allowed by the implementation it is unspecified whether the character is considered as IFS white space if it is white space at the time it is assigned to the IFS variable, or if it is white space at the time field splitting occurs (the locale may have changed between those events).
If the IFS variable is unset, then for the purposes of this section, but without altering the value of the variable, its value shall be considered to contain the three single byte characters <space>, <tab> and <newline> from the portable character set, all of which are IFS white-space characters.
The shell shall use the byte sequences that form the characters in the value of the IFS variable as delimiters. Each of the characters <space> <tab> and <newline> which appears in the value of IFS shall be a single byte delimiter. The shell shall use these delimiters as field terminators to split the results of expansions, along with other adjacent bytes, into separate fields, as described below. Note that these delimiters terminate a field; they do not, of themselves, cause a new field to start, subsequent bytes that are not from the results of an expansion, or that do not form IFS white-space characters are required for a new field to begin.
Note that the shell processes arbitrary bytes from the input fields; there is no requirement that those bytes form valid characters.
If results of the algorithm are that no fields are delimited, that is, if the input field is wholly empty or consists entirely of IFS white space, the result shall be zero fields (rather than an empty field).
For the purposes of this section, when a field is said to be delimited, then the candidate field, as generated below shall become an output field. When the algorithm transforms a candidate into an output field it shall be appended to the current list of output fields.
Each field containing the results from an expansion shall be processed in order, intermixed with fields not containing the results of expansions, processed as described above, as if as follows, examining bytes in the input field, from beginning to end:Begin with an empty candidate field and the input as specified above.
When instructed to start the next iteration of the loop, this is the start of the loop. While the input (as modified by earlier iterations of this loop) is not empty:Consider the leading remaining byte or byte sequence of the input. No such byte sequence shall contain data such that some bytes in the sequence resulted from an expansion, and others did not, or which contains bytes resulting from the results of more than one expansion. If the byte or sequence of bytes is:- A byte (or sequence of bytes) in the input that did not result from an expansion:
Append this byte (or sequence) to the candidate, and remove it from the input. Start the next iteration of the loop. - A byte sequence in the input which resulted from an expansion that does not form a character in IFS:
Append the first byte of the sequence to the candidate, and remove that byte from the input. Start the next iteration of the loop. - A byte sequence in the input which resulted from an expansion that forms an IFS white space character:
Remove that byte sequence from the input, consider the new leading input byte sequence, and repeat this step. - A byte sequence in the input that resulted from an expansion that forms an IFS character, which is not IFS white space:
Remove that byte sequence from the input, but note it was observed.
At this point, if the candidate is not empty, or if a sequence of bytes representing an IFS character that is not IFS white space was seen at step 4, then a field is said to have been delimited, and the candidate becomes an output field.
Empty (clear) the candidate, and start the next iteration of the loop. Once the input is empty, the candidate becomes an output field if and only if it is not empty. The ordered list of output fields so produced, which may be empty, replaces the list of input fields.
|
|
(0006490)
kre (reporter)
2023-09-26 13:14
|
Re: Note: 0006488
Just a couple of minor formatting issues remain, which I expect can be
fixed when all of this is converted to *roff for inclusion in the draft.
First, there's an unnecessary (unintended I'm sure) line break in the
9th paragraph (I think 9th) in the phrase "subsequent data bytes" where
the line break after 'subsequent" shouldn't be there.
Second, the whole bunch of paragraphs from "Begin with an empty ..."
down to and including "Once the input is empty ..." should be indented
slightly, retaining the relative (but absolute is immaterial) indentations
already existing. That is, that whole group of paras is the object of the
previous "Each field containing the results from an expansion shall be
processed in order, [...] from beginning to end:" and is describing the
processing of each field in turn.
On the other hand, the final paragraph ("The ordered list...") applies
after all fields have been processed, it isn't part of that "for each
field". Currently that distinction isn't obvious - even worse, the
excess vertical space before "Once the input is empty" might suggest that
it applies only after all fields are processed, which isn't the case,
a non-empty candidate becomes a field of its own if it exists after each
field has been processed. Eg: given default IFS, and S=' ' (a space),
then a${S}b c${S}d as 2 input fields makes 4 output fields, a b c d,
both "b" and "d" are what is in the candidate when the input (field) is
empty after performing the previous steps. |
|
(0006491)
Don Cragun (manager)
2023-09-29 02:28
|
The comments in Note: 0006490 were discussed during the 2023-09-28 conference call. We agreed with the suggestions and I agreed to update Note: 0006488 after the call concluded.
I fixed the unintended line break in the 9th paragraph (and also one in the 1st paragraph).
I got rid of several extraneous empty lines.
I think I have fixed the indentation problems. Please let me know if something else needs to be done. |
|
(0006492)
kre (reporter)
2023-09-29 08:59
|
I see no further issues with the edited Note: 0006488. Thanks. |
|
(0006495)
larryv (reporter)
2023-09-29 20:42
|
Some nitpicking on Note: 0006488:
- The text contains several comma splices, which are broadly considered to be incorrect in formal writing. The errant commas should be replaced with semicolons, or the sentences should be split into multiple sentences. The following suggestions generally do the former.
Change:Note that this occurs before quote removal, any input field that contains any quoting characters can never be empty at this point. to:Note that this occurs before quote removal; any input field that contains any quoting characters can never be empty at this point.
Change:In the remainder of this description, it is assumed that there is present in the field at least one expansion result, this assumption will not be restated. to:In the remainder of this description, it is assumed that there is present in the field at least one expansion result; this assumption will not be restated.
Change:Note that these delimiters terminate a field, they do not, of themselves, cause a new field to start, subsequent data bytes that are not from the results of an expansion, or that do not form IFS white-space characters are required for a new field to begin. to:Note that these delimiters terminate a field; they do not, of themselves, cause a new field to start. Subsequent data bytes that are not from the results of an expansion or that do not form IFS white-space characters are required for a new field to begin.
Change:Note that the shell processes arbitrary bytes from the input fields, there is no requirement that those bytes form valid characters. to:Note that the shell processes arbitrary bytes from the input fields; there is no requirement that those bytes form valid characters. - A typo:
Change:the the candidate field to:then the candidate field
|
|
(0006496)
stephane (reporter)
2023-10-01 16:00
|
Re: Note: 0006488
> The shell shall use the byte sequences that form the characters
> in the value of the IFS variable as delimiters. Each of the
> characters <space> <tab> and <newline> which appears in the value
> of IFS shall be a single byte delimiter. The shell shall use
> these delimiters as field terminators to split the results of
> expansions, along with other adjacent bytes, into separate fields,
> as described below. Note that these delimiters terminate a field,
> they do not, of themselves, cause a new field to start, subsequent
> data bytes that are not from the results of an expansion, or that
> do not form IFS white-space characters are required for a new
> field to begin.
The wording in the last sentence would imply that:
var='var ' sh -fc ' printf "[%s]\n" $var"" '
should output only
[var]
Instead of
[var]
[]
set -o noglob
IFS=:
for dir in $PATH''; do
dir=${dir:-.}
...
done
Is a well known idiom to loop over the components of $PATH without discarding the empty component in PATH=/bin:/usr/bin: for instance. |
|
(0006497)
kre (reporter)
2023-10-01 22:22
|
Re: Note: 0006495 - thank you, yes, all of those changes look good to me,
obviously at least the final one (no idea how that slipped through all of
the re-reading that has been happening - to make it easier, that's in the
paragraph which begins "For the purposes" which is about the 12th para I
think).
For the rest, those changes, or something similar, look to be an improvement.
One of the reasons that I never expect wording I write to survive intact, is
that I tend to write long run-on sentences, which can go on forever sometimes,
even though I know that's not good writing. Corrections by those better at
this than I am are always welcome.
Re Note: 0006496
No, there is no implication anything like you are suggesting.
That "last sentence" you mean includes (requoting it here)
subsequent data bytes that are not from the results of an expansion,
or that do not form IFS white-space characters are required for a new
field to begin.
which is the part of it I believe you're focussing on. Let me slightly
restate your example, inspired by the later PATH related example, just to
avoid there being spaces (any white space) here - just because that is hard
to display cleanly in text like this (and the PATH example shows that the
issue isn't in any way related to the IFS white space delimiter your original
example used).
var='var:' sh -fc 'IFS=:; printf "[%s]\n" $var"" '
After variable expansion (there are no arith erxpansions or command subs to
muddy the waters here) we have 3 initial fields in that printf command.
The first two are just the "printf" word, and the format string, neither of
which contain any expansion results, and so aren't touched by field splitting
(not that the note was suggesting anything different, just for completeness
here). So we have just the third field, which after variable expansion
will contain the following byte sequence (into which I have inserted white
space just for this note, to make things more obvious - this is why I needed
to change the example from the original - just for easier visualisation in
text like this):
v a r : " "
in which the first 4 bytes are the results of the variable expansion, and
the final 2 are not.
Now we do field splitting, the ':' in this case is an IFS char (in the original
a space was used, which is also IFS white space, but in this example that makes
no difference) when we see that the candidate contains "var" so is not empty,
and becomes an output field.
Now we're at the point where the sentence in question is thought to apply
subsequent data bytes that are not from the results of an expansion,
however does apply. there are still those two " (double quote) characters
present, which are not from the results of an expansion, and hence satisfy
that condition, which then allow a new candidate field to start. That one
has no delimiter, so we reach the end of the input with the candidate
containing two double quote characters, which is not empty, hence forms
another output field - so now after field splitting we have (in this example)
4 fields
printf
"[%s]\n"
var
""
Subsequently (after pathname expansion, which is disabled in this example,
so does nothing, not that there are any unquoted glob magic chars to invoke
it anyway) we will do quote removal, which never alters the number of fields
simply removes the quoting chars, still leaving the 4 fields, with all the "
characters removed (so the final field will be a null string).
That whole sentence though is redundant with the actual algorithm, which
shows how that works - it was added merely as added reinforcement that if
the input had not had those two trailing quote characters, then there would
not be an extra field produced, just 3; which in the case you gave was never
in doubt, IFS white space is always handled like that, but in the case with
a ':' delimiter, many people assume there should be an empty field produced
as the IFS are "field separators" implied by the 'S' in the var name - the
whole sentence can (and probably should) be removed if it is thought to
potentially cause more confusion than it removes (but this example isn't
really a case of that I think - the problem here was that the quote characters
weren't considered as being in the input, as if they'd already been replaced
by a null string.
The sentence earlier in the proposed spec (as it currently appears in
Note: 0006488):
Note that this occurs before quote removal, any input field that contains
any quoting characters can never be empty at this point.
was another sentence I feared might be considered redundant, as other text
in the standard makes that clear already - it should not really need to be
restated here - but I have found that some people find it difficult to
comprehend this part of the abstract algorithms in the standard - particularly
when real implementations rarely act quite like this, so I included it as an
additional reinforcement of this issue. True, it is in a paragraph that's
primarily concerned with what do do when IFS='' applies - but whether or not
quotes have already been removed from the input fields cannot (and does not)
differ in that case, so I considered that just making sure that any reader
has in mind that quote characters (in the abstract implementation) are still
present, unaltered, in the fields in this one place would be sufficient.
I'll leave it for others to decide whether more, or less, or adjusted,
forms of these two redundant sentences should remain in the final version. |
|
(0006498)
stephane (reporter)
2023-10-02 06:27
|
OK, I see you PoV now. Then the thing I would object to is the use of "*data* byte". For me, and I'd expect most people reading the text, those quotes in $var"" are syntax, not data.
(I've myself always disliked the use of "quote removal" in the POSIX specification as that's a detail of implementation, while the spec should focus on specifying the language, though I acknowledge it's probably too late to change that).
Just a comment on:
> but in the case with
a ':' delimiter, many people assume there should be an empty field produced
as the IFS are "field separators" implied by the 'S' in the var name
Note that it used to be the case in most shells, including pdksh, yash, zsh (still the case in zsh) and I believe ash or at least some variant after they switched from Bourne-like to POSIX-like behaviour and they later begrudgingly changed to align with Korn/POSIX.
In the Bourne shell, there was no such question as all characters were treated as Korn's IFS-whitespace characters, that is ::foo:::bar::: was split into foo and bar only.
Korn in effect changed IFS from being an Internal Field Separator to being an Expansion Field Terminator, without ever documenting it so I'm not even sure the change (from separator to terminator) was intentional.
That gets in the way for "read" where one can't use that "appending ''" work-around.
Where IFS=: read -r header rest fails to read the "rest" properly if it contains one and only one ":" and it's at the end of the line. |
|
(0006499)
kre (reporter)
2023-10-02 12:25
|
Re Note: 0006498 those quote characters are data bytes as far as field
splitting is concerned. The abstract model used by the standard has that
as a feature throughout.
I will admit to also not much liking it when I first saw it - and I suspect
makes some things far more difficult to describe than they ought be. But
it is very clear that in the model used throughout, quoting characters are
retained in the input until quote removal removes them.
The shell also needs to keep track of which quote chars were in the original
input (any resulting from expansions are just data) and which characters were
quoted, or result from a quoted expansion, which you'd think would be enough,
but just doing that is not the model.
To field splitting they're irrelevant, except if they're quoting characters
they can't have resulted from an expansion. They're just data.
If you have any suggestions for wording changes to make any of this clearer
please make them here, but don't bother attempting to alter the parsing model
that the standard presumes, that's far to big a change to even contemplate
at the minute.
As fas as the history of separators vs terminators, that's all in the past
and no longer relevant to anything - all shells (except zsh in --emuate sh
mode, in its native mode, zsh doesn't appear to do field splitting at all)
agree on how this works now - it isn't going to alter, so speculating upon
how we got here isn't serving any useful purpose. Even mksh, which seems
to be the one shell (other than the ancient broken pkksh in NetBSD_ where
field splitting is broken, mostly seems to agree on this.
Thanks for mentioning read though - I have known for a while that the
description of how field splitting is done for read also needs corrections,
but I never actually remembered to do anything about it when I was in a
position to send a defect report for it. I will do that now, it should
be treated as related to this bug, and the two processed together.
If anyone knows of anywhere else in the standard, where the field splitting
algorithm is used (I don't mean places where field splitting is said to apply,
or not apply, those should all be fine as is) in a way similar to how read
uses it, please let us know, so if more corrections/updates are needed, those
places can also be corrected now. |
|
(0006508)
Don Cragun (manager)
2023-10-02 17:57
|
The comments in Note: 0006495, Note: 0006496, Note: 0006497, Note: 0006498, and Note: 0006499 were discussed during the 2023-10-02 conference call. All of the changes suggested in Note: 0006495 were accepted. We also accepted the suggested change from "data bytes" to "bytes". All of these changes have be updated in Note: 0006488.
This bug remains resolved with the changes now specified in Note: 0006488. |
|
(0006525)
geoffclare (manager)
2023-10-10 09:31
|
When applying this bug I made some minor editorial wording changes, such as use of "shall" instead of present tense in some places. |
|
ifs [^] (675 bytes) 2023-03-31 01:55
