Re: Bash-5.2-rc3 available

[Kerin Millar]

On Wed, 31 Aug 2022 10:55:58 -0400
Chet Ramey <chet.ramey@case.edu> wrote:

> On 8/29/22 12:28 PM, Kerin Millar wrote:
> 
> >>> For extglob to be arbitrarily enabled in either context is unexpected 
> >>> (and undesirable).
> >>
> >> The conditional command behavior is compatible with ksh93, and has been
> >> this way for nearly twenty years. It's documented to behave that way. One
> >> would think people are used to it by now.
> > 
> > I am aware of how the == and != operators work, and that the pattern on the 
> > right hand side is always treated as an extglob in the shell performing the 
> > test [*]. I don't see what bearing that has on my report or its wording.
> 
> OK. You wrote "either context" above. There are two contexts in which 
> extglob is enabled internally: conditional commands and (now) command
> substitutions. The former has been the case for a long time; the latter
> is new.
> 
> How am I supposed to interpret "either context?" A reasonable person could
> assume that you're objecting to established behavior.

I disagree. The sample clearly demonstrated extglob unexpectedly being turned 
on in two contexts, that of the initial shell and that of the subshell. 
Nevertheless, I shall attempt to be more specific next time.

> 
>   Might I enquire as to what it is that I am supposed to be used to on 
> account of the passage of those twenty years? Certainly not for a command 
> substitution on the right hand-side of the == operator to implicitly 
> perform the equivalent of shopt -s extglob within the implied subshell 
> without so much as a by-your-leave on the part of the programmer, much less 
> doing so in the shell that specified the command substitution.
> 
> 
> Wow, that was quite a sentence. In any event, you should report it, as you
> did.
> 
> > 
> > For the benefit of anyone else reading, the following amounts to an error 
> > in bash 5.1.16, as it should.
> > 
> > $ shopt extglob
> > extglob             off
> > $ [[ '' == $( : !() ) ]]
> > bash: command substitution: line 3: syntax error near unexpected token `('
> > bash: command substitution: line 3: ` : !() )'
> 
> It will produce exactly the same error in bash-5.2. On the other hand,

Very good. In the meantime, I tried -rc2 and saw that it behaves as expected 
there.

> there are valid constructs that bash-5.1 did not handle, and no amount of
> coercion could make it do so. I included one in a previous message thread.

Interesting. I will look for this thread.

> 
> The ad-hoc comsub parser in previous versions didn't understand extglob
> patterns at all, period, whether or not extglob was enabled. That resulted
> in it accepting bad code, invalid constructs that later resulted in errors,
> and valid patterns that were later accepted when the command was executed,
> as in a shell function definition. The current compatibility mode behavior
> is a concession to that previous reality. The problem is that neither you
> nor I know how many instances exist.

Fair enough.

> 
> In this case, the backwards compatibility setting doesn't matter, of course --
> parsing the word on the rhs of the ==/!= operator forces extglob on, and
> the command substitution parsing inherits that. That is not new behavior.

Indeed it is not. I find "forces extglob on" to be a loaded phrase, however.

$ shopt extglob
extglob         off 
$ [[ a == @(a|b) ]] # the rhs is treated as if extglob were enabled
$ shopt extglob
extglob         off # I wouldn't refer to this outcome as extglob being forced 
on; it's still off

In my estimation, it makes more sense to say that the rhs of ==/!== is taken to 
be an extglob, irrespective of the state of the extglob shopt. Indeed, the 
manual states that it behaves "[...] as if the extglob shell option were 
enabled", which makes perfect sense. If it were written as "[...] forcing the 
extglob option on" instead, it would be misleading, wouldn't it? On the other 
hand, to say that a comsub as the rhs of ==/!= forces extglob on in 5.2-rc3 
rather captures the essence of my report.

> 
> 
> > I understand that you are striving to do right by a majority of your users 
> > at all times. The reported issue was a consequence of a regression in a 
> > late-stage release candidate introduced by a new backward compatibility 
> > feature that I had not elected to enable. 
> 
> Yes, it's good that you reported it. There will probably be another release
> candidate as a result.
> 
> >>
> >> I understand from Sam James that Gentoo globally forces the compatibility
> >> mode to 5.0. That, too, is a choice.
> > 
> > Yes, that's right - subject to the "EAPI" level defined by a given ebuild.
> 
> So your users, or those who prepare ebuilds, have made a decision to set
> that. One of the consequences of that choice is this undesirable (as you
> put it) behavior of having extglob enabled during command substitution
> parsing.

Allow me attempt to clear this up. Taking [[ '' = $(shopt extglob >&2) ]] as an 
example, the state of the extglob option was that of being enabled at the point 
that the code within the command substitution is executed by the subshell. My 
first contention was that this should never happen and that it qualifies as a 
bug, no matter whether it be a consequence of overhauling the parser internals 
and/or the introduction of a new backward compatibility feature. I simply do 
not expect to see extglob be turned on during 'runtime' without having 
requested that it be on. I stand by my definition of this as being undesirable. 
I don't think that any prior release does this. If this behaviour were to make 
it into a final 5.2 release in any capacity, I would say with reasonable 
certainty that it would be a backwards compatibility predicament in the making 
and oppose it in the strongest possible terms. However, I think that we are in 
agreement that it is a bug.

My second contention was that it may not have been worth adding a backward 
compatibility feature that further affects the behaviour of the parser beyond 
the changes already made during the 5.2 development cycle. Such is merely my 
opinion. As long as the intended effect is only to relax comsub parsing - and 
not to proceed to affect the execution thereof - it constitutes a perfectly 
valid development decision, even though I may not be for it.

Regarding the consequences of decisions, this new feature will probably not end 
up benefiting Gentoo. There was one instance of an extglob being erroneously 
used for pathname expansion within a comsub, which was rectified on the 27th 
July. That aside, I'm not keen on the practice of setting BASH_COMPAT. Every 
new major release of bash tends to be accompanied by some issue that needs 
attending to, yet is invariably trivial in nature [1]. As you pointed out, 
backward compatibility features presently exist so as to buy time to fix 
scripts, rather than be a cast-iron guarantee of forward compatibility.

> 
>   Yet, I can't shake the feeling I had from the outset that, in this 
> particular instance, the compromise may simply not be worth it.
> 
> We'll see. There may not be any impact. On the other hand, I can't shake my
> feeling that there are a lot of scripts out there doing this because it
> "just worked" before.

Granted, that may be. Given that BASH_COMPAT usage seems to be quite rare in 
the wild, time will tell.

[1] For 5.1, it was https://bugs.gentoo.org/show_bug.cgi?id=bash-5.1, which 
didn't require much effort either.

-- 
Kerin Millar