[lmi] UTF-8-ization [Was: PATCH: Switch to using std::filesystem]

[Greg Chicares]

On 4/29/21 5:08 PM, Vadim Zeitlin wrote:
[...]
>  Concerning UTF-8-ization, I'll make another patch soon adding plenty of
> missing conversions to/from UTF-8. I know that you don't care much about
> supporting non-ASCII characters (unless it has changed since the last time
> we've discussed it),

Today, lmi source files are restricted to something like
ISO/IEC 8859, which is enough for English but not French
(my sister eats eggs, e.g.) and certainly not APL. I can't
even write '∞', which would be very handy.

I think we should lift that restriction now, at least for
comments. I'm not sure we should lift it for string
literals as long as we're still using toupper() etc.

> but I think we really need to handle them correctly
> even when not dealing with foreign languages because they very commonly
> occur even in English texts nowadays (e.g. they do appear regularly in
> this mailing list archives) and we really don't want to lose the entire
> string contents just because it contains a "smart quote" or something like
> this in the middle (but this is what happens now).

I can't paste gcc warnings into lmi files that enforce
the ISO-8859 restriction, because of its `quotes'.

My cell phone erases the entire contents of text messages
that contain emojis.

>  In fact, my goal is to define wxNO_IMPLICIT_WXSTRING_ENCODING for lmi
> build to ensure that an explicit encoding is always specified when
> converting between Unicode and narrow strings (and, of course, this
> encoding will always be UTF-8 in lmi). Actually enforcing this will have to
> wait for the next wx update, but I can already make the required changes to
> prepare for it.

Sounds like a good idea to prepare now.