Re: Some experience with the igc branch

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  ofv@wanadoo.es,  emacs-devel@gnu.org,
>   eller.helmut@gmail.com,  acorallo@gnu.org
> Date: Wed, 25 Dec 2024 06:23:49 +0100
> 
> > If they don't have separate stacks, and we interrupt MPS, the signal
> > handler cannot look at any MPS-modifiable memory (including roots, which
> > may be in an inconsistent state mid-GC), ever.  This includes the
> > specpdl.  We can't write to MPS-known memory, ever.  This includes any
> > area we might want to copy the backtrace or specpdl to.
> 
> And I don't think that's right :-). It's completely right that in the
> SIGPROF handler everything can be inconsistent. That's true both for MPS
> and Emacs. For example, the bindings stack (specpdl) may be inconsistent
> when SIGPROF arrives.

Theoretically, maybe.  But in practice, you'd need to identify the
code which manipulates specpdl that could have specpdl in inconsistent
state if interrupted at some opportune point.  Can you identify such
places in the code?

> Literally everything we do in the SIGPROF runs the
> risk of encountering inconsistencies.

Only if we interrupt code which leaves the global state inconsistent,
and if what the SIGPROF handler does involves accessing those
potentially-inconsistent data.

> I think that's already true for the old GC. There is nothing
> guaranteeing that the contents of the binding stack is consistent, for
> example. But we get away with it well enough that the profiler is
> useful.

With the old GC, we have special code to deal with this:

  /* Signal handler for sampling profiler.  */

  static void
  add_sample (struct profiler_log *plog, EMACS_INT count)
  {
    if (EQ (backtrace_top_function (), QAutomatic_GC)) /* bug#60237 */
      /* Special case the time-count inside GC because the hash-table
         code is not prepared to be used while the GC is running.
         More specifically it uses ASIZE at many places where it does
         not expect the ARRAY_MARK_FLAG to be set.  We could try and
         harden the hash-table code, but it doesn't seem worth the
         effort.  */
      plog->gc_count = saturated_add (plog->gc_count, count);

So all we need is for backtrace_top_function to be safe when SIGPROF
arrives while we are in GC.  Are you saying backtrace_top_function is
unsafe in that case?

> With MPS, from my POV, the situation is pretty similar. Try to get away
> with it by not triggering MPS while in a state that we must assume is
> inconsistent.

The difference with MPS is that the old GC is synchronous with the
Lisp machine, so it couldn't possibly start while we are modifying
specpdl.  That is no longer true with MPS, AFAIU, because MPS could
start GC asynchronously.

> >> The SIGPROF handler does two things: (1) get the current backtrace,
> >> which does not trip on memory barriers, and
> >
> > Even if the specpdl were an ambiguous root, we'd be making very
> > permanent and far-reaching assumptions about how MPS handles such roots
> > if we assumed that we could even look at such roots during GC.  This
> > goes doubly for assuming that we can extract references to
> > ambiguously-rooted objects and put them into other areas of MPS-visible
> > memory.  Even if this worked perfectly with current MPS on all
> > platforms, it would still be unreasonable for us to rely on such
> > implementation details.
> >
> > We can't do (1).
> 
> I disagree, abviously :-)
> 
> For me, it's not about a theoretical or even practical solution that
> somehow ensures a consistent state in MPS, or some future changes in MPS
> or something. It's about getting away with what we do in the profiler
> _now_, as we do with the old GC. which is already seeing potentially
> inconsistent state in Emacs' memory.

See above: there's a difference.  So I would really like to hear why
you think accessing specpdl from a SIGPROF handler in an igc build is
safe.

> I think the _now_ is also important. From my POV, we could discuss
> better solutions later.

If you are right in your conclusions, certainly.