[Pan-devel] Re: Want to fix memory consumption issues

[Duncan]

Calin A. Culianu posted
<address@hidden>, excerpted
below,  on Sat, 29 May 2004 18:43:48 -0500:

> 
> I really really want to fix the memory consumption issues in Pan. Namely,
> the problem that loading articles from groups with large numbers of
> headers (like > 500,000)  tends to bring most boxes to a crawl, because
> Pan ends up eating tons of memory.  It seems like roughly 1kb per article
> header!
> 
> So if I wanted to experiment with changing the memory needs of Pan (a
> change that might be very invasive actually..).. What are the good places
> to look?  One obvious place to optimize for memory use is changing struct
> Article or the way that articles get loaded.  Another place might be the
> gtk widget used to display the header pane...
> 
> How _does_ the header pane get loaded (I'm blurry-eyed to read any more of
> the sources now..)?  Ideally one would load text (from the struct Article
> pointers) into it piecemeal as the user scrolls through the groups.  Is
> that what happens.. or is the gtk widget for the header pane populated all
> at once with possibly millions of articles?

I'm not a source groker myself, but the consensus has always been that
it's the GTK widgets that are the performance and memory hogs.

The trouble as I understand it is that PAN uses them as they were never
intended to be used, forcing them to scale to entry-counts they were never
intended to handle.  The problem is exacerbated by PAN using what is
primarily a GUI widget that happens to have minor data-widget
capabilities, as a data-widget that happens to be a GUI-widget as
well.  This problem should to a large extent go away when PAN switches to
the sqlite back-end, as has been planned for some time.  A couple months
ago, there was some discussion here from a volunteer with some database
programming experience that wished to help there, but I believe the
discussion was taken to private e-mail and I haven't heard if he lost
interest or if it's coming along nicely and is just waiting on Charles to
get some time to fit it into mainline CVS or what.

Anyway, I believe that's the big place to look, right now, as the easy
performance enhancements have already been done with the existing setup,
and additional work would likely be dead-ended when the switch IS made. 
That database back-end work seems to be more and more the thing holding up
additional large-scale changes, so it needs to be tackled by someone, and
now is as good a time as any, since as Charles mentioned in his replies to
the other guy b4 it went off-list, he doesn't have a lot of time for
other PAN work right now, and PAN is fairly stable where it is anyway.

I've more than once gotten the feeling, however, that Charles is a bit
hesitant to make the switch himself, because he has no experience in the
area, and it's going to be a big enough change on its own, even if someone
with db experience helps.  If you have it, that's IMO DEFINITELY help
that's needed.  If you don't, perhaps now's the time to get it, because
that db work is IMO the single biggest thing holding back PAN, at this
point.

Look in the archives for the previous discussion, and go from there, would
be my suggestion.

-- 
Duncan - List replies preferred.   No HTML msgs.
"They that can give up essential liberty to obtain a little
temporary safety, deserve neither liberty nor safety." --
Benjamin Franklin