Re: [Pan-devel] musings on memory consumption

[K. Haley]

Hi Anatoly,

I hope your still reading this list.  I'm not one of the devs but here 
are my thoughts.

Anatoly Vorobey wrote:

> I ended up with a patch that allows me to browse a 1-million-headers 
> newsgroup comfortably on my machine, which is more or less what I 
> needed. Basically, I use refcounted strings and normalised subjects. 
> There's a new string type, RString, which stores unique strings only 
> once by using a global hash table and a refcount field to keep track of 
> how many times the string was referenced, allowing it to be freed when 
> the refcount drops to 0. 
>  
Sounds usefull.  I can certainly make use of it in the db code I've been 
working on for Pan.  This way I won't have to implement something 
similar myself.

> After this, starting Pan and loading a 1 million headers newsgroup 
> results in about 340Mb memory used, which is tolerable for me. The 
> slowest reactions now come from the GTK header pane, presumably because
> it finds it hard to cope with such a large tree. The widget spends 
> around 10 seconds initialising or freeing the entire header pane (when 
> entering/leaving the group). I'm not sure whether dynamic feeding of 
> data into the widget, along the lines Evan Martin suggested in a recent 
> message, would speed that up.
>  
Pan currently uses GtkCTree for the article list.  I don't know if it 
can pull in data as needed or if it requires its own copy of all 
string.  The code will be converted to GtkTreeView as part of the 
rewrite but it currently has a performance problem.  After adding a new 
GtkTreeModel it will display the visible rows then set about validating 
every row in the model.  Even on my small group list, only 43000 rows, 
it makes my system un-usable for about 20 seconds.  Here 
http://bugzilla.gnome.org/show_bug.cgi?id=116642 is a bug report on what 
appears to be the same issue.  It also mentions the proper fix but 
no-one has coded it yet.

> Anyway, I understand that the work I've done might not be useful to 
> official Pan development if it's been decided to focus on moving to a DB 
> backend. I did it primarily for myself, to scratch my own private itch. 
> If, however, there's interest from project maintainers, and they think 
> it might be considered for inclusion, or if other people want to try it 
> out, do let me know; I can find some time to clean it up, remove 
> debugging junk, test it some more and put it up for download.
By all means post it.  I would suggest opening an enhancement request on 
bugzilla and attaching two thing.  First a zip containing just your 
RString files for those of us attempting our own back end rewrites.  
Second a patch against cvs with all your RString changes, including the 
RString files.  That way people that want to try it can.  It is posible 
that it could make it into cvs as a temporary solution given how long 
the rewrite is likely to take.

signature.asc
Description: OpenPGP digital signature