Re: [Pan-devel] The Blue Sky Becomes Clear - part 01: Basic Thoughts - revision 00

[Jeffrey Stedfast]

On Thu, 2003-04-03 at 11:03, Charles Kerr wrote:
> On Thu, Apr 03, 2003 at 08:33:44PM +0530, Haran Shivanan wrote:
> > Module 'Bas Mevissen' core dumped on Wed, 2 Apr 2003 with the following 
> > output
> > 
> > > * Article database: more fine tuning of the article cache, purging and 
> > > the 
> > How about a new way of storing the cached articles?
> > Currently, each article is stored in a separate file, which means the 
> > directory is littered with hundreds or thousands of files with each 
> > burning a (typically) 4KB inode (for linux installations).
> > I know the mbox format was already tried and thrown away but why?
> > It would not only be more effecient to store articles in this format, but 
> > would make it easier to use from another reader or more appropriately, 
> > read articles from an existing mbox. I think the main argument against the 
> > mbox format was the cost in parsing the flat file and deleting messages 
> > from it. But correct me if I'm wrong but isn't this the format used by 
> > many other news readers? (definitely Netscape and possible Mozilla use it)
> > I havn't extensively used them but were there any performance problems in 
> > using mbox?
> > 
> > http://www.jwz.org/doc/mailsum.html
> 
> I don't have time to respond to this properly right now,
> but I will point out that jwz's mail summary files statement
> is six years old.  I'd counter with this:
> 
> http://www.courier-mta.org/mbox-vs-maildir/
> 
> Dan Bernstein of qmail fame wrote a good description of the
> mbox problems addressed by maildir, but I can't find the link
> right now and have to run to a meeting.

I'm not going to argue for or against mbox, but I will say this:

Evolution uses a variation of jwz's mail-summary file w/ mbox and
maildir backends, and it is wicked fast. Faster than doing maildir
alone, even. Reason is that it takes parsing completely out of the
equation... you just load pre-parsed data back into memory.

however, being mostly an outsider to the pan internals (other than
gmime... which can parse an mbox exceptionally fast), I can't argue that
mbox is really the right way to go for pan.

in fact, I might even say mbox is definetely the wrong way to go for a
cache simply because the overhead of expunging an mbox is so high. since
a cache is meant to expire articles either based on age and/or
cache-size, this expunging could be going on a fair bit?

then again... I suppose you could just mark messages as deleted and do a
lazy expunge (perhaps at shutdown, or during periods of long idleness?
you might also just force an expunge once the cache size gets larger
than X bytes over the user's "max cache size" setting?)

I dunno how expunging is done currently, probably also "lazy" but since
it just has to do an unlink(), no need to worry about the overhead.

another thought... maildir was brought up here, but I'm not sure pan
really uses true maildir? if not, perhaps it can be modified to use a
more true maildir format cur/ new/ tmp/ etc so that the original poster
can read with another client such as mutt? *shrug*

Those are just some thoughts... again, I am not arguing for or against
mbox - just something to think about.

Jeff

> 
> 
> _______________________________________________
> Pan-devel mailing list
> address@hidden
> http://mail.nongnu.org/mailman/listinfo/pan-devel
-- 
Jeffrey Stedfast
Evolution Hacker - Ximian, Inc.
address@hidden  - www.ximian.com