[Pan-devel] More database thoughts

[Tom]

Hi,

I'm the one that brought up the database backend back in March.  I'm
sorry that I accidentally missed the recent discussions until now.
Basically what happened in March was that I mentioned some ideas, and
Charles said he had already started on it, and had some notes he was
going to post. I didn't want to duplicate his work, so I waited for his
notes, never saw them, and decided to just drop the idea until Charles
seemed to have more time. That said, I'm glad others are working on it.

I like Calin's incremental approach. If you add the DB stuff to what's
already there, it makes it easy to cross-check for debugging purposes.
Also, unless you're confident that SQLite will never corrupt the DB,
having the files as a backup is an insurance factor. In any case, adding
a "(re)build DB" menu option someplace might be a good starting point. 

It looks to me that the Article structure is a big memory user. The
thing is, you rarely if ever display anything other than part 0 or 1, so
to me it makes sense to only keep the part 0/1 in memory, and retrieve
from DB/display the others on an as-needed basis. Correct me if I'm
wrong, but I suspect that few (text or binary) groups would have more
than 100,000 "unique" subjects (part 0/1). It seems to me that trying to
truncate the (xx/yy) from the subject string would be a small saving by
comparison. 

Tweaking question: what commit interval are you using? I would assume
that somewhere in the 100-10,000 area would be ideal.

Anyway, I'm glad that someone is doing something with this.