[Pan-devel] Re: newsrc with DB

[Duncan]

K. Haley posted <address@hidden>, excerpted below,
 on Thu, 10 Jun 2004 01:08:37 -0600:

>>ok, exception to #2 is that using the first 8 bytes of the md5sum'd
>>(canonical) message-id might be just as good. plus it saves a ton more
>>memory :-)
>>  
>>
> Good idea, although we might need to use the full 16 bytes just to be 
> safe.  Since the article table will hold all the article summaries for 
> all groups there will be more than a few users with several MILLION 
> entries in it.

If you are going to start hashing millions of msg-ids, it might be a
/very/ good idea to go read what Hans Reiser has to say about that, based
on his work with ReiserFS.  There are at least three hash options one can
use with ReiserFS, one that it says not to use, one that is the default
for normal use, and one that he says could be used if the default version
produces hash collisions (which he says shouldn't normally happen until
one gets well into the millions of files -- per directory -- range).  I
haven't bothered to really grok the stuff, because it's not been something
I needed to do, but he's had VAST amounts of experience with such things,
and reading his articles and code should produce numerous insights into
CPU efficiency tricks and hash collision dangers with millions of items.

Of course, I think it'd be really neat to create the PAN db such that it
could use the capacities of ReiserFS Four directly, but we'd need at least
a choice, since ReiserFS isn't available for MSWormOS, AFAIK.  (Tho I
wouldn't be surprised if /someone/ hadn't contracted /some/ sort of work
in that area, perhaps even MS itself, for Longhorn plus <nibble-int>.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"They that can give up essential liberty to obtain a little
temporary safety, deserve neither liberty nor safety." --
Benjamin Franklin