[Pan-devel] Re: Move to database back-end

[Duncan]

Tom posted <address@hidden>, excerpted
below,  on Fri, 05 Mar 2004 20:15:51 -0500:

> I have seen occasional mentions on the pan-users list about plans to move
> to a database back-end. I have some database experience, so I thought I'd
> find out what the status of that work is, and see if I could be of help.

> Questions:
> 1. First, is this the right place to raise these questions? The list seems
> to be fairly low in traffic.
> 2. How much of this work is already done? 
> 3. What db will be used? I thought Charles had mentioned it at some point
> on pan_users, but I couldn't find it now.
> 
> Oh, and who am I? I mostly lurk, but I have C [] and database experience
> [.] I'm a newbie to Unix/Linux development[.]

Charles or Chris will be able to answer in more detail from the PAN
development angle.  I'm just a list groupie, more than anything, who
understands just enough about development to get what's being discussed,
but I can at least fill you in on the list history on the topic..

1. Right place to raise these questions? Yes. The list IS indeed generally
low in traffic, but that's mainly because Charles and Chris are the only
major developers currently and they can as well converse privately.  There
are patch submitters and packagers around as well, and the occasional
thread related to that, but one gets the impression that too is mostly
handled thru Bugzilla and private correspondence, as it's generally
trivial stuff that wouldn't need public list coordination.

That said, this IS the place for such as this, as it has little to do with
stuff "users" would be interested in at this point, so should be here
rather than on users.

2. What's already done?  I don't know the details, but it appears to be
between the "basic research into feasibility and options" stage, and "code
generally nailed down" stage.  From Charles' remarks, it appears he's
settled on the tool to be used (see below), absent strong objections from
someone willing to help with it, anyway, and that he has a good idea of
how it would work and the flexibility and features it would make possible,
and has put some time into database record fields and potential layout,
but I've seen no hint that he has begun a code framework for it or
anything.

3. Tool of choice?  Charles has mentioned SQLite, in the context of
integrating it as a library much as PAN does with gnet for the network
side.  I know little about it, but from comments he's made, SQLite is a
fairly light library (again, similar to gnet) that would allow PAN to
borrow its more robust data implementation, while not requiring that users
install an entire database application such as MySQL, just to support PAN.
An additional advantage, however, is that SQLite is MySQL compatible, so
users that wanted to expand on PAN's abilities with full database
functionality could do so using MySQL.  That would lend PAN some serious
flexibility in terms of potential third party extensions and tie-in apps.

Some additional comments, in terms of current solutions as implemented
elsewhere and the types of things this would allow PAN or third party
data-base integrated software as extensions to PAN, to do (tho if you have
experience in the database space, you could be well ahead of me here)..

1.  The current "reference" in terms of binary news harvesters is BNR2,
aka Binary News Reaper  It is developed on Borland Delphi/Kylix, thus,
is usable on both Linux and MSWormOS, but obviously, isn't as open as
a non-proprietary-ware compilable solution would be.

Among its features is the ability to d/l from multiple servers, tracking
messages between them as PAN does.  However, that's just the beginning,
as it integrates servers rather than treating them separately as PAN
currently does.  It does this with integrated "virtual servers" and
"virtual groups". 

A virtual server combines a number of physical servers, each of which has
a priority and allowed connections setting.   Within a virtual server,
the user sees all the posts to the active group on all the physical
servers, only combined into a single list as if it was one server.  The
user then doesn't have to care about which physical server the post is
downloading from, as the app manages all that in the background, grabbing
according to priority for example all that can be from the user's ISP news
server, then from the "cheap" NSP, then from the expensive one that has
excellent completion but at bandwidth prices to expensive to grab anything
but the parts not available elsewhere from them.

A virtual newsgroup combines several physical newsgroups in the same way. 
If the virtual group is within a virtual server, the app will obviously
have to accurately track not only multiple servers and multiple groups,
but potentially different groups on each server, all the while keeping
priorities straight so the user doesn't get surprised by the bill from the
expensive server when the app screws up and d/ls everything from it.

Obviously, tracking all this is exactly what a database is good at.  In
fact, I'm not sure how BNR2 handles it, whether it depends on Delphi/Kylex
tools and libraries or whether it manages its own (I've never used Delphi
myself, so..), but whatever it does, it definitely isn't as robust as it
COULD be, because from the posts I've read of users, many/most of them are
quite familiar with the procedure of deleting the database files and
letting BNR2 rebuild them due to corruption.  This could be PAN's
opportunity to out-compete this competitor, once it starts playing in the
same competitive space.

A fairly regular request is to support this sort of thing in PAN, and
with a name like the "Pimp-Ass Newsreader", PAN obviously has pretensions
in that direction.  However, the just-as-regular and absolutely
reasonable answer, to this point, has always been "after we institute the
database backend." 

That's at least one area where PAN could make good use of the database
backend itself, and is inline with PAN's ultimate destiny as the "Pimp-Ass
Newsreader" to beat ALL newsreaders!  (Note that BNR2 doesn't handle text
well.  If PAN can match it in binary handling while handling text as well,
*PAN* should be able to lay claim to that "reference" title that now
belongs to BNR2, particularly since PAN already works on both MSWormOS and
Linux, PLUS other Unix type platforms, INCLUDING OSX, now.  Thus, if it
can do that, it WILL be beating or at minimum matching ALL newsreaders,
and will be deserving of its name.)  Two, below, is an example of where
PAN and use of its database could be extended by third party apps.  From
here on is entirely my ideas and opinions, not necessarily that of any of
the PAN development team, as I've only minimally hinted at this, before,
and they've had no opportunity to tell me what they think about it.

2. About a year b4 I left MSWormOS, I came across an "adware" supported
newsreader called Tifny3.  It was IIRC developed in Visual Basic, and used
a *.mdb format db along with the appropriate MS libraries to track its
posts.  The neat thing about it was how it automatically decompressed
attachments into real file system files that could be browsed or used by
any other app as it would files in any other file system folder, while at
the SAME time keeping the post the attachment came it linked to it thru
the database.  That way, metadata about the poster, date and time posted,
and group posted to, along with anything the poster said in either the
subject or body message, wasn't lost, and could be used (within Tifny3, of
course) to dynamically sort or group the attachments as desired.

I remember this as a very powerful feature and STILL miss the convenience
of effortlessly maintaining all that metadata while still having the
attachments available for use as files by other applications.

In other ways, Tifny wasn't all that spectacular, because it tried to do
to much.  It included not only its own still image viewer but also a media
player.  Not only were these less than spectacular, however, they weren't
even all THAT good implementations as previewers, because they at once
attempted to be MORE than a preview player with lots of extras, and tried
to fit it all into the same interface with the news client and database. 
It also included a browser, BTW, tho again, a rather poor one that IMO
would have been better left out altogether.  I expect what they were
trying to do was increase "stickiness" in view of getting more ad revenue,
but other than the rather unusual idea of using a database to track post
metadata while still making the attachments available in the file system,
it was an altogether unremarkable application, simply because it tried to
do to much.

The implications as they apply to my opinion what PAN's approach should
be, particularly with the MySQL compatibility, should now be obvious.  I
think that PAN should continue being mainly a newsreader, even as it
begins to live up to its name as THE Pimp-Ass Newsreader.

Viewing still images in PAN is great.  I wouldn't have it any other way. 
However, we do NOT need to implement media players or other sorts of
extensions directly in PAN.  Rather, we should include in the
configuration preference management for audio and movie players, and for a
separate still-image viewer as well (as PAN had back in the pre-GTK2 era,
when it was a Gnome app, not GTK, only different).  To keep it GTK only,
these need to be independently configurable not dependent on Gnome
settings or whatever. However, in the interest of compatibility, perhaps a
button to "take my Gnome settings" could be included, and made visible
only if Gnome was available to be used. (This said as a KDE user..) 
However, this button would grab the settings and use them to quick-fill
the application prefs, rather than depend on them, to keep PAN a separate
entity.  (Thus, changing Gnome settings wouldn't change PAN settings.  One
would need to hit "the button" again.)

That would handle preview and direct open functionality.  However, third
party apps could take advantage of the MySQL compatibility to do all sorts
of other things, and PAN COULD provide API hooks to allow further
integration with PAN directly, if desired.  This MIGHT start an entire
cottage industry of PAN extensions, all with the goal of letting each
application take care of what it did best, without any of them,
PARTICULARLY PAN itself, having to do everything, and therefore, doing
"everything" poorly or anyway unremarkably.  PAN could remain the
"Pimp-Ass Newsreader", made even MORE "Pimp-Ass", because it allowed
whatever third party app was the user's idea of a "pimp-ass media-player",
for instance, to integrate smoothly and operate from the same original
post metadata.  Get it right, and people would  be using an import feature
to import the REST of their MP3 collection, or JPEG collection, or
whatever, into PAN (tho it would actually be into the PAN database), so it
could all be managed and used together.  Of course, the other piece of the
puzzle, then, is that database manager itself, capable of importing and
managing all this, quite apart from PAN and its role as a tool in
downloading from usenet.

Quite a vision, I think, tho of course that of others might be different
(but this vision would allow that..).  PAN has quite a ways to go, but the
solid foundation as a newsreader is already there, and with the proper
implementation of the database backend, provided it's done with this sort
of flexibility in mind, we could be easily be on our way..

(Of course, the OTHER thing that'd be neat would be to integrate such a
database with Reiser4, if available and at the user's option, of course,
but that's an entirely different topic. <g>)

-- 
Duncan - List replies preferred.   No HTML msgs.
"They that can give up essential liberty to obtain a little
temporary safety, deserve neither liberty nor safety." --
Benjamin Franklin