Re: IMAP server (was: Re: [Savannah-hackers] gnu-socketserver)

[Alain Magloire]

Bonjour

: I am aware of JavaMail. I'm working on a free version of the IMAP
: provider for the GNU Project (I'm also the architect for a big webmail
: system in the UK).
: 

Cool.

Somebody asked me if it was possible to write a full JavaMail using mailutils
as the back end.  The answer was yes.  His ideas were to cut development time
by implementing most of the functions with JNI wrappers calling mailutils.
Although feasible it is not a good idea, JavaMail provides a very nice
framework which is _pure_ JAVA(i.e portable).

: 
: My problem, as far as it goes, is that I need an IMAP server. I don't
: have one that I like so I thought I'd write one since they're not that
: hard on the skin.

A full IMAP server/client, will be hard on the skin, IMAP4 protocol is complex
compare, to say, POP3.  I'm sure you are quite aware of the rfc2060(IMAP4), 
the coming IMAP5 and all the protocol extensions (idle, threading, etc ...)

: Here's what I'm doing:
: 
: - abstracted backend,

Good, I'm always advocating for decoupling, abstraction layer and define
interfaces.

: use files as messages or files as folders

Not sure I follow here, if you have a Unix mailbox format, you are
going to explode the messages into files ?

: - efficient indexing based on simple directory system

Certainly has some advantages for speed.  Netscape use to implement
a folder summary.  Camel is also doing some indexing for speed.
The disadvantage is you have to maintain an external file and sync,
the trade off maybe worth it.

: - all delivery through the server, using an EXIM transport agent that
: speaks IMAP

Again, not sure I follow, IMAP is primary for dowloading/browsing emails
from multiple folders,  mail delivery usually go to SMTP.

: - scripting with Kawa (since the mail server will use Scheme anyway)

No comment.

: - shared folders implemented as shared directories and over protocol
: (possibly using one of the Java NFS servers)

Interesting.

: 
: >> Question: With Java servers, how do you overcome
: >> the overhead of having a JVM running for every new
: >> server (one for pop3, one for imap).  It seems like it could
: >> be cost prohibitive.
: 
: >Yes, at first glance, it looks strange to try to write a daemon
: >which should be space and io efficient in Java.  Nowadays
: >JIT's are very good but still ....
: 
: I'm not writing a daemon in the inetd sense of the word. That would
: be foolish in Java because of the startup time.
: 
: But Java is no slower than a C app when you come to using threads for
: servers

hmm, nitpicking.  I would say : a VM _with_ JIT(hotspot etc ..) can reach
the speed of a C program.  Altough this will vary very much on the VM.
We're not talking about Java VM interpreters.

: (and GCJ will make it no slower).

Yes, good point.

: I'm using the GNU-SocketServer API (which I wrote for a Java
: webserver called Paperclips, also a GNU app) which is a Java lib for
: pooling threads over a socket listener.
: 
: What that means is that a number of threads are instantiated and
: never killed. As connactions arrive individual threads are awoken and
: passed the incomming socket. It's very fast because there is no object
: churn.
: 
: If the thread pool needs to grow it can, when the threads are no
: longer needed they die. The pool under idle conditions always returns
: to the number of threads you configured.

Allright!  Glad to see you've applied some good patterns to your code,
do you have an URL handy?

This design patterns can also be implemented in C or C++.
Agreed that using the synchronize semantics offered by java
can make it is easier to develop and I suppose debug.

The pool threads, the work crew, the pipelining are all design patterns
that can be implement in C/C++ or any other language.

: I'm hoping to eventually have SocketServer do most of the things that
: you can do with inetd. For example, a config option to cause
: SocketServer to read the hosts.allow and hosts.deny files would be
: rather cool. Automatic instantiation of servers (that it knows about)
: through the inetd.conf might also be usefull.
: 

So far, I've never see the behaviour of a server written in pure Java
put under heavy load i.e if the webservers of google.com or yahoo and
the databases were written in pure java, what would be the performance?

I've written some industrial programs with Java in another life but the
emphasize was on AWT and nice little windows.  The heavy stuff was still
in C/Ada and fortran.

: 
: As far as multiple servers within the JVM are concerned: I don't
: intend writing a POP server but there's no reason one couldn't be
: written and used with my IMAP server.

Agreed.

: With Java one doesn't write a "server for X" but instead "an object
: to be a server for X". Once you've got an object to be a "server for
: X" then you instantiate the object in whatever environment you want.
: If I want to instantiate a POP server object and an IMAP server object
: in the same VM there's nothing to stop me. And the SocketServer API
: will let me use one pool of threads for both servers if I want.
: 

The problem beside the performance concern is security.  Servers usually
fork() for security concerns, many daemons have to setuid()/setgid()
to do some priviledge operations.  Threads  do not isolate enough
since it will affect all the other threads with nasty race conditions.
In the context where tight security policies are to be applied fork()ing
is a good thing, if not the critical sections will be the bottleneck
when doing priviledge operations.  For example it is consider, in certain
circle,  a bad omen to flip flop the (e)uid within a process.
 
: Having said all that the reason I'm using Java is because that's what
: I use for nearlly everything now. To me it's just a good RAD tool.

Yes, do not get me wrong, although I sound very critical, this is by
no mean bashing Java.

: With the IMAP server I'm not going to be using any of Java's platform
: independance features, the code will be tied pretty heavily to the
: GNU-Linux filesystem (since I have an abstracted backend you could
: swap that out I guess).

Interesting, did not BeOS had there mails tigth up with the filesystem ?

: 
: >> Having said that I would be happy to describe the
: >> design of my server if you're interested and if you can
: >> see areas where we might co-operate I'd be happy.
: 
: >We're here for the fun and to exchange, so unless it is
: >some cie trade secrets, lets hear it.
: 
: The above should be enough for idle interest... if you're more
: interested I'll tell you more.

Thank you for sharing them.  Any URL handy ?
The project sound very interesting/promessing.

--
alain