Re: mailman keeps holding for non-subscribers

[Bob Proulx]

Eric Wong wrote:
> OK, so I'm following half the recommendations
> 
> The ones I'm going against are:
> 
>       generic_nonmember_action=hold (I want Accept)
>       default_member_moderation=yes (I want no)

May I try to convince you otherwise?  Because there are good reasons
for the recommended settings.

> So, should I remove address@hidden from moderators?
> I still want automated spam filters such as SpamAssassin, though.

The listhelper anti-spam SpamAssassin et al cancel-bot depends upon
the hold actions.  If messages do not get held then it has no ability
to filter spam.  That's fundamental to how it works with Mailman.

> > Additionally any non-spam messages are also approved by the human
> > team, and their senders either unmoderated or whitelisted.  This
> > results in the avoidance of spam to the mailing lists while at the
> > same time avoiding delays in posting as only the initial contact is
> > held for moderation.  This has been necessary because spammers
> > routinely subscribe and then post spam.  Therefore we moderate new
> > addresses as they appear.
> 
> I've found automated spam filters good enough on their own and
> would like to just have those without human moderation.

My experience is that even with highly tuned automation that it still
needs continuous training feedback in order to keep accuracy to
acceptably levels.  Therefore instead of avoiding giving it feedback
we are giving it continuous feedback.

Another task of the listhelpers is to periodically review and
train-on-error the learning engines.  (The learning engines include
SpamAssassin's Bayes engine and the Bogofilter engine.  But also
specifically the CRM114 engine which does the best classification for
us and has been weighted more heavily due to it being most
successful.)  When we run the queues we are also providing training
for those engines.  That way as spam is continuously changing in
character the filters are also continuously being updated.

However Mailman doesn't have a lot of built in anti-spam ability.  The
listhelper system is bolted onto the moderation system.  Therefore it
can only anti-spam the moderated messages.  If the moderation is
bypassed then so is the anti-spam.  To improve this Mailman itself
would need to be modified.

> I don't want to have to whitelist anybody, it doesn't scale.

Perhaps in the large it does not but we are only handling 1500+
mailing lists and all of the associated subscribers at this time.  I
don't know how many subscribers in total.  There are 1521 mailing
lists using listhelper anti-spam right now.  But for small numbers
such as these it works okay.

But the real reason is that we are working within the limitations of
Mailman.  It's the only system Mailman supports.  Therefore it is the
system we are using.  Improvements would require changes to Mailman.

> > The resulting process means that as a general statement project
> > mailing lists need no explicit maintenance.  If you as a project
> > maintainer and also a maintainer of the mailing list do nothing then
> > everything happens as needed anyway.  You are however free to be as
> > involved in the mailing lists as you want.
> 
> So if I'm away and unable to administer address@hidden, and
> generic_nonmember_action is "Hold"; does the "human team" at GNU
> will eventually accept postings in my absence?

Yes.  Eventually usually means a few hours.

If you had done nothing you would have experienced what any other
poster to the mailing lists would experience.  There would have been a
short delay until a moderator from the listhelper team saw the message
and approved it, whitelisted or unmoderated your address, and then
subsequent messages would have been passed through by Mailman without
delay.

Sometimes there are longer or shorter delays for someone to see a
message.  Personally my own schedule allows me to look at the message
queues several times a day on most days.  But sometimes I am busy away
from the keyboard for a day.  However I am but one of the team and
there are several of us who look at the queues and it is the
overlapping schedules of everyone that fills in time periods.  It's
not organized and a bit of chaos and randomness but a new address
rarely sits in the hold queue for more than half a day.  And worst
case one of us would get to the queues at least once a day at the
worst.  But most days it would be a few hours.  I really should do
some statistical work to plot the delay times out.  It's on my list
for one of these days to do.

And that moderation hold is only for the initial contact.  Subsequent
messages are passed through without delay.  Therefore the usual
posters to a mailing list will see no delays.  They will converse all
as normal.  Plus there is no need to be subscribed to post a message.
There may be mechanical delays due to all of the normal reasons of
email however but that is outside of the anti-spam processes.

> > > The list in question is address@hidden
> > 
> > I don't recall any interaction with that mailing list.  It doesn't
> > ring a bell with me.
> > 
> > > I don't want to force users to subscribe to the mailing list to
> > > post(*).
> > 
> > Agreed.  How is that statement related to generic_nonmember_action set
> > to Hold?  Seems unrelated.
> 
> I mean that I don't want any artificial delays in handling new,
> unsubscribed users (in case the admins are away or unavailable).
> I'd rather let an occasional spam through.

It is your mailing list and this is up to you.  But people tend to be
very intolerant of spam on mailing lists.

For example if people receive their mail at Gmail or Yahoo or
wherever, and then spam to the mailing list is received at their
mailbox, and they push the Spam button, this teaches Google and Yahoo
and so forth that lists.gnu.org is a source of spam and may create
problems for normal mailing list delivery.  This has been more of a
problem with Yahoo than most other places.  Some spam is of course
inevitable but we try to keep it to a minimum.  If it becomes a
problem then if not us volunteers then FSF admin will need to become
involved.  Getting blacklisted due to spam is a pain to deal with.

The only thing we really must insist upon is to discard spam and not
reject spam.  Most spam uses forged from addresses.  Therefore
rejecting spam ala Mailman Reject usually sends a rejection message to
an innocent 3rd party who then gets "backscatter" spam.  They validly
report lists.gnu.org as a spam source in that case and it gets us in
trouble with the DNSBLs.  Therefore please do not Reject random spam
messages.

It is okay however to send a Reject to a human who would benefit from
the message.  I usually include a note with the rejection as to why.
There are a number of reasons.  Such as sending administrivia to the
mailing list members.  Or trying to subscribe to a private closed
list.  Or whatever.  A Reject for specific explicit reasons is okay.
Just not to the incoming spam generally.  Use judgement and Do The
Right Thing.

> > We never want to require people to subscribe to post bug reports or
> > other messages.  The GNU mailing lists are open mailing lists.  Can
> > you imagine requiring someone to subscribe in order to post a bug
> > report?  That would be inconvenient enough to drive most bug reporters
> > away.
> > 
> > Although some maintainers have made subscription a requirement for
> > their project mailing lists.  It goes against our recommendation and
> > guidelines.  I strongly recommend against it.
> 
> OK, I'm glad we agree there :>

I will take the convergences where I can! :-)

> > > In my case, it was myself since I've been changing email
> > > addresses because of the uncertainty around being able to afford
> > > .org down the line.
> > 
> > I will guess that you changed your email address, your first message
> > sent to the mailing list was therefore new and never before seen, it
> > was held for moderation.  Is that the issue here?
> 
> Maybe.  I had the same issue on Feb 3, 2020 and pushed my
> message through.  I refused to whitelist myself out of
> principle.

I keep thinking I will jump in and start hacking on Mailman.  But life
and time is what keeps everything from happening all at once.  Until
then I am much more pragmatic and work with the tools that are
provided to me to work with.  However having said that I created the
entire bolted-on listhelper anti-spam because I hated being a user of
the mailing lists with all of the spam that was on the lists before
and was not happy with the tools provided.  But at the time I had no
way to modify Mailman.  And I am not a python person, the source
language for Mailman, so that was an activation energy situation for
me to hack on Mailman.  So I did something else.

Bob