Mailman

See also

• meta:Mailing lists
• meta:Mailing lists/Administration

How To

Add a list to Gmane

A new list can be added to Gmane very easily if the first message has not already been sent or history is not important. If the history needs to be imported, a ticket should be file in Phabricator with projects 'Wikimedia-Mailing-list' and 'operations' (easy link stating which list needs to imported and to which Gmane group if it exists already.

Docs and links

• Site Administrator docs on gnu.org

Authorized recipients for ops@lists.wikimedia.org

Anyone with a signed NDA can be on the ops@lists.wikimedia.org mailing list. For a list of signed NDAs for staff/contractors there is T83783, WMF-NDA and ldap/nda (or ldap/wmf).

Migrate servers

See Mailman/Migration for comments made about past migrations (lily to sodium, sodium to fermium, fermium to lists1001).

Upgrading Mailman3

When upgrading Mailman3 packages for a version that includes schema changes, use the following process.

1. Downtime host in icinga
2. Disable puppet
3. Manually stop mailman3 and mailman3-web systemd services
4. Run apt to install the new packages
5. Run sudo mailman-wrapper help to apply the mailman3 schema updates (yes, running just the "help" command will apply the schema changes)
6. Run sudo mailman-wrapper help again to make sure the schema update didn't obviously break anything.
7. Run sudo mailman-web migrate to apply the mailman3 schema updates
8. Clear static CSS/JS caches: sudo mailman-web collectstatic --clear --noinput && sudo mailman-web compress
9. Start mailman3 and mailman3-web systemd services again, keep an eye on errors in the logs
10. Re-enable puppet and remove icinga downtime if it hasn't expired yet.

Configuration details

The new Mailman setup lives on lists1001, and uses the standard Debian package mailman. The mailing list state is under /var/lib/mailman/, the global configuration is in /etc/mailman/.

The mail server used is Exim, the web server used is Apache.

Mailman setup

The local mailman configuration file is available to see on Diffusion. Configuration is mostly default version config with few changes (L96 and below are non-default).

Mail server setup

Near the top of the exim4.conf file, there are several macros related to Mailman. These define system-specific settings/locations used by the router(s) and transport(s) in the rest of the configuration file. For a Debian/Ubuntu Mailman package, the following macro's are accurate:

# Mailman
MAILMAN_HOME = /usr/lib/mailman
MAILMAN_LISTS_HOME = /var/lib/mailman
MAILMAN_WRAP = MAILMAN_HOME/mail/mailman
MAILMAN_UID = list
MAILMAN_GID = list

There's a domain list that contains a list of all domains that can "contain" mailing lists, i.e. the domains for which the Mailman router(s) should run. This list is also used as part of the "local domains" list, the list for which this mail server accepts mail and handles it locally.

domainlist mailman_domains = lists.wikimedia.org
domainlist local_domains = +system_domains : +mailman_domains

Main configuration

Several tweaks have been made to the main configuration to make Mailman delivery go smooth.

In case of high load / lots of incoming connections, mail from the local host (including Mailman) and other Wikimedia servers are given preference:

smtp_reserve_hosts = <; 127.0.0.1 ; ::1 ; +wikimedia_nets

For big mailing lists, Mailman needs to send a lot of recipients per mail / connection. Per default, Exim only queues mails that have > 10 recipients, to be delivered by a subsequent queue runner, which can cause significant delays. The default Mailman limit is 500 recipients per connection, so make Exim accept that:

smtp_accept_queue_per_connection = 500

Allow Exim to do 50 deliveries to remote hosts in parallel (this means 50 processes):

remote_max_parallel = 50

Routers

In Exim, the routers determine if a certain e-mail address is accepted for delivery or mail transport, and how it's going to be handled (routed). For Mailman, the following list router accepts a recipient that:

• has a domain in the domain list mailman_domains
• has a Mailman configuration file matching the local part (i.e. the mailing list exists)

Certain postfixes of the localpart, e.g. "-bounces" are accepted as well.

When the router accepts the recipient address, it's set up for delivery using the list transport (see below).

# Mailman list handling. Test the mailing list address without suffix
# first, as a mailing list like wikifi-admin is a valid list name.

list:
        driver = accept
        domains = +mailman_domains
        require_files = MAILMAN_LISTS_HOME/lists/$local_part/config.pck
        transport = list

list_suffix:
        driver = accept
        domains = +mailman_domains
        require_files = MAILMAN_LISTS_HOME/lists/$local_part/config.pck
        local_part_suffix = -bounces : -bounces+* : \
                                -confirm+* : -join : -leave : \
                                -owner : -request : -admin : \
                                -subscribe : -unsubscribe
        transport = list

If the conditions for this router fail (i.e. the router is not run) then the no_more makes sure that no subsequent routers will be tried (in the current configuration there are none that might accept), and the recipient address is failed.

Transports

An Exim transport configures a way of transporting a message, e.g. over the network (SMTP), to a file (MBOX/Maildir/etc) or using a pipe to a process. The following transport sets up delivery to Mailman:

# Mailman pipe transport

list:
        driver = pipe
        command = MAILMAN_WRAP \
                '${if def:local_part_suffix \
                        {${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}} \
                        {post}}' \
                $local_part
        current_directory = MAILMAN_LISTS_HOME
        home_directory = MAILMAN_LISTS_HOME
        user = MAILMAN_UID
        group = MAILMAN_GID

For content scanning, temporary mbox files are written to /var/spool/exim4/scan, and deleted after scanning. Similarly, Exim keeps "hints" databases in /var/spool/exim4/db, which are non-essential caches. To improve performance somewhat, these directory is mounted as a tmpfs filesystem, using the following line in /etc/fstab:

tmpfs   /var/spool/exim4/scan   tmpfs   defaults        0       0
tmpfs   /var/spool/exim4/db     tmpfs   defaults        0       0

Mailing list privacy protection

It has happened in the past that by hitting on the Reply All button in one's e-mail client, private info from an internal list leaked to a public mailing list because it was listed in the CC list, and the imprudent sender did not notice. In order to try to catch these incidents, a little filter has been implemented.

1. If the To: or CC: of a message body matches an item in a list of private mailing list addresses, and
2. the list of recipients as known by the mailing list server contains a Wikimedia mailing list that's not a private mailing lists, then
3. the message is bounced with the message Message rejected for privacy protection: The list of recipients contains both private and public lists.

It's possible to circumvent this restriction by sending to the private list as a BCC, Blind Carbon Copy.

This filter is implemented using an Exim system filter:

# Exim filter # Mailing list privacy protection if foranyaddress $h_To:,$h_Cc: ( $thisaddress matches "\\N^(internal-l|private-l)@(lists\.|mail\.)?wiki[mp]edia\.org$\\N" ) then if foranyaddress $recipients ( $thisaddress matches "\\N@lists\.wikimedia\.org$\\N" and $thisaddress does not match "\\N^(internal-l|private-l)@\\N" ) then fail text "Message rejected for privacy protection: The list of recipients contains both private and public mailing lists" endif endif

This filter is enabled in the configuration file using

system_filter = CONFDIR/system_filter

Address header rewriting

It turned out that, after the migration, many users kept sending mails to both the old and the new mailing list addresses, thereby causing duplicate messages. To reduce this, Exim has been configured to rewrite the old mailing list addresses to the new ones in the To: and CC: headers, using the following option on the list transport:

list: ... # Rewrite body headers of old mailing list addresses to new ones headers_rewrite = \N^.*@(mail\.)?wiki[mp]edia\.org$\N "${if exists{MAILMAN_LISTS_HOME/lists/$local_part/config.pck}{$local_part@lists.wikimedia.org}fail}" ct

Web server setup

To get Mailman running with Apache the puppet class role::lists is applied which uses module mailman. Puppet module mailman uses puppet module Apache which sets up Apache and the site config is in the template lists.wikimedia.org.erb.

See also http://www.gnu.org/software/mailman/mailman-install/node10.html

SpamAssassin

SpamAssassin is installed using the default Ubuntu spamassassin package. A couple of configuration changes were made.

By default, spamd, if enabled, runs as root. To change this:

# adduser --system --home /var/lock/spamassassin --group --disabled-password --disabled-login spamd

The following settings were modified in /etc/default/spamassassin:

# Change to one to enable spamd
ENABLED=1

User preferences are disabled, spamd listens on the loopback interface only, and runs as user/group spamd:

OPTIONS="--max-children 5 --nouser-config --listen-ip=127.0.0.1 -u spamd -g spamd"

Run spamd with nice level 10:

# Set nice level of spamd
NICE="--nicelevel 10"

Fighting spam in mailman

All list mail is sent through spamassassin which adds headers with a spam status and score.

This does not mean though that it automatically discards any messages. Only messages with a really high spam score (>= 6) are discarded immediately, below that messages are just tagged with a header. What actions to take based on this information is left to the list admins.

Spam from non-members

Most spam comes from generic spammers which harvested potential email address over the web and mailed them all: such spammers don't bother subscribing to the list. In contrast, it's rather rare for a public list to ever need receiving emails from non-members.

Multiple list admins report that they greatly reduced the amount of spam received simply by stopping acknowledging the messages, because in this way the spammer doesn't know the address is real.

This configuration is in Privacy options > Sender filter > generic_nonmember_action and you can set to "Discard". You can also reach this configuration at https://lists.wikimedia.org/mailman/admin/YOURLIST/?VARHELP=privacy/sender/hold_these_nonmembers .

Do NOT discard emails from non-members if there is any chance that real people need to write to your list's subscribers without subscribing (typical examples: private committee lists, feedback lists). Just avoid "bounce" and hold for moderation; if it becomes impossible to handle moderated messages, try to stop acknowledging the receipt of messages held for moderation.

Spam scores

The mailman UI supports this via the configuration variable header_filter_rules aka. 'Spam Filter Regexp' (description: Filter rules to match against the headers of a message.). See also https://www.gnu.org/software/mailman/mailman-admin/sender-filters.html

This can be found in the administrative interface in Privacy options...-> [Spam filters] -> Spam Filter Regexp (or visit directly the URL, replacing YOURLIST with your list name: https://lists.wikimedia.org/mailman/admin/YOURLIST/?VARHELP=privacy/spam/header_filter_rules ).

You can filter X-Spam-Score, which on lists.wikimedia.org as of 2017 is indicated with pluses (+) instead of stars (*):

X-Spam-Score:[^+]*[+]{4,}

This line matches when 4 or more pluses are encountered on the same line, meaning a score of 4 or more. SpamAssassin considers messages to be spam over 4 points. See some examples of real-life SpamAssassin scores (X-Spam-Report).

Additionally you have to pick an action for that which can be one of:

Defer  Hold  Reject  Discard  Accept

If you use "Hold" you will have to manually check the moderation queue every once in a while, but can avoid possible false positives, if you use "Discard" you don't have to do anything but _might_ discard some ham without noticing, though that is unlikely with the scores proposed above.

If spam is rampant, you could try and add other, more specific blacklists. For instance, if you find that a good portion of the spam is in BRBL or in MailSpike whitelist, you could add a rule after the score-based one, like

(RCVD_IN_BRBL_LASTEXT|RCVD_IN_MSPIKE_H2|RCVD_IN_MSPIKE_H1)

and set it to "Hold" messages. Do not discard messages based on hand-made filters like this, they may match good messages too. Instead, use "Hold" and go check the moderation queue in the next days, to ensure there are no false positives. Note however that "Hold" seems to override other "Discard" filters, so such a filter may prevent discarding of messages from non-members etc. if you have such discards configured.

DMARC Compatibility

DMARC is an email authentication mechanism intended to combat phishing and spam. To quote from dmarc.org: "DMARC, which stands for “Domain-based Message Authentication, Reporting & Conformance”, is an email authentication policy, and reporting protocol. It builds on the widely deployed SPF and DKIM protocols, adding linkage to the author (“From:”) domain name, published policies for recipient handling of authentication failures, and reporting from receivers to senders, to improve and monitor protection of the domain from fraudulent email."

Several large mail providers (yahoo, aol, etc.) have deployed strict DMARC policies. Unfortunately, these policies interfere with the operation of mailing list software like mailman. This is because mailman modifies the email contents (causing DKIM failure) and attempts to re-mail messages using from addresses that belong to remote mail systems (causing SPF failure). This results in lost email from list users with a strict DMARC policy set by their provider.

To work around this issue, beginning August 1, 2017, messages whose original From: domain publishes a DMARC policy of p=reject or p=quarantine will...

• Have the From: header rewritten (Munged) to the posters name 'via the list' <list_address> and
• Merge the poster's address into the Reply-To: header

To reiterate, only when a users email provider enforces a strict DMARC policy will from address be rewritten.

While rewriting the from header may not be ideal this is a necessary default setting to ensure reliability of the mailing list system. Without it messages may be lost silently, or with cryptic error DSN messages returned to the poster.

This is implemented in mailman like so:

# /etc/mailman/mm_cfg.py
DEFAULT_DMARC_MODERATION_ACTION=1

List administrators may also choose a different DMARC moderation action on a per-list basis by navigating to Privacy options > Sender filters. The possible dmarc moderation actions include:

• Munge From (default) - Rewrite the From: and Reply-To:
• Wrap Message - Wrap the message as a message/rfc822 sub-part in a MIME format outer message with From: and Reply-To: as above.
• Reject - Reject the post 
• Discard - Discard the post

Backups

lists1001 is backed up to helium using Bacula. The path on the source is /var/lib/mailman. The path on the target is whatever you choose in bconsole.

Tested failure modes

Because mail delivery and transport should be reliable, I have tested what happens in certain failure modes, e.g. when SpamAssassin's spamd daemon is not running.

Spamd not running

Because of the /defer_ok modifiers in the Exim ACLs, Exim will act as if no spam filtering attempts are made when spamd is not running, and will accept the message. The following lines are logged:

spam acl condition: warning - spamd connection to 127.0.0.1, port 783 failed: Connection refused
spam acl condition: all spamd servers failed
H=xxx.xxxxxxx.xx [xx.xx.xx.xx]:xxxx I=[145.97.39.157]:25 U=exim Warning: ACL "warn" statement skipped: condition test deferred

Mailman not running

If the Mailman queue runner daemons are not running, incoming messages will still get delivered to the Mailman queue by Exim. However, nothing else will happen until the Mailman processes are started.

Monitoring

We monitor a few key components:

• Processes
  • mailman_ctl: Is the control process in the list of running processes?
    • If not, check logs in /var/log/mailman and/or try restarting mailman.service.
  • mailman_qrunner: Is the queue runner process in the list of running processes?
    • If not, check logs in /var/log/mailman and/or try restarting mailman.service.
• HTTP
  • mailman archives: Are the archives for wikimedia-l reachable?
    • If not, check apache.service.
  • mailman list info: Is the subscribe page for wikimedia-l reachable?
    • If not, check apache.service.
• Queues
  • mailman_queue_size: Is there a backlog in the bounces, in, and virgin queues?
    • If so, check:
      • dashboard
      • logs in /var/log/mailman
      • you may have to look at what is in /var/lib/mailman/qfiles/(bounces|in|virgin) to see what is not sending.
  • Mailman outbound queue hours until empty: How long will it take to drain the out queue?
    • If it's going to take a long time, check:
      • Date and time of day. Historically, the first of the month has the longest queue of about 15+ hours long and 08:00 UTC each day about 1+ hours long.
      • dashboard
      • logs in /var/log/mailman and /var/log/exim4
      • you may have to look at the messages themselves in /var/lib/mailman/qfiles/out to see what is not sending.

Who has the passwords?

The following people have the master (site) password that can be used to login to the admin interface of all lists.

• all members of the operations team or users with root access on the ops bastion host
• Patrick Earley (WMF)
• Joe Sutherland (WMF)
• Nick Wilson (WMF)

The following people have the list creator password that can be used to create new lists but not login to existing lists. This section is outdated as of 2019-03-18

• all members of the operations team or users with root access on the ops bastion host
• James Alexander
• (potentially other CA team members via James)
• User:Odder
• User:Barras

The following people have shell access on the list server. They can run mailman binaries which includes changing list configs and passwords.

• all members of the operations team or users with root access on the ops bastion host