Build Your Own Mail Server From Scratch

Allocate and activate the swap file. Allocate in 1KB chuncks doing it in one go, could fail on low memory systems

Check if swap is mounted then activate on boot

Install the certbot PPA.

Old kernels pile up over time and take up a lot of disk space, and because of Mail-in-a-Box changes there may be other packages that are no longer needed. Clear out anything apt knows is safe to delete.

This is a non-interactive setup so we can't ask the user. If /etc/timezone is missing, set it to UTC.

This is supposedly sufficient. But because we're not sure if hardware entropy is really any good on virtualized systems, we'll also seed from Ubuntu's pollinate servers:

Between these two, we really ought to be all set.

We need an ssh key to store backups via rsync, if it doesn't exist create one

Allow incoming connections to SSH.

Add a listen-on directive if it doesn't exist inside the options block.

First we'll disable systemd-resolved's management of resolv.conf and its stub server. Breaking the symlink to /run/systemd/resolve/stub-resolv.conf means systemd-resolved will read it for DNS servers to use. Put in 127.0.0.1, which is where bind9 will be running. Obviously don't do this before installing bind9 or else apt won't be able to resolve a server to download bind9 from.

Restart the DNS services.

On first installation, the log files that the jails look at don't all exist. e.g., The roundcube error log isn't normally created until someone logs into Roundcube for the first time. This causes fail2ban to fail to start. Later scripts will ensure the files exist and then fail2ban is given another restart at the very end of setup.

Install openssl.

Create a directory to store TLS-related things like "SSL" certificates.

Generate a new private key.

The key is only as good as the entropy available to openssl so that it can generate a random key. "OpenSSL’s built-in RSA key generator .... is seeded on first use with (on Linux) 32 bytes read from /dev/urandom, the process ID, user ID, and the current time in seconds. [During key generation OpenSSL] mixes into the entropy pool the current time in seconds, the process ID, and the possibly uninitialized contents of a ... buffer ... dozens to hundreds of times."

A perfect storm of issues can cause the generated key to be not very random:

• improperly seeded /dev/urandom, but see system.sh for how we mitigate this
• the user ID of this process is always the same (we're root), so that seed is useless
• zero'd memory (plausible on embedded systems, cloud VMs?)
• a predictable process ID (likely on an embedded/virtualized system)
• a system clock reset to a fixed time on boot

Since we properly seed /dev/urandom in system.sh we should be fine, but I leave in the rest of the notes in case that ever changes. Set the umask so the key file is never world-readable.

Generate a self-signed SSL certificate because things like nginx, dovecot, etc. won't even start without some certificate in place, and we need nginx so we can offer the user a control panel to install a better certificate. Generate a certificate signing request.

Generate the self-signed certificate.

Delete the certificate signing request because it has no other purpose.

Symlink the certificate into the system certificate path, so system services can find it.

Generate some Diffie-Hellman cipher bits. openssl's default bit length for this is 1024 bits, but we'll create 2048 bits of bits per the latest recommendations.

Prepare nsd's configuration.

Add log rotation

Since we have bind9 listening on localhost for locally-generated DNS queries that require a recursive nameserver, and the system might have other network interfaces for e.g. tunnelling, we have to be specific about the network interfaces that nsd binds to.

Create DNSSEC signing keys.

TLDs don't all support the same algorithms, so we'll generate keys using a few different algorithms. RSASHA1-NSEC3-SHA1 was possibly the first widely used algorithm that supported NSEC3, which is a security best practice. However TLDs will probably be moving away from it to a a SHA256-based algorithm.

Supports RSASHA1-NSEC3-SHA1 (didn't test with RSASHA256):

• .info
• .me

Requires RSASHA256

• .email
• .guide

Supports RSASHA256 (and defaulting to this)

• .fund

Create the Key-Signing Key (KSK) (with -k) which is the so-called Secure Entry Point. The domain name we provide ("domain") doesn't matter -- we'll use the same keys for all our domains.

ldns-keygen outputs the new key's filename to stdout, which we're capturing into the KSK variable.

ldns-keygen uses /dev/random for generating random numbers by default. This is slow and unecessary if we ensure /dev/urandom is seeded properly, so we use /dev/urandom. See system.sh for an explanation. See #596, #115.

Now create a Zone-Signing Key (ZSK) which is expected to be rotated more often than a KSK, although we have no plans to rotate it (and doing so would be difficult to do without disturbing DNS availability.) Omit -k and use a shorter key length.

These generate two sets of files like:

• K_domain_.+007+08882.ds: DS record normally provided to domain name registrar (but it's actually invalid with _domain_)
• K_domain_.+007+08882.key: public key
• K_domain_.+007+08882.private: private key (secret!)

The filenames are unpredictable and encode the key generation options. So we'll store the names of the files we just generated. We might have multiple keys down the road. This will identify what keys are the current keys.

And loop to do the next algorithm...

Force the dns_update script to be run every day to re-sign zones for DNSSEC before they expire. When we sign zones (in dns_update.py) we specify a 30-day validation window, so we had better re-sign before then.

Permit DNS queries on TCP/UDP in the firewall.

Tweak some queue settings: * Inform users when their e-mail delivery is delayed more than 3 hours (default is not to warn). * Stop trying to send an undeliverable e-mail after 2 days (instead of 5), and for bounce messages just try for 1 day.

Install the outgoing_mail_header_filters file required by the new 'authclean' service.

Modify the outgoing_mail_header_filters file to use the local machine name and ip on the first received header line. This may help reduce the spam score of email by removing the 127.0.0.1 reference.

Enable TLS on these and all other connections (i.e. ports 25 and 587) and require TLS before a user is allowed to authenticate. This also makes opportunistic TLS available on incoming mail. Set stronger DH parameters, which via openssl tend to default to 1024 bits (see ssl.sh).

Prevent non-authenticated users from sending mail that requires being relayed elsewhere. We don't want to be an "open relay". On outbound mail, require one of:

• permit_sasl_authenticated: Authenticated users (i.e. on port 587).
• permit_mynetworks: Mail that originates locally.
• reject_unauth_destination: No one else. (Permits mail whose destination is local and rejects other mail.)

Because of a spampd bug, limit the number of recipients in each connection. See https://github.com/mail-in-a-box/mailinabox/issues/1523.

Who can send mail to us? Some basic filters.

• reject_non_fqdn_sender: Reject not-nice-looking return paths.
• reject_unknown_sender_domain: Reject return paths with invalid domains.
• reject_authenticated_sender_login_mismatch: Reject if mail FROM address does not match the client SASL login
• reject_rhsbl_sender: Reject return paths that use blacklisted domains.
• permit_sasl_authenticated: Authenticated users (i.e. on port 587) can skip further checks.
• permit_mynetworks: Mail that originates locally can skip further checks.
• reject_rbl_client: Reject connections from IP addresses blacklisted in zen.spamhaus.org
• reject_unlisted_recipient: Although Postfix will reject mail to unknown recipients, it's nicer to reject such mail ahead of greylisting rather than after.
• check_policy_service: Apply greylisting using postgrey.

Postfix connects to Postgrey on the 127.0.0.1 interface specifically. Ensure that Postgrey listens on the same interface (and not IPv6, for instance). A lot of legit mail servers try to resend before 300 seconds. As a matter of fact RFC is not strict about retry timer so postfix and other MTA have their own intervals. To fix the problem of receiving e-mails really latter, delay of greylisting has been set to 180 seconds (default is 300 seconds).

We are going to setup a newer whitelist for postgrey, the version included in the distribution is old

Increase the message size limit from 10MB to 128MB. The same limit is specified in nginx.conf for mail submitted via webmail and Z-Push.

Allow the two SMTP ports in the firewall.

Restart services

The dovecot-imapd, dovecot-pop3d, and dovecot-lmtpd packages automatically enable IMAP, POP and LMTP protocols.

Set basic daemon options.

The default_process_limit is 100, which constrains the total number of active IMAP connections (at, say, 5 open connections per user that would be 20 users). Set it to 250 times the number of cores this machine has, so on a two-core machine that's 500 processes/100 users). The default_vsz_limit is the maximum amount of virtual memory that can be allocated. It should be set reasonably high to avoid allocation issues with larger mailboxes. We're setting it to 1/3 of the total available memory (physical mem + swap) to be sure. See here for discussion: - https://www.dovecot.org/list/dovecot/2012-August/137569.html - https://www.dovecot.org/list/dovecot/2011-December/132455.html

The inotify max_user_instances default is 128, which constrains the total number of watched (IMAP IDLE push) folders by open connections. See http://www.dovecot.org/pipermail/dovecot/2013-March/088834.html. A reboot is required for this to take effect (which we don't do as as a part of setup). Test with cat /proc/sys/fs/inotify/max_user_instances.

Set the location where we'll store user mailboxes. '%d' is the domain name and '%n' is the username part of the user's email address. We'll ensure that no bad domains or email addresses are created within the management daemon.

Create, subscribe, and mark as special folders: INBOX, Drafts, Sent, Trash, Spam and Archive.

Enable SSL, specify the location of the SSL certificate and private key files. Disable obsolete SSL protocols and allow only good ciphers per http://baldric.net/2013/12/07/tls-ciphers-in-postfix-and-dovecot/. Enable strong ssl dh parameters

Disable in-the-clear IMAP/POP because there is no reason for a user to transmit login credentials outside of an encrypted connection. Only the over-TLS versions are made available (IMAPS on port 993; POP3S on port 995).

Make IMAP IDLE slightly more efficient. By default, Dovecot says "still here" every two minutes. With K-9 mail, the bandwidth and battery usage due to this are minimal. But for good measure, let's go to 4 minutes to halve the bandwidth and number of times the device's networking might be woken up. The risk is that if the connection is silent for too long it might be reset by a peer. See #129 and How bad is IMAP IDLE.

Set POP3 UIDL. UIDLs are used by POP3 clients to keep track of what messages they've downloaded. For new POP3 servers, the easiest way to set up UIDLs is to use IMAP's UIDVALIDITY and UID values, the default in Dovecot.

Setting a postmaster_address is required or LMTP won't start. An alias will be created automatically by our management daemon.

Configure sieve. We'll create a global script that moves mail marked as spam by Spamassassin into the user's Spam folder.

• sieve_before: The path to our global sieve which handles moving spam to the Spam folder.

• sieve_before2: The path to our global sieve directory for sieve which can contain .sieve files to run globally for every user before their own sieve files run.

• sieve_after: The path to our global sieve directory which can contain .sieve files to run globally for every user after their own sieve files run.

• sieve: The path to the user's main active script. ManageSieve will create a symbolic link here to the actual sieve script. It should not be in the mailbox directory (because then it might appear as a folder) and it should not be in the sieve_dir (because then I suppose it might appear to the user as one of their scripts).

• sieve_dir: Directory for :personal include scripts for the include extension. This is also where the ManageSieve service stores the user's scripts.

Copy the global sieve script into where we've told Dovecot to look for it. Then compile it. Global scripts must be compiled now because Dovecot won't have permission later.

PERMISSIONS

Ensure configuration files are owned by dovecot and not world readable.

Ensure mailbox files have a directory that exists and are owned by the mail user.

Same for the sieve scripts.

Allow the IMAP/POP ports in the firewall.

Allow the Sieve port in the firewall.

Restart services.

Create an empty database if it doesn't yet exist.

Specify how the database is to be queried for user authentication (passdb) and where user mailboxes are stored (userdb).

Configure the SQL to query for a user's metadata and password.

Have Dovecot provide an authorization service that Postfix can access & use.

And have Postfix use that service. We disable it here so that authentication is not permitted on port 25 (which does not run DKIM on relayed mail, so outbound mail isn't correct, see #830), but we enable it specifically for the submission port.

Postfix will query the exact address first, where the priority will be alias records first, then user records. If there are no matches for the exact address, then Postfix will query just the domain part, which we call catch-alls and domain aliases. A NULL permitted_senders column means to take the value from the destination column.

SQL statement to check if we handle incoming mail for a domain, either for users or aliases.

SQL statement to check if we handle incoming mail for a user.

SQL statement to rewrite an email address if an alias is present.

Postfix makes multiple queries for each incoming mail. It first queries the whole email address, then just the user part in certain locally-directed cases (but we don't use this), then just @+the domain part. The first query that returns something wins. See http://www.postfix.org/virtual.5.html.

virtual-alias-maps has precedence over virtual-mailbox-maps, but we don't want catch-alls and domain aliases to catch mail for users that have been defined on those domains. To fix this, we not only query the aliases table but also the users table when resolving aliases, i.e. we turn users into aliases from themselves to themselves. That means users will match in postfix's first query before postfix gets to the third query for catch-alls/domain alises.

If there is both an alias and a user for the same address either might be returned by the UNION, so the whole query is wrapped in another select that prioritizes the alias definition to preserve postfix's preference for aliases for whole email addresses.

Since we might have alias records with an empty destination because it might have just permitted_senders, skip any records with an empty destination here so that other lower priority rules might match.

Restart Services

Make sure configuration directories exist.

Used in InternalHosts and ExternalIgnoreList configuration directives. Not quite sure why.

We need to at least create these files, since we reference them later. Otherwise, opendkim startup will fail

Add various configuration options to the end of opendkim.conf.

Create a new DKIM key. This creates mail.private and mail.txt in $STORE/mail/dkim. The former is the private key and the latter is the suggested DNS TXT entry which we'll include in our DNS setup. Note that the files are named after the 'selector' of the key, which we can change later on to support key rotation.

A 1024-bit key is seen as a minimum standard by several providers such as Google. But they and others use a 2048 bit key, so we'll do the same. Keys beyond 2048 bits may exceed DNS record limits.

Ensure files are owned by the opendkim user and are private otherwise.

Add OpenDKIM and OpenDMARC as milters to postfix, which is how OpenDKIM intercepts outgoing mail to perform the signing (by adding a mail header) and how they both intercept incoming mail to add Authentication-Results headers. The order possibly/probably matters: OpenDMARC relies on the OpenDKIM Authentication-Results header already being present.

Be careful. If we add other milters later, this needs to be concatenated on the smtpd_milters line.

The OpenDMARC milter is skipped in the SMTP submission listener by configuring smtpd_milters there to only list the OpenDKIM milter (see mail-postfix.sh).

We need to explicitly enable the opendmarc service, or it will not start

Restart services.

Allow spamassassin to download new rules.

Configure pyzor, which is a client to a live database of hashes of spam emails. Set the pyzor configuration directory to something sane. The default is ~/.pyzor. We used to use that, so we'll kill that old directory. Then write the public pyzor server to its servers file. That will prevent an automatic download on first use, and also means we can skip 'pyzor discover', both of which are currently broken by something happening on Sourceforge (#496).

check with: pyzor --homedir /etc/mail/spamassassin/pyzor ping

Configure spampd: * Pass messages on to docevot on port 10026. This is actually the default setting but we don't want to lose track of it. (We've configured Dovecot to listen on this port elsewhere.) * Increase the maximum message size of scanned messages from the default of 64KB to 500KB, which is Spamassassin (spamc)'s own default. Specified in KBytes. * Disable localmode so Pyzor, DKIM and DNS checks can be used.

Spamassassin normally wraps spam as an attachment inside a fresh email with a report about the message. This also protects the user from accidentally openening a message with embedded malware.

It's nice to see what rules caused the message to be marked as spam, but it's also annoying to get to the original message when it is an attachment, modern mail clients are safer now and don't load remote content or execute scripts, and it is probably confusing to most users.

Tell Spamassassin not to modify the original message except for adding the X-Spam-Status & X-Spam-Score mail headers and related headers.

To mark mail as spam or ham, just drag it in or out of the Spam folder. We'll use the Dovecot antispam plugin to detect the message move operation and execute a shell script that invokes learning.

Enable the Dovecot antispam plugin.

Configure the antispam plugin to call sa-learn-pipe.sh.

Have Dovecot run its mail process with a supplementary group (the spampd group) so that it can access the learning files.

Here's the script that the antispam plugin executes. It spools the message into a temporary file and then runs sa-learn on it. from http://wiki2.dovecot.org/Plugins/Antispam

Create empty bayes training data (if it doesn't exist). Once the files exist, ensure they are group-writable so that the Dovecot process has access.

Initial training? sa-learn --ham storage/mail/mailboxes///cur/ sa-learn --spam storage/mail/mailboxes///.Spam/cur/

Kick services.

Install nginx and a PHP FastCGI daemon.

Turn off nginx's default website.

Copy in a nginx configuration file for common and best-practices SSL settings from @konklone. Replace STORAGE_ROOT so it can find the DH params.

Fix some nginx defaults. The server_names_hash_bucket_size seems to prevent long domain names! The default, according to nginx's docs, depends on "the size of the processor’s cache line." It could be as low as 32. We fixed it at 64 in 2014 to accommodate a long domain name (20 characters?). But even at 64, a 58-character domain name won't work (#93), so now we're going up to 128.

Tell PHP not to expose its version number in the X-Powered-By header.

Set PHPs default charset to UTF-8, since we use it. See #367.

Switch from the dynamic process manager to the ondemand manager see #1216

Bump up PHP's max_children to support more concurrent connections

Other nginx settings will be configured by the management service since it depends on what domains we're serving, which we don't know until mail accounts have been created.

Create the iOS/OS X Mobile Configuration file which is exposed via the nginx configuration at /mailinabox-mobileconfig.

Create the Mozilla Auto-configuration file which is exposed via the nginx configuration at /.well-known/autoconfig/mail/config-v1.1.xml. The format of the file is documented at: https://wiki.mozilla.org/Thunderbird:Autoconfiguration:ConfigFileFormat and https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration/FileFormat/HowTo.

make a default homepage

Start services.

Open ports.

Install Roundcube from source if it is not already present or if it is out of date. Combine the Roundcube version number with the commit hash of plugins to track whether we have the latest version of everything.

paths that are often reused.

checks if the version is what we want install roundcube

install roundcube persistent_login plugin

install roundcube html5_notifier plugin

download and verify the full release of the carddav plugin

unzip and cleanup

record the version we've installed

Create a configuration file.

For security, temp and log files are not stored in the default locations which are inside the roundcube sources directory. We put them instead in normal places.

Configure CardDav

Create writable directories.

Ensure the log file monitored by fail2ban exists, or else fail2ban can't start.

Password changing plugin settings The config comes empty by default, so we need the settings we're not planning to change in config.inc.dist...

so PHP can use doveadm, for the password changing plugin

set permissions so that PHP can use users.sqlite could use dovecot instead of www-data, but not sure it matters

Fix Carddav permissions:

root.www-data need all permissions, others only read

Run Roundcube database migration script (database is created if it does not exist)

Enable PHP modules.

Download and verify

Remove the current owncloud/Nextcloud

Extract ownCloud/Nextcloud

The two apps we actually want are not in Nextcloud core. Download the releases from their github repositories.

Starting with Nextcloud 15, the app user_external is no longer included in Nextcloud core, we will install from their github repository.

Fix weird permissions.

Create a symlink to the config.php in STORAGE_ROOT (for upgrades we're restoring the symlink we previously put in, and in new installs we're creating a symlink and will create the actual config later).

Make sure permissions are correct or the upgrade step won't run. $STORE/owncloud may not yet exist, so use -f to suppress that error.

If this isn't a new installation, immediately run the upgrade script. Then check for success (0=ok and 3=no upgrade needed, both are success). ownCloud 8.1.1 broke upgrades. It may fail on the first attempt, but that can be OK.

Add missing indices. NextCloud didn't include this in the normal upgrade because it might take some time.

Run conversion to BigInt identifiers, this process may take some time on large tables.

Nextcloud Version to install. Checks are done down below to step through intermediate versions.

Current Nextcloud Version, #1623 Checking /usr/local/lib/owncloud/version.php shows version of the Nextcloud application, not the DB $STORE/owncloud is kept together even during a backup. It is better to rely on config.php than version.php since the restore procedure can leave the system in a state where you have a newer Nextcloud application version than the database.

If config.php exists, get version number, otherwise CURRENT_NEXTCLOUD_VER is empty.

If the Nextcloud directory is missing (never been installed before, or the nextcloud version to be installed is different from the version currently installed, do the install/upgrade

Stop php-fpm if running. If theyre not running (which happens on a previously failed install), dont bail.

Backup the existing ownCloud/Nextcloud. Create a backup directory to store the current installation and database to

If ownCloud or Nextcloud was previously installed.... Database migrations from ownCloud are no longer possible because ownCloud cannot be run under PHP 7.

If we are running Nextcloud 13, upgrade to Nextcloud 14

During the upgrade from Nextcloud 14 to 15, user_external may cause the upgrade to fail. We will disable it here before the upgrade and install it again after the upgrade.

Create an initial configuration file.

Create an auto-configuration file to fill in database settings when the install script is run. Make an administrator account here or else the install can't finish.

Set permissions

Execute Nextcloud's setup step, which creates the Nextcloud sqlite database. It also wipes it if it exists. And it updates config.php with database settings and deletes the autoconfig.php file.

Update config.php. * trusted_domains is reset to localhost by autoconfig starting with ownCloud 8.1.1, so set it here. It also can change if the box's box.yourdomain.com changes, so this will make sure it has the right value. * Some settings weren't included in previous versions of Mail-in-a-Box. * We need to set the timezone to the system timezone to allow fail2ban to ban users within the proper timeframe * We need to set the logdateformat to something that will work correctly with fail2ban * mail_domain' needs to be set every time we run the setup. Making sure we are setting the correct domain name if the domain is being change from the previous setup. Use PHP to read the settings file, modify it, and write out the new settings array.

just the local part, matches our master administrator address

Enable/disable apps. Note that this must be done after the Nextcloud setup. The firstrunwizard gave Josh all sorts of problems, so disabling that. user_external is what allows Nextcloud to use IMAP for login. The contacts and calendar apps are the extensions we really care about here.

When upgrading, run the upgrade script again now that apps are enabled. It seems like the first upgrade at the top won't work because apps may be disabled during upgrade? Check for success (0=ok, 3=no upgrade needed).

Set PHP FPM values to support large file uploads (semicolon is the comment character in this file, hashes produce deprecation warnings)

Set Nextcloud recommended opcache settings

Configure the path environment for php-fpm

If apc is explicitly disabled we need to enable it

Set up a cron job for Nextcloud.

There's nothing much of interest that a user could do as an admin for Nextcloud, and there's a lot they could mess up, so we don't make any users admins of Nextcloud. But if we wanted to, we would do this: for user in $(tools/mail.py user admins); do sqlite3 $STORE/owncloud/owncloud.db "INSERT OR IGNORE INTO oc_group_user VALUES ('admin', '$user')" done

Enable PHP modules and restart PHP.

Copy Z-Push into place.

checks if the version Download

Extract into place.

Configure default config.

Configure BACKEND

Configure IMAP

Configure CardDav

Configure CalDav

Configure Autodiscover

Some directories it will use.

Add log rotation

Restart service.

Fix states after upgrade

libcgi-fast-perl is needed by /usr/lib/munin/cgi/munin-cgi-graph

edit config

The Debian installer touches these files and chowns them to www-data:adm for use with spawn-fcgi

ensure munin-node knows the name of this machine and reduce logging level to warning

Update the activated plugins through munin's autoconfiguration.

Deactivate monitoring of NTP peers. Not sure why anyone would want to monitor a NTP peer. The addresses seem to change (which is taken care of my munin-node-configure, but only when we re-run it.)nd /etc/munin/plugins/ -lname /usr/share/munin/plugins/ntp_ -print0 | xargs -0 /bin/rm -f

Deactivate monitoring of network interfaces that are not up. Otherwise we can get a lot of empty charts.

Create a 'state' directory. Not sure why we need to do this manually.

Create a systemd service for munin.

Restart services.

generate initial statistics so the directory isn't empty (We get "Pango-WARNING **: error opening config file '/root/.config/pango/pangorc': Permission denied" if we don't explicitly set the HOME directory when sudo'ing.) We check to see if munin-cron is already running, if it is, there is no need to run it simultaneously generating an error.