This patch adds a missing docstrings for exported identifiers, and
adjust some of the existing ones to match the standard style.
In some cases, the identifiers were un-exported after noticing they had
no external users.
Besides improving documentation, it also reduces the linter noise
significantly.
One of the SMTP tests was doing an external DNS lookup for a
non-existing host, which is reasonably harmless but makes the test less
hermetic.
This patch changes the non-existing host for an invalid address, which
has the same effect but avoids the network lookup.
This is a hack; ideally we would be able to override the resolver, but
Go does not implement that yet, and changing the code is not worth the
additional complexity.
We have many places in our tests where we create temporary directories,
which we later remove (most of the time). We have at least 3 helpers to
do this, and various places where it's done ad-hoc (and the cleanup is
not always present).
To try to reduce the clutter, and make the tests more uniform and
readable, this patch introduces two helpers in a new "testutil" package:
one for creating and one for removing temporary directories.
These new functions are safer, better tested, and make the tests more
consistent. All the tests are updated to use them.
The outgoing security level checks are not being performed, because of a
bug: the courier thinks the "to"'s domain is always empty.
This patch fixes the bug by simplifying the logic, as there's no need
for the conditional (there is always a domain in the "to" address if it
got to the SMTP courier).
This commit removes the experimental MTA-STS (Strict Transport Security)
implementation for now, as it's not up to date with the latest draft.
Development will continue on the "sts" branch, but this way it won't
block releases until it is ready.
Commits reverted:
- cb6500b993
- 0eeb964534
- e66288e4b4
- 216cf47ffa
- d66b06de51
- fe00750e39
- 933ab54cd8
The current default is "procmail" for historical reasons, but the
program has been unmaintained for years and its use is not generally
advisable.
This patch changes the default MDA binary to "maildrop", which is a more
modern and reasonable MDA to use.
We keep the courier.Procmail name for now, as that's internal, but it
may be changed later. Its documentation is updated to note that the
name is just for reference but it actually works with almost any binary.
Currently, if we can't find a mail server for a domain, we consider that
a transient failure (semi-accidentally, as we iterate over the (empty)
list of MXs and fall through the list).
It should be treated as a permanent error, in line with other DNS
issues (which is not ideal but seems to be generally accepted
behaviour).
This patch extends the SMTP courier to (optionally) do STS policy
checking when delivering mail.
As STS support is currently experimental, we gate this behind a flag and
is disabled by default.
Currently, we pick the first host in the MX list, and attempt delivery
there. If it fails, we just report the failure to the queue, which will
wait for some time and then try again.
This is not ideal: we should fall back to the other MXs in the list, as
the first host could be having issues for a long time, and not
attempting with the rest just delays delivery.
This patch implements the fallback, so we try all MXs before deciding to
report a failed delivery (unless, of course, an MX returned a permanent
failure).
When building the Debian package, the tests are run in such a way that
test/util/exitcode is not found, which causes them to fail.
This patch is a workaround which skips the test if test/util/exitcode is
not found; for now we should consider it temporary, until either the
Debian packaging is fixed, or we decide that its environment is
reasonably enough to make it permanent.
:
glog works fine and has great features, but it does not play along well
with systemd or standard log rotators (as it does the rotation itself).
So this patch replaces glog with a new logging module "log", which by
default logs to stderr, in a systemd-friendly manner.
Logging to files or syslog is still supported.
This patch introduces a new "domaininfo" package, which implements a
database with information about domains. In particular, it tracks
incoming and outgoing security levels.
That information is used in incoming and outgoing SMTP to prevent
downgrades.
The logic that picks the domain we use for EHLO does not need to live
within the TLS retry cycle, and makes it harder to understand.
This patch moves it out of the way, to improve readability.
This patch introduces expvar counters to chasquid and the queue
packages.
For now there's only a handful of counters, but they will be expanded in
future patches.
The SMTP courier was not properly closing the connection, and chasquid's
closing of incoming connections was not ideal (it was closing the
underlying one, not necessarily the active one, like in the case of a jump
to TLS).
This patch fixes both by adding the missing calls to Close.
This patch reviews various debug and informational messages, making more
uniform use of tracing, and extends the monitoring http server with
useful information like an index and a queue dump.
In the SMTP courier, we should always include a domain when saying hello, as
many MTAs will be pick about it.
An empty domain can happen if the envelope-from is <>. In that case, we fall
back to our hostname.
This patch makes the SMTP courier distinguish permanent errors, so the queue
can decide whether to retry or not.
It's based on the SMTP replies: 5xy is considerent permanent, anything else is
transient (including errors other than protocol issues).
This patch adds SMTPUTF8 to the SMTP courier.
It introduces a new internal/smtp package that extends Go's net/smtp with
SMTPUTF8 (in a very narrow way, not for general use), and then makes the
courier use it.
Also use an IDNA-safe version when saying hello, otherwise servers could
complain if the hostname is not ASCII, and at that stage we don't know if they
support it or not.
This patch implements the first steps of support for IDNA (Internationalized
Domain Names for Applications).
Internally, we maintain the byte-agnostic representation, including
configuration.
We support receiving IDNA mail over SMTP, which we convert to unicode for
internal handling.
Local deliveries are still kept agnostic.
For sending over SMTP, we use IDNA for DNS resolution, but there are some
corner cases pending in the SMTP courier that are tied with SMTPUTF8 and will
be fixed in subsequent patches.
Make the SMTP courier fall back to the A record when MX does not exist, as per
standard behaviour.
This is not implemented nicely, because Go's API does not give a clear signal
if the answer was that there are no MX records or something else happens.
For now, we implement it with a heuristic that should work pretty reliably,
but it's definitely not very nice.
Some servers, like postfix, will pay close attention to the domain we say as a
part of the EHLO.
By default, Go's smtp package will use "localhost", causing it to complain.
This patch fixes that by using the envelope-from's domain.
It's not clear if that's better than using what we are resolving to, but
that's much more involved so we're going to do this for now.
This patch performs some minor cleanups for things detected by "go vet":
- Remove one line of unreachable code.
- Don't leak contexts until their deadline expires, cancel them.
We can send and received with a <> "mail from", which is explicitly allowed.
Internally, we use "<>" to represent it. This requires special-casing in a
couple of places, but makes sure the handling is explicit, and we don't
accidentally confuse it with not having a source address.
This patch fixes some inconsistencies with this handling.
This patch makes the queue and couriers distinguish between permanent and
transient errors when delivering mail to individual recipients.
Pipe delivery errors are always permanent.
Procmail delivery errors are almost always permanent, except if the command
exited with code 75, which is an indication of transient.
SMTP delivery errors are almost always transient, except if the DNS resolution
for the domain failed.
This patch tidies up the Procmail courier:
- Move the configuration options to the courier instance, instead of using
global variables.
- Implement more useful string replacement options.
- Use exec.CommandContext for running the command with a timeout.
As a consequence of the first item, the queue now takes the couriers via its
constructor.
This patch adds a new test which makes chasquid send and receive email to/from
Exim.
To make it work we need to add two testing flags to the SMTP courier, which is
not ideal but doesn't muddle the code much.
The test is not very portable, and assumes an exim binary is available, but
does not have to be installed in the system. It includes a helper script to
fetch one from the Debian archives.
This patch does various minor style and simplification cleanups, fixing things
detected by tools such as go vet, gofmt -s, and golint.
There are no functional changes, this change is purely cosmetic, but will
enable us to run those tools more regularly now that their output is clean.
The routing courier is a nice idea in theory, but at least for now, we want
the queue to be aware of when a destination is local so we can implement
differentiated logic.
This may change in the future, though, but at the moment it's not clear that
the abstractions will be worth it.
So this patch removes it, and makes the queue do the routing. There is no
difference in how the two are handled yet, those will come in subsequent
patches.
This patch introduces an "envelope" package which, for now, provides simple
utilities for getting the user and domain of an address.
It also changes the couriers to use it (but other implementations remain, will
be moved over in subsequent patches).
This patch adds configuration options for the MDA binary and command line
arguments, and changes the (soon to be renamed) procmail courier to make use
of them.
Sanitize only lets some ascii characters go through, which is very unfriendly
to non-ascii usernames.
This patch changes it to be more inclusive, and filter out only selected
characters that may be problematic for the subprocesses, specially considering
UNIX shell environments. It's not meant to catch all possible problems, just
help prevent some common ones.
This patch introduces the couriers, which the queue uses to deliver mail.
We have a local courier (using procmail), a remote courier (uses SMTP), and a
router courier that decides which of the two to use based on a list of local
domains.
There are still a few things pending, but they all have their basic
functionality working and tested.
