There are a couple of places where it's handy to print TLS constants in
human-readable form.
To do so, we need functions that take TLS constants (usually in uint16 form)
and give us friendly strings.
Go's crypto/tls package does not provide us with this, so this patch
introduces a new module with that purpose.
Having the certificates inside the domain directory may cause some confusion,
as it's possible they're not for the same name (they should be for the MX we
serve as, not the domain itself).
So it's not a problem if we have domains with no certificates (we could be
their MX with another name), and we could have more than one certificate per
"domain" (if we act as MXs with different names).
So this patch moves the certificates out of the domains into a new certs/
directory, where we do a one-level deep lookup for the files.
While at it, change the names of the files to "fullchain.pem" and
"privkey.pem", which match the names generated by the letsencrypt client, to
make it easier to set up. There's no general convention for these names
anyway.
This patch performs some minor cleanups for things detected by "go vet":
- Remove one line of unreachable code.
- Don't leak contexts until their deadline expires, cancel them.
It's more convenient and in line with standard practice to fail RCPT TO if the
user does not exist.
This involves making the server and client aware of aliases, but it doesn't
end up being very convoluted, and simplifies other code.
This patch adds a print-config option that will parse a config and print it
(in text protobuf format).
It can be used to validate configurations, and see what the effective
configuration is (that is, including the defaults).
It's often useful to run the tests with the race detector (-race) enabled.
Unfortunately, building with it is too slow to enable unconditionally.
So for now this patch adds an option, in the form of an environment variable,
to enable it manually.
Today, if the aliases file does not exist when chasquid starts up, the entire
domain will be skipped from aliases resolution.
That's a bug, as it means we don't perform character and suffix replacements
for known domains, and is also an inconvenience as it forces us to reload the
daemon when adding a file for a known domain.
This patch fixes this by adding them unconditionally, even if the file does
not exist.
This patch adds a test for delivery status notifications and null address
deliveries, that check that chasquid can both receive and send DSNs.
To do this, we extend the mail_diff utility to support wildcards in the
comparisons, to skip over variable parts of the messages (like dates).
When we permanently failed to deliver to one or more recipients, send delivery
status notifications back to the sender.
To do this, we need to extend a couple of internal structures, to keep track
of the original destinations (so we can include them in the message, for
reference), and the hostname we're identifying ourselves as (this is arguable
but we're going with it for now, may change later).
We can send and received with a <> "mail from", which is explicitly allowed.
Internally, we use "<>" to represent it. This requires special-casing in a
couple of places, but makes sure the handling is explicit, and we don't
accidentally confuse it with not having a source address.
This patch fixes some inconsistencies with this handling.
This patch makes the queue and couriers distinguish between permanent and
transient errors when delivering mail to individual recipients.
Pipe delivery errors are always permanent.
Procmail delivery errors are almost always permanent, except if the command
exited with code 75, which is an indication of transient.
SMTP delivery errors are almost always transient, except if the DNS resolution
for the domain failed.
This patch removes chasquid-userdb and adds a more generic and extensive
chasquid-util, that supports various operations on user databases as well as
aliases lookups.
The code is not very pretty but for now I took a more practical approach, the
tool is ancillary and can be tidied up later.
This patch tidies up the Procmail courier:
- Move the configuration options to the courier instance, instead of using
global variables.
- Implement more useful string replacement options.
- Use exec.CommandContext for running the command with a timeout.
As a consequence of the first item, the queue now takes the couriers via its
constructor.
With the introduction of aliases, the queue may now be delivering mail to
pipes. This patch implements pipe delivery.
It uses a fixed 30s timeout for now, as these commands should really not take
much time, and we don't want to overly complicate the configuration for now.
aliases databases can be very useful, so this patch adds a package to parse
and resolve aliases.
It uses an existing, well known and widely used format for aliases, although
it doesn't necessarily match 100% any existing implementation at the moment.
This patch makes the queue read and write items to disk.
It uses protobuf for serialization. We serialize to text format to make
manual troubleshooting easier, as the performance difference is not very
relevant for us.
This patch adds a "data_dir" option, that chasquid will use to store
persistent data. It defaults to "/var/lib/chasquid" (for now, at least).
Users will come in subsequent patches.
In some cases, it's be useful to have references to directories relative
to the configuration itself.
So this patch makes chasquid do a Chdir into it, so we can assume it in
the rest of the code and config.
To avoid user and automation confusion, prefix the temporary files with
a ".". That way, if something scans the directory for files, it's less
likely to encounter one of our temporary files.
This will become very relevant in subsequent patches.
This is just a run of "go generate ./..." which re-generates the protocol
buffers code.
There is no actual change in the logic or APIs, it's just a rebuild with a
newer proto compiler. While it's not strictly necessary, it will help make
subsequent patches cleaner.
This patch adds checks that verify:
- The envelope from must match the authenticated user. This prevents
impersonation at the envelope level (while still allowing bounces, of
course).
- If the destination is remote, then the user must have completed
authentication. This prevents unauthorized relaying.
The patch ends up adjusting quite a few tests, as they were not written
considering these restrictions so they have to be changed accordingly.
We want to be able to distinguish between connections for SMTP and connections
for submission, so we can make different policy decisions.
To do that, we first make the configuration aware of the different kinds of
addresses. This is done in this patch in a backwards-incompatible way, but at
this point in time it is ok to do so.
Then, we extend systemd's socket passing library to support socket naming, so
we can tell the different sockets apart. This is done via the
LISTEN_FDNAMES/FileDescriptorName mechanism.
And finally we make the server and connection types aware of the socket mode.
This patch adds a new test which makes chasquid send and receive email to/from
Exim.
To make it work we need to add two testing flags to the SMTP courier, which is
not ideal but doesn't muddle the code much.
The test is not very portable, and assumes an exim binary is available, but
does not have to be installed in the system. It includes a helper script to
fetch one from the Debian archives.
This patch does various minor style and simplification cleanups, fixing things
detected by tools such as go vet, gofmt -s, and golint.
There are no functional changes, this change is purely cosmetic, but will
enable us to run those tools more regularly now that their output is clean.
This patch introduces a new directory, test/, which contains a simple local
end-to-end test which runs a chasquid binary and uses msmtp to send an email,
which is delivered locally.
As it's the first one, it adds a bunch of common infrastructure to simplify
writing these kinds of tests.
More end-to-end tests will follow, and it's expected that the common
infrastructure will also change significantly to accomodate their needs.
This patch adds a "chasquid-userdb" command line tool to check and add users
to chasquid's userdb files.
It's not pretty or very friendly, as it's meant to be used for testing for
now.
This patch makes chasquid flush logs periodically, and also before exiting,
even if it gets killed via the usual signals (including Ctrl-C).
Both things help troubleshooting when there is not much traffic, or we kill
the server manually (including during automated tests).
MAIL FROM commands usually come in the form of:
MAIL FROM:<from@from> BODY=8BITMIME
Note that there's extra parameters after the address, which for now we want to
ignore.
The current parser doesn't ignore them, and relies on mail.ParseAddress doing
so (that is, on mail.ParseAddress("<from> BODY=8BITMIME") working).
However, in go 1.7, the parser will get more strict and start to fail these
cases.
To fix this, we change the way we parse the line to use fmt.Sprintf, which is
much nicer than splitting by hand, and is more readable as well.
The routing courier is a nice idea in theory, but at least for now, we want
the queue to be aware of when a destination is local so we can implement
differentiated logic.
This may change in the future, though, but at the moment it's not clear that
the abstractions will be worth it.
So this patch removes it, and makes the queue do the routing. There is no
difference in how the two are handled yet, those will come in subsequent
patches.
This patch introduces an "envelope" package which, for now, provides simple
utilities for getting the user and domain of an address.
It also changes the couriers to use it (but other implementations remain, will
be moved over in subsequent patches).
This patch implements the AUTH SMTP command, using per-domain user databases.
Note that we don't really use or check the validation for anything, this is
just implementing the command itself.
This patch adds configuration options for the MDA binary and command line
arguments, and changes the (soon to be renamed) procmail courier to make use
of them.
This patch adds a package implementing a simple user database, called userdb.
It has a human readable space-separated extensible format, and uses scrypt for
password storage (but supports plain as well, for debugging and testing).
chasquid is not using it yet, that will come in later patches.
This patch adds a new internal/safeio package, which is meant to implement
safer version of some I/O related functions.
For now, only an atomic version of ioutil.WriteFile is implemented. More may
be added later if there's a need for them.
We implement the VRFY and EXPN commands, but as no-ops. The RFC allows this,
and most implementations seem to do it this way too.
While at it, merge the tests for simple commands like these into one.
Sanitize only lets some ascii characters go through, which is very unfriendly
to non-ascii usernames.
This patch changes it to be more inclusive, and filter out only selected
characters that may be problematic for the subprocesses, specially considering
UNIX shell environments. It's not meant to catch all possible problems, just
help prevent some common ones.
We will need a few string sets in different places, create it as a separate
package for convenience.
For now, it only implements strings, but it may be extended to other data
types in the future.
