We're running against the usage limits in Gitlab CI (500), and Github
Actions should have more (2000).
So this patch replaces Gitlab CI with Github actions for running
integration tests, and build and push Docker images (to Dockerhub and
Gitlab registry).
We'll see how the usage levels are in a few months.
This patch updates the shell scripts with some of the common best
practices, which should make them more resilient to unusual failures and
unexpected environments (in particular, directories with spaces).
Most of these were identified by shellcheck.
Cirrus CI runs golangci-lint, which is often way too noisy with false
positives and irrelevant checks. Last one was errcheck on
http.ResponseWriter.Write, but there have been others before.
Disable it for now, since it doesn't add enough value to justify the
introduced friction.
The integration tests spend a lot of time on some ancilliary actions,
which slows them down: generating certificates, adding users, and
waiting for things to happen.
To improve the performance of those actions, this patch:
- Makes (most) tests use plain passwords (-20%)
- Adds a certificate cache to reuse certs (-10%)
- Tightens the sleep loops (-5%)
In aggregate, this patch results in a speedup of the integration tests
of ~30-40%.
Note that some of the tests required adjusting the username, because
`chasquid-util user-add` would convert them to lowercase as per PRECIS
mapping.
This patch changes several internal packages to receive and pass tracing
annotations, making use of the new tracing library, so we can have
better debugging information.
This commit introduces a new tracing library, that replaces
golang.org/x/net/trace, and supports (amongts other thing) nested
traces.
This is a minimal change, future patches will make use of the new
functionality.
ioutil package was deprecated in Go 1.16, replace all uses with their
respective replacements.
This patch was generated with a combination of `gofmt -r`, `eg`, and
manually (for `ioutil.ReadDir`).
The latest version of some packages we depend on (golang.org/x/sys,
indirectly) do not support 1.15 anymore.
We kept 1.15 because it's the version on Debian stable, but that release
is more than 2y old, and newer versions are readily available in Debian
stable backports.
So this patch updates the minimum supported Go version to 1.17.
govulncheck will complain when run on older Go versions that are known
to have security issues. Github setup-go action has a cache for the
"latest" go version and sometimes the latest isn't really used.
So sometimes, this generates false positives because govulncheck is run
on an older Go version, which is not a problem in chasquid itself.
To prevent those false positives, this patch asks the Github CI to
always check for the latest Go version when running govulncheck.
Newer versions of msmtp now set In-Reply-To and References header,
causing t-16-spf test to fail because we expect them to be empty (for no
particular reason).
So this patch changes our expected DSN used for testing to ignore their
values.
This patch does a general pass updating Go modules to recent versions.
The main purpose is to make sure people building from source are using
relatively recent versions of our dependencies.
Having systemd open sockets and pass them to chasquid is neat, but also
adds some complexity to the default config, for very little value in
practice.
This patch simplifies the default config by having chasquid open the
sockets instead.
Note that systemd file descriptor passing continues to be supported, and
existing installations will not be affected.
By default, `git describe` uses only annotated tags. Since some may not
be annotated (like v1.10) this causes some scripts to pick a confusing
version identifier.
This patch fixes the issue by passing `--tags`, which means all tags
will be considered.
This patch is the result of running Go 1.19's `gofmt` on the codebase,
which automatically updates all Go doc comments to the new format.
https://tip.golang.org/doc/go1.19#go-doc
When running a diff for dkimpy's output, we expect that diff to exit with
non-zero code.
Unfortunately, the way we set that expectation (by prefixing the diff
invocation with `!` is incorrect.
Running `! diff ...` will not cause the hook to fail if diff exits with
0, instead `!` will cause the exit code to be ignored.
This patch fixes the problem by running `diff ... && exit 1` instead.
This was caught by shellcheck, https://www.shellcheck.net/wiki/SC2251.
We've accumulated a few linter issues around comments and a couple of
variable names.
While none of them is major, this patch cleans them up so it's easier to
go through the linter output, and we can start being more strict about
it.
This patch does a general pass updating Go modules to recent versions, and
regenerates the protobufs accordingly.
The main purpose is to make sure people building from source are using
relatively recent versions of our dependencies.
The current generate_cert helper was originally taken from Go's source,
and is more complex than we need it to be.
This patch replaces it with our own version, rewritten from scratch
independently.
This patch moves the test helper binaries to a "one directory per
helper" layout, and also makes them to be ignored in the coverage build
instead of all builds.
With this change, "go build ./..." will build all binaries including the
test helpers, which helps make sure that module manage automation also
considers them. In particular, this makes "go mod tidy" work fine.
Today, we do setfacl unconditionally; this can be a problem for
user-provided certificates because they may be located somewhere else.
This patch fixes the problem by only doing setfacl after renewing the
certificates.
Externally provided certificates will be untouched, and the user is
responsible for ensuring that chasquid can read them.
Thanks to Alex Ellwein (aellwein@github) for reporting this in
https://github.com/albertito/chasquid/issues/29!
Dovecot's `state_dir` usually defaults to be at `/var/lib/dovecot`, or a
similar system-wide path.
Under some conditions, our test Dovecot instance can fail, because it's
wanting to write to state_dir, but it is not writeable by us in the test
environment.
This was reported by foxcpp in
https://github.com/albertito/chasquid/issues/28.
This patch fixes the problem by setting a custom state_dir to be within
our testing directory.
Thanks to foxcpp for reporting this problem and suggesting a fix.
This patch updates the dependency on blitiri.com.ar/go/spf from v1.3.0
to v1.4.0.
There are no code changes needed, but the new version contains some
important fixes around lookup limits counting.
Some dkimpy versions have a bug where it can't parse the keys generated
by its own key generator. That causes the dkimpy test to fail.
See https://bugs.launchpad.net/dkimpy/+bug/1978835 for more details.
This patch adds a workaround which detects the buggy version, and skip
the test if needed.
This patch implements support for catch-all aliases, where users can add
a `*: destination` alias. Mails sent to unknown users (or other aliases)
will not be rejected, but sent to the indicated destination instead.
Please see https://github.com/albertito/chasquid/issues/23 and
https://github.com/albertito/chasquid/pull/24 for more discussion and
background.
Thanks to Alex Ellwein (aellwein@github) for the alternative patch and
help with testing; and to ThinkChaos (ThinkChaos@github) for help with
testing.
This patch does a general pass updating Go modules to recent versions, and
regenerates the protobufs accordingly.
The main purpose is to make sure people building from source are using
relatively recent versions of our dependencies.
This patch updates the dependency on blitiri.com.ar/go/spf from v1.2.0
to v1.3.0, which includes a few bug fixes.
There are no code changes needed, just some minor adjustment to the
tests due to error strings changing.
The go.mod "go" keyword is also bumped up to 1.15 since it's the minimum
supported version since commit e444fe1f (2021-10-05).
The install guide mentions that the `certs/` directory can be symlinked,
but doesn't cover permissions much, so it's easy for users to be
confused (like it happened in
https://github.com/albertito/chasquid/issues/26).
This patch adds a bit more details on how to set up certificates, to
hopefully reduce the chances of confusion.
This patch skips the resolution logic if the address is not local.
Today, the resolution logic handles that case transparently, and returns
the original email address, so this should be a no-op.
However, having an explicit early check makes the resolution logic more
robust, and will simplify future patches.
Note this also means that the `alias-resolve` hook is no longer run for
non-local aliases, which should also help simplify their implementation.
This patch simplifies the internal alias lookup logic, unifying it
across Resolve and Exists.
As part of this, the `alias-exists` hook is removed. It was redundant to
begin with, although it enabled a potential optimization, it isn't worth
the complexity. The timeout for execution of both was the same.
This change should be backwards-compatible because `alias-resolve` is
still used, and the semantics haven't changed.
If the `drop_characters` or `suffix_separators` options are set to "",
currently instead of the empty string, their default value is used instead.
This is a bug, and it also happens on other config options, but because
the others have to be set in order for chasquid to function, it's not a
problem in practice.
Thanks Björn Busse (bbusse@github) for finding and reporting this
problem, on irc and in https://github.com/albertito/chasquid/issues/25.
This patch fixes the problem by marking these fields explicitly
optional, which enables presence testing, as described in the protobuf
documentation:
https://github.com/protocolbuffers/protobuf/blob/master/docs/field_presence.md.
Document that only haproxy's PROXY protocol v1 is supported. This can
help users configure their instance and avoid trying to set up other
versions.
Thanks to Björn Busse (bbuse@github) for reporting this!
This patch adds tests for STS policy checks in combination with TLS
security levels.
This helps ensure we're detecting mis-matches of TLS status
(plain/insecure/secure) and STS policy enforcement.
The `which` command isn't guaranteed to be available, it is just
extremely common; `command -v` is the standard way to do find an
executable program. See https://lwn.net/Articles/874049/ for more
details.
This patch replaces the uses of `which` with `command -v`, which only
appears in a couple of tests.
When using STARTTLS, the SMTP courier needs to determine whether the
server certificates are valid or not.
Today, that's implemented via connecting once with full certificate
verification, and if that fails, reconnecting with verification
disabled.
This works okay in practice, but it is slower on insecure servers (due
to the reconnection), and some of them even complain because we connect
too frequently, causing delivery problems. The latter has only been
observed once, on the drv-berlin-brandenburg.de MX servers.
To improve on that situation, this patch makes the courier do the TLS
connection only once, and uses the verification results directly.
The behaviour of the server is otherwise unchanged. The only difference
is that when delivering mail to servers that have invalid certificates,
we now connect once instead of twice.
The tests are expanded to increase coverage for this particular case.
When resolving MX records, we need to distinguish between "no such
domain" and other kinds of errors. Before Go 1.13, this was not
possible, so we had a workaround that assumed any permanent error was a
"no such domain", which is not great, but functional.
Now that our minimum supported version is Go 1.15, we can remove the
workaround.
This patch replaces the workaround with proper logic using
DNSError.IsNotFound to identify NXDOMAIN results when resolving MX
records.
This requires to adjust a few tests, that used to work on environments
where resolving unknown domains (used for testing) returned a permanent
error, and now they no longer do so. Instead of relying on this
environmental property, we make the affected tests use our own DNS
server, which should make them more hermetic and reproducible.
tls.BuildNameToCertificate has been deprecated, and calling it is no
longer necessary since Go 1.14.
Now that our minimum supported Go version is 1.15, we can remove it.
In the Dovecot integration test, we can now simplify the configuration
as we assume Dovecot 2.3 is the minimum version supported for testing
(as that's the one from Debian stable at the moment).
Today, we use `golang.org/x/crypto/ssh/terminal` to read passwords. That
package is obsolete, replaced with `golang.org/x/term`.
We couldn't move them because term wasn't compatible with Go 1.11 which
was our oldest supported Go version.
Now that we moved to Go 1.15 as the oldest supported version, we can do
the update.
