Alberto Bertogli
6b64c9ccba
If chasquid runs out of file descriptors, that can cause problems accepting connections, writing to the queue, and even reloading configs. Some of those errors are fatal, as they should not happen under normal conditions. Since a remote client can open a lot of connections, this can end up consuming all file descriptors, and resulting in a denial of service. To reduce the impact of that kind of event, this patch makes chasquid limit the number of concurrent SMTP connections to half the maximum number of file descriptors. Connections over that limit will get closed immediately. The server will log (rate-limited) when such event happens, and new metrics make these events visible through monitoring too. This approach complements the previous patch, which made the server retry Accept errors, to reduce the impact of connection overloads. This mainly affects deployments with low file descriptor limits; most modern deployments have very high limits by default, so they are unlikely to experience this problem in the first place. Users who have to be very defensive can deploy OS-level safeguards (such as firewall rules to limit number of connections, etc.) which are more effective and tunable. Thanks to John JEAN <john@wargan.com> for finding and reporting this problem. The detailed report can be found in https://johnjean.io/advisories/chasquid-unauthenticated-remote-dos/.
Testing
Go tests
All Go packages have their own test suite, which provides easy and portable tests with decent enough coverage.
Integration tests
In the test/ directory there is a set of end to end integration tests,
written usually in a combination of bash and Python 3.
They're not expected to be portable, as that gets impractical very quickly, but should be usable in most Linux environments.
They provide critical coverage and integration tests for real life scenarios, as well as interactions with other software (like Exim or Dovecot).
Dependencies
The tests depend on the following things being installed on the system (listed as Debian package, for consistency):
msmtputil-linux(for/usr/bin/setsid)
Some individual tests have additional dependencies, and the tests are skipped if the dependencies are not found:
t-02-eximExim interaction tests:gettext-base(for/usr/bin/envsubst)- The
eximbinary available somewhere, but it doesn't have to be installed. There's a scriptget-exim4-debian.shto get it from the archives.
t-11-dovecotDovecot interaction tests:dovecot
t-15-driusan_dkimDKIM integration tests:- The
dkimsign dkimverify dkimkeygenbinaries, from driusan/dkim (no Debian package yet).
- The
t-18-haproxyHAProxy integration tests:haproxy
t-19-dkimpy: dkimpy integration tests:python3-dkim
t-23-prometheus: Prometheus integration tests:prometheus promtool
For some tests, python >= 3.5 is required; they will be skipped if it's not available.
Most tests depend on the
$HOSTALIASES
environment variable being functional, and will be skipped if it isn't. This
works by default in most Linux systems, but note that the use of
systemd-resolved can prevent it from working properly.
Stress tests
Also in the test/ directory there is a set of stress tests, which generate
load against chasquid to measure performance and resource consumption.
While they are not exhaustive, they are useful to catch regressions and track improvements on the main code paths.
Fuzz tests
Some Go packages also have instrumentation to run fuzz testing against them, using the Go native fuzzing support.
This is critical for packages that handle sensitive user input, such as authentication encoding, aliases files, or username normalization.
Command-line tool tests
Each command-line tool has their own set of tests, see the test.sh file on
their corresponding directories.
Docker
The test/Dockerfile can be used to set up a suitable isolated environment to
run the integration and stress tests.
This is very useful for automated tests, or running the integration tests in constrained or non supported environments.
Automated tests
There are two sets of automated tests which are run on every commit to upstream, and also weekly:
-
Github Actions, configured in the
.githubdirectory, runs the Go tests, the integration tests, checks for vulnerabilities, and finally also builds the public Docker images. -
Cirrus CI, configured in the
.cirrus.ymlfile, runs Go tests on FreeBSD.
Coverage
The test/cover.sh script runs the integration tests in coverage mode, and
produces a code coverage report in HTML format, for ease of analysis.
The target is to keep coverage of the chasquid binary above 90%.