smtpsrv: Set a limit on concurrent SMTP connections

If chasquid runs out of file descriptors, that can cause problems
accepting connections, writing to the queue, and even reloading configs.
Some of those errors are fatal, as they should not happen under normal
conditions.

Since a remote client can open a lot of connections, this can end up
consuming all file descriptors, and resulting in a denial of service. To
reduce the impact of that kind of event, this patch makes chasquid limit
the number of concurrent SMTP connections to half the maximum number of
file descriptors.

Connections over that limit will get closed immediately. The server
will log (rate-limited) when such event happens, and new metrics
make these events visible through monitoring too.

This approach complements the previous patch, which made the server
retry Accept errors, to reduce the impact of connection overloads.

This mainly affects deployments with low file descriptor limits; most
modern deployments have very high limits by default, so they are
unlikely to experience this problem in the first place.

Users who have to be very defensive can deploy OS-level safeguards (such
as firewall rules to limit number of connections, etc.) which are more
effective and tunable.

Thanks to John JEAN <john@wargan.com> for finding and reporting this
problem. The detailed report can be found in
https://johnjean.io/advisories/chasquid-unauthenticated-remote-dos/.

test/t-24-connection_limit/config/chasquid.conf

@@ -0,0 +1,10 @@
+smtp_address: ":1025"
+submission_address: ":1587"
+submission_over_tls_address: ":1465"
+monitoring_address: ":1099"
+
+mail_delivery_agent_bin: "test-mda"
+mail_delivery_agent_args: "%to%"
+
+data_dir: "../.data"
+mail_log_path: "../.logs/mail_log"

test/t-24-connection_limit/content

@@ -0,0 +1,4 @@
+Subject: Prueba desde el test
+
+Crece desde el test el futuro
+Crece desde el test

test/t-24-connection_limit/run.sh

@@ -0,0 +1,64 @@
+#!/bin/bash
+
+set -e
+. "$(dirname "$0")/../util/lib.sh"
+
+init
+
+generate_certs_for testserver
+add_user user@testserver secretpassword
+
+# Build in advance, so the build doesn't run under the low fd limit below.
+go-build-cached "${TBASE}/../../"
+
+# Run chasquid with a low file descriptor limit, which is easy to exhaust.
+# Reload very often: reloading needs to open files, so if we ran out of file
+# descriptors, chasquid would fail to reload and exit.
+mkdir -p .logs
+(
+	ulimit -n 64
+	chasquid -v=2 --logfile=.logs/chasquid.log --config_dir=config \
+		--testing__reload_every=50ms
+) &
+CHASQUID_PID=$!
+wait_until_ready 1025
+
+# The connection limit should be half the file descriptor limit.
+if ! grep -q "Incoming connection limit: 32$" .logs/chasquid.log; then
+	fail "unexpected connection limit"
+fi
+
+# Open and hold as many connections as the limit allows.
+conngen -addr=localhost:1025 -count=32 -wait > .logs/conngen.log 2>&1 &
+CONNGEN_PID=$!
+wait_until "grep -q 'done, created' .logs/conngen.log"
+
+# Open and hold more connections than there are file descriptors left. The
+# server should close them right away (so we read EOF, not time out).
+for _ in $(seq 1 40); do
+	exec {fd}<>/dev/tcp/localhost/1025
+	if read -r -t 5 line <&"$fd"; then
+		fail "connection over the limit got: $line"
+	elif [ $? -gt 128 ]; then
+		fail "connection over the limit was not closed"
+	fi
+done
+
+if ! grep -q "Incoming connection limit (32) reached" .logs/chasquid.log; then
+	fail "connection limit not logged"
+fi
+
+# Give it time to reload a few times, and check it's still running.
+sleep 0.5
+if ! kill -0 $CHASQUID_PID; then
+	fail "chasquid is not running"
+fi
+
+# Once the connections are closed, it should accept new ones, and work
+# normally. The server notices the connections were closed asynchronously, so
+# we retry until it works.
+kill $CONNGEN_PID
+wait_until "smtpc user@testserver < content 2>/dev/null"
+wait_for_file .mail/user@testserver
+
+success

test/t-24-connection_limit/smtpc.conf

@@ -0,0 +1,4 @@
+addr localhost:1465
+server_cert config/certs/testserver/fullchain.pem
+user user@testserver
+password secretpassword