fix(rest): make WebSocket monitor listener teardown safe under concurrency (#623)

* fix(rest): make WebSocket monitor listener teardown safe under concurrency

The commit-1 tests and race detector exposed three related defects in the
monitor listener's Close, whose check-then-act select was unsound:

1. With a full event queue (a slow or disconnecting client), the
   `case <-ml.c:` branch received a buffered event, mistook it for the
   closed-channel signal, and skipped both RemoveListener and close: the
   listener stayed registered, and the hub actor then blocked forever
   inside Receive on the next dispatch, wedging every hub consumer.
2. The read and write pumps both call Close when the connection drops;
   two callers could pass the select's default branch together and the
   second close panicked with "close of closed channel".
3. Even made exactly-once, closing the event queue was still unsafe:
   RemoveListener returns once its op is queued, not processed, so the
   hub actor could be mid-send into the queue when Close closed it
   (send on closed channel panic inside the actor).

Fix: Close deregisters and closes a `done` channel via sync.Once; the
event queue is never closed.  Receive and Delete select their enqueue
against done and return errListenerClosed when aborted, so the hub
removes the listener itself even if RemoveListener has not run yet, and
a blocked enqueue aborts instead of wedging the actor.  The write pump
exits on done and sends the closing frame.

Each defect has a regression test that fails against the previous code
(verified: no-op close delivered post-close events, concurrent closes
panicked, and close raced the actor's blocked send).  The write pump
keeps its c-close branch for safety; the black-box monitor tests from
commit 1 pass unchanged.

* fix(rest): close listener done channel before enqueueing removal

Copilot review on PR #623 correctly identified a remaining deadlock in
msgListener.Close: when the hub actor is blocked inside this listener's
Receive (full event queue) and the hub op queue is also full,
RemoveListener cannot enqueue, so execution never reached close(ml.done) --
the only operation able to release the actor.  Under the same slow-client
condition, concurrent dispatchers keeping the hub queue saturated made
Close hang permanently.

Fix: close the done channel (guarded by the existing sync.Once) before
queueing the RemoveListener op.  The blocked actor is released first, keeps
draining the op queue, and the removal can then be queued even when the
queue is full.  As a side effect, a Close racing a not-yet-processed
AddListener now also deregisters correctly, since the removal op can no
longer be overtaken by the earlier registration op.

Regression test TestMsgListenerCloseFullHubOpQueue wedges the actor in
Receive with a saturated op queue before calling Close; verified it fails
("Close deadlocked") against the previous ordering and passes with -race
after the fix, along with the full pkg/rest and pkg/msghub suites.

pkg/rest/socket_controller.go

@@ -1,7 +1,9 @@
 package rest
 
 import (
+	"errors"
 	"net/http"
+	"sync"
 	"time"
 
 	"github.com/gorilla/websocket"
@@ -43,9 +45,11 @@ var upgrader = websocket.Upgrader{
 type msgListener[T any] struct {
 	hub       *msghub.Hub
 	c         chan T
-	mailbox   string // Name of mailbox to monitor, "" == all mailboxes
+	done      chan struct{} // Closed by Close; stops event delivery and the write pump
+	mailbox   string        // Name of mailbox to monitor, "" == all mailboxes
 	toStored  func(msg event.MessageMetadata) T
 	toDeleted func(mailbox string, id string) T
+	closeOnce sync.Once
 }
 
 // newMsgListener creates a listener and registers it with the hub.  The optional mailbox
@@ -59,6 +63,7 @@ func newMsgListener[T any](
 	ml := &msgListener[T]{
 		hub:       hub,
 		c:         make(chan T, socketChanLen),
+		done:      make(chan struct{}),
 		mailbox:   mailbox,
 		toStored:  toStored,
 		toDeleted: toDeleted,
@@ -72,13 +77,22 @@ func (ml *msgListener[T]) watches(mailbox string) bool {
 	return ml.mailbox == "" || ml.mailbox == mailbox
 }
 
-// Receive handles an incoming message.
+// errListenerClosed is returned by Receive/Delete after Close; the hub removes listeners
+// that return errors, providing deregistration even if RemoveListener has not run yet.
+var errListenerClosed = errors.New("monitor listener closed")
+
+// Receive handles an incoming message.  If the listener is closed while its event queue is
+// full, the enqueue is abandoned rather than blocking the hub actor forever.
 func (ml *msgListener[T]) Receive(msg event.MessageMetadata) error {
 	if !ml.watches(msg.Mailbox) {
 		return nil
 	}
-	ml.c <- ml.toStored(msg)
-	return nil
+	select {
+	case ml.c <- ml.toStored(msg):
+		return nil
+	case <-ml.done:
+		return errListenerClosed
+	}
 }
 
 // Delete handles a deleted message.
@@ -86,19 +100,30 @@ func (ml *msgListener[T]) Delete(mailbox string, id string) error {
 	if ml.toDeleted == nil || !ml.watches(mailbox) {
 		return nil
 	}
-	ml.c <- ml.toDeleted(mailbox, id)
-	return nil
+	select {
+	case ml.c <- ml.toDeleted(mailbox, id):
+		return nil
+	case <-ml.done:
+		return errListenerClosed
+	}
 }
 
-// Close removes the listener registration and closes the event queue.
+// Close removes the listener registration and stops event delivery.  Safe for concurrent
+// use: the read and write pumps both invoke it when the connection drops.  The done channel
+// is closed before the removal is queued: a hub actor blocked in this listener's Receive or
+// Delete is released first, so it keeps draining the hub op queue and the removal can be
+// queued even when that queue is full.  The event queue is not closed: RemoveListener
+// returns once the op is queued, not processed, so a concurrent hub-actor send into the
+// queue could otherwise panic on the close; the done channel ends delivery and unblocks
+// any such send.
 func (ml *msgListener[T]) Close() {
-	select {
-	case <-ml.c:
-		// Already closed
-	default:
+	ml.closeOnce.Do(func() {
+		// Release a hub actor blocked in Receive or Delete before queueing the removal:
+		// with the removal queued first, a full hub op queue would block it here forever,
+		// never reaching the close below -- the only operation able to release the actor.
+		close(ml.done)
 		ml.hub.RemoveListener(ml)
-		close(ml.c)
-	}
+	})
 }
 
 // socketReadPump makes sure the websocket client is still connected, discarding any messages
@@ -140,9 +165,9 @@ func socketReadPump(conn *websocket.Conn, closeFn func()) {
 	}
 }
 
-// socketWritePump relays events from c to the peer as JSON messages until c is closed, sending
-// pings while idle to keep the connection alive.  closeFn is invoked on exit.
-func socketWritePump[T any](c chan T, conn *websocket.Conn, closeFn func()) {
+// socketWritePump relays events from c to the peer as JSON messages until done is closed,
+// sending pings while idle to keep the connection alive.  closeFn is invoked on exit.
+func socketWritePump[T any](c chan T, done <-chan struct{}, conn *websocket.Conn, closeFn func()) {
 	slog := log.With().Str("module", "rest").Str("proto", "WebSocket").
 		Str("remote", conn.RemoteAddr().String()).Logger()
 
@@ -168,6 +193,10 @@ func socketWritePump[T any](c chan T, conn *websocket.Conn, closeFn func()) {
 				// Write failed
 				return
 			}
+		case <-done:
+			// Listener closed, exit
+			_ = conn.WriteMessage(websocket.CloseMessage, []byte{})
+			return
 		case <-ticker.C:
 			// Send ping
 			if err := conn.SetWriteDeadline(time.Now().Add(socketWriteWait)); err != nil {
@@ -205,7 +234,7 @@ func serveMonitor[T any](
 		Str("remote", conn.RemoteAddr().String()).Msg("Upgraded to WebSocket")
 	// Create, register listener; then interact with conn.
 	ml := newListener(hub, mailbox)
-	go socketWritePump(ml.c, conn, ml.Close)
+	go socketWritePump(ml.c, ml.done, conn, ml.Close)
 	socketReadPump(conn, ml.Close)
 	return nil
 }