Nezha Monitoring: Unbounded WebSocket Streams — Resource Exhaustion DoS

Overview

1. Description

The Nezha dashboard exposes two endpoints that create long-lived WebSocket streams to monitored agents:

• POST /api/v1/terminal → createTerminal() (terminal.go:27-67)
• POST /api/v1/file → createFM() (fm.go:28-67)

Both call rpc.NezhaHandlerSingleton.CreateStream(streamId, ...) which inserts a new ioStreamContext into an unbounded map[string]*ioStreamContext (s.ioStreams in io_stream.go:59-67). There is no per-user rate limit, no global semaphore, and no per-server connection cap. Each stream allocates:

1. A ioStreamContext struct with several channels and sync primitives
2. Two goroutines via StartStream() (io_stream.go:358-369) — bidirectional io.CopyBuffer
3. A gRPC IOStream between the dashboard and the agent
4. An agent-side PTY/shell process

Vulnerable code:

terminal.go:27-67 — createTerminal:

func createTerminal(c *gin.Context) (*model.CreateTerminalResponse, error) {
    // ... validation ...
    rpc.NezhaHandlerSingleton.CreateStream(streamId, getUid(c), server.ID)
    // ... sends TaskTypeTerminalGRPC to agent ...
    return &model.CreateTerminalResponse{...}, nil
}

fm.go:28-67 — createFM:

func createFM(c *gin.Context) (*model.CreateFMResponse, error) {
    // ... validation ...
    rpc.NezhaHandlerSingleton.CreateStream(streamId, getUid(c), server.ID)
    // ... sends TaskTypeFM to agent ...
    return &model.CreateFMResponse{...}, nil
}

io_stream.go:55-67 — CreateStreamWithPurpose (inserts into unbounded map):

func (s *NezhaHandler) CreateStreamWithPurpose(...) {
    s.ioStreamMutex.Lock()
    defer s.ioStreamMutex.Unlock()
    s.ioStreams[streamId] = &ioStreamContext{
        creatorUserID:  creatorUserID,
        targetServerID: targetServerID,
        purpose:        purpose,
        userIoConnectCh:  make(chan struct{}),
        agentIoConnectCh: make(chan struct{}),
        revokedCh:        make(chan struct{}),
    }
}

io_stream.go:319-372 — StartStream spawns two goroutines per stream:

func (s *NezhaHandler) StartStream(streamId string, timeout time.Duration) error {
    // ...
    go func() {
        _, innerErr := io.CopyBuffer(userIo, agentIo, bp.buf)
        errCh <- innerErr
    }()
    go func() {
        _, innerErr := io.CopyBuffer(agentIo, userIo, bp.buf)
        errCh <- innerErr
    }()
    return <-errCh
}

The NezhaHandler.ioStreams map is initialized as a plain make(map[string]*ioStreamContext) in nezha.go:36 — no capacity limit, no eviction policy beyond explicit CloseStream / RevokeStreamsForServer.

The HasPermission check at terminal.go:41-43 and fm.go:43-45 controls access scope but does not limit creation volume. A user with ScopeServerExec (terminal) or ScopeServerRead+Write+Delete (file manager) can open unlimited streams.

2. PoC

A conceptual attack (no Docker needed):

# As an authenticated user with a valid JWT or PAT:
for i in {1..1000}; do
  curl -X POST "https://dashboard.example.com/api/v1/terminal" \
    -H "Authorization: Bearer $JWT" \
    -H "Content-Type: application/json" \
    -d '{"server_id": 1}' &
done
wait

Each request:

• Creates a new stream entry in ioStreams
• Sends a TaskTypeTerminalGRPC task to the agent
• When the WebSocket attachment occurs (GET /ws/terminal/{id}), spawns 2 goroutines for I/O relay and allocates a 1 MB buffer per goroutine

The attack targets three resource domains:

1. Dashboard memory/goroutines — each stream adds goroutines, channels, and buffers
2. Agent resources — each stream spawns a PTY/shell process on the monitored server
3. gRPC connection pool — concurrent IOStreams consume gRPC multiplexing capacity

The POST /file (createFM) endpoint provides an alternative path with the same unbounded behavior, using ScopeServerRead+Write+Delete instead of ScopeServerExec.

3. Impact

• Denial of Service against the dashboard: memory exhaustion, goroutine starvation, or gRPC stream table overflow from rapid stream creation
• Denial of Service against monitored agents: each terminal session spawns a PTY process on the agent — an attacker can crash or degrade all agents behind the dashboard
• Operational cascade: if the dashboard OOMs, all agent monitoring and alerting is lost
• PAT connection-registry bypass: rapid create-connect-disconnect cycles may evade cleanup tracking

The attack requires only authenticated access with standard scopes — no special privileges. Any team member with terminal access to a server can DoS the entire infrastructure.

4. Remediation

Implement layered rate limiting and concurrency control:

1. Per-user stream cap in CreateStream — reject if the user already has N active streams (e.g., 10 per user):

   func (s *NezhaHandler) CreateStreamWithPurpose(...) {
       s.ioStreamMutex.Lock()
       defer s.ioStreamMutex.Unlock()
       count := 0
       for _, ctx := range s.ioStreams {
           if ctx.creatorUserID == creatorUserID { count++ }
       }
       if count >= maxStreamsPerUser { return error }
       // ... existing code ...
   }
   

2. Per-server semaphore — limit concurrent streams to any single server (e.g., 20 per server)

3. Rate limiter on createTerminal and createFM — mirror the existing MCP rate limiter (mcp_ratelimit.go) for legacy WebSocket endpoints

4. Add a configurable MaxStreamsPerUser / MaxStreamsPerServer setting so operators can tune limits without code changes

Affected packages

• github.com/nezhahq/nezha >= 1.0.0, < 2.2.0

Remediation

Upgrade to a patched release:

• github.com/nezhahq/nezha 2.2.0