ssh

ssh checks that an SSH server answers. The ssh check opens a TCP connection to the server's address and waits for the server's greeting. The greeting is the line with which every SSH server starts, such as SSH-2.0-OpenSSH_10.0p2 Debian-7. The check logs in nowhere, so it needs no account and no key.

Choose ssh over a plain tcp check when the port must be served by an SSH server: a tcp check is healthy when any program accepts the connection, and ssh is healthy only when that program greets as an SSH server does.

Inputs

The module takes one value, which a check writes after the module's name, and no other input.

  • the value of ssh (string; required): The address of the SSH server, as host:port.

When the check fails, and when it cannot check

A run of the check is healthy when the server sends its greeting within the check group's time_limit_per_run (timings of a check group). The message holds the greeting: the SSH server at 127.0.0.1:22 greeted with SSH-2.0-OpenSSH_10.0p2 Debian-7.

The run fails when:

  • The connection does not open. The message is the reason that the node's system gives, such as dial tcp 127.0.0.1:2: connect: connection refused, or the connection to 127.0.0.1:22 did not open within the time limit of the run.
  • The server sends something else, or closes the connection, before a greeting. The message is the server at 127.0.0.1:22 sent no SSH greeting. The body of the alert shows what the server sent, such as the server sent: 220 mail.example.com ESMTP.
  • No greeting arrives within time_limit_per_run. The message is the server at 127.0.0.1:22 sent no SSH greeting within the time limit of the run. A non-SSH server usually causes the run to reach the time limit because the server greets and then waits for the client.

Every failed run counts toward the check's failing time, which decides when the alert goes out. MMWatchdog can always run an ssh check, so the check is never unknown for a reason of its own. An ssh check is unknown only for the reasons that hold for every check, such as a run that does not end (scheduling and observed state).

Examples

The SSH server of the node itself

# /etc/mmwatchdog/check_groups/ssh_server.yml
check_groups:
  ssh_server:
    check_every: 1m
    time_limit_per_run: 10s
    alert_after_failing_for: 3m
    recovered_after_healthy_for: 1m
    checks:
      ssh_server_answers:
        ssh: 127.0.0.1:22
        alert_severity: sos

Once a minute, MMWatchdog on the node connects to port 22 of the node itself and waits up to 10 seconds for the greeting. MMWatchdog names itself to the server as an SSH client does and closes the connection. When the check has been failing for 3 minutes, MMWatchdog opens an incident and sends the sos alert. When the SSH server has then greeted for 1 minute, MMWatchdog resolves the incident and sends the recovery.

The SSH servers of other machines

# /etc/mmwatchdog/check_groups/ssh_of_other_machines.yml
check_groups:
  ssh_of_other_machines:
    check_every: 1m
    time_limit_per_run: 10s
    alert_after_failing_for: 2m
    recovered_after_healthy_for: 1m
    checks:
      ssh_server_of_web_1_answers:
        ssh: web-1.internal.example.com:22
        alert_severity: warn
      ssh_server_of_backup_answers:
        ssh: 10.0.0.9:2222
        alert_severity: warn

MMWatchdog on the node with /etc/mmwatchdog/check_groups/ssh_of_other_machines.yml connects to each remote machine over the network once a minute. The second server listens on port 2222. Each check has its own failing time and its own incident, so an alert names the machine whose SSH server does not answer.

How it works

  1. MMWatchdog opens a TCP connection to the address. The address is written as host:port, and the check has no default port.
  2. MMWatchdog reads lines from the server until one starts with SSH-. A server may send other lines before its greeting, and MMWatchdog keeps them for the body of the alert. MMWatchdog reads at most 16 KiB from the server.
  3. When the greeting arrives, MMWatchdog sends SSH-2.0-MMWatchdog, as an SSH client names itself, and closes the connection. So the server's log shows an ordinary connection that ended before a login, and no error.

MMWatchdog replaces every character that changes how text is shown, such as a control character, in the server's text.

OpenSSH from version 9.8 slows down an address that connects often and never logs in. A check every 30 seconds or every minute stays far below the connection rate that triggers the OpenSSH slowdown. Do not check an SSH server several times a second.

Errors and what they mean

The table below lists validation errors for the ssh input. mmwatchdog validate and mmwatchdog reload print each validation error after the configuration file's path, line number, and column number, such as /etc/mmwatchdog/check_groups/ssh_server.yml:9:9:.

The error Meaning of the error, and what to do
check_groups.ssh_server.checks.ssh_server_answers.ssh: address 127.0.0.1: missing port in address The address has no port. Write the address as host:port, such as 127.0.0.1:22. An empty value, ssh: "", gives check_groups.ssh_server.checks.ssh_server_answers.ssh: missing port in address.
check_groups.ssh_server.checks.ssh_server_answers.ssh: port must be a number from 1 to 65535 The port is not an integer from 1 to 65535.
check_groups.ssh_server.checks.ssh_server_answers.ssh: expected a string, found a mapping The check writes ssh as a mapping, such as ssh: {address: 127.0.0.1:22}. The value of ssh is the address itself: ssh: 127.0.0.1:22.

Tested on

  • Go tests: TestSSHCheck in internal/checksandalerts/protocol_checks_test.go runs the check against a stand-in server: a greeting after other lines, the line with which MMWatchdog names itself, a server of another protocol, and a server that says nothing until the time limit. These tests run on the machine that runs the suites and on every target distribution.
  • On a node: the services scenarios (tests/integration/service_scenarios.py) check the guest's own OpenSSH server, as the distribution installs it, on every target distribution. The healthy message holds the same greeting that ssh-keyscan shows, a closed port fails the check, and the SSH server's log shows no error for MMWatchdog's connections. The scenario also checks Postfix's SMTP port, which fails the check with sent no SSH greeting and shows Postfix's greeting in the body. The scenarios never stop the SSH server, because the test harness works through it.

The validation record has the latest results.

Why it works this way

  • The check reads the greeting and logs in nowhere. The greeting shows that an SSH server, and not only some program, listens on the configured port. A check that needs no key leaves no credential on the node.
  • MMWatchdog names itself before closing the TCP connection. A client that closes the connection without sending an SSH identification line causes an error in the server's log. MMWatchdog's connections must not fill the SSH server's log with errors.
  • The address is the value of ssh, as host:port, with no default port. A reader sees on each check where it connects (protocols and common server software).
  • MMWatchdog reads at most 16 KiB. A server that sends without end cannot make the check use all available memory on the node.
  • ssh is a core module. Checking that an SSH server answers is common work, so the check is built into mmwatchdog and tested with it.

This page is docs/reference/modules/ssh.md in the MMWatchdog repository.