ssh
ssh checks that an SSH server answers. The ssh check opens a TCP
connection to the server's address and waits for the server's greeting.
The greeting is the line with which every SSH server starts, such as
SSH-2.0-OpenSSH_10.0p2 Debian-7. The check logs in nowhere, so it needs no
account and no key.
Choose ssh over a plain tcp check when the port must be
served by an SSH server: a tcp check is healthy when any program accepts
the connection, and ssh is healthy only when that program greets as an
SSH server does.
Inputs
The module takes one value, which a check writes after the module's name, and no other input.
- the value of
ssh(string; required): The address of the SSH server, as host:port.
When the check fails, and when it cannot check
A run of the check is healthy when the server sends its greeting within the
check group's time_limit_per_run
(timings of a check group).
The message holds the greeting:
the SSH server at 127.0.0.1:22 greeted with SSH-2.0-OpenSSH_10.0p2 Debian-7.
The run fails when:
- The connection does not open. The message is the reason that the
node's system gives, such as
dial tcp 127.0.0.1:2: connect: connection refused, orthe connection to 127.0.0.1:22 did not open within the time limit of the run. - The server sends something else, or closes the connection, before a
greeting. The message is
the server at 127.0.0.1:22 sent no SSH greeting. The body of the alert shows what the server sent, such asthe server sent: 220 mail.example.com ESMTP. - No greeting arrives within
time_limit_per_run. The message isthe server at 127.0.0.1:22 sent no SSH greeting within the time limit of the run. A non-SSH server usually causes the run to reach the time limit because the server greets and then waits for the client.
Every failed run counts toward the check's
failing time, which decides when the
alert goes out. MMWatchdog can always run an ssh check, so the check is never
unknown for a reason of its own. An ssh check is unknown only for the reasons that
hold for every check, such as a run that does not end
(scheduling and observed state).
Examples
The SSH server of the node itself
# /etc/mmwatchdog/check_groups/ssh_server.yml
check_groups:
ssh_server:
check_every: 1m
time_limit_per_run: 10s
alert_after_failing_for: 3m
recovered_after_healthy_for: 1m
checks:
ssh_server_answers:
ssh: 127.0.0.1:22
alert_severity: sos
Once a minute, MMWatchdog on the node connects to port 22 of the node itself and
waits up to 10 seconds for the greeting. MMWatchdog names itself to the
server as an SSH client does and closes the connection. When the check has been
failing for 3 minutes, MMWatchdog opens an incident and sends the sos alert.
When the SSH server has then greeted for 1 minute,
MMWatchdog resolves the incident and sends the recovery.
The SSH servers of other machines
# /etc/mmwatchdog/check_groups/ssh_of_other_machines.yml
check_groups:
ssh_of_other_machines:
check_every: 1m
time_limit_per_run: 10s
alert_after_failing_for: 2m
recovered_after_healthy_for: 1m
checks:
ssh_server_of_web_1_answers:
ssh: web-1.internal.example.com:22
alert_severity: warn
ssh_server_of_backup_answers:
ssh: 10.0.0.9:2222
alert_severity: warn
MMWatchdog on the node with
/etc/mmwatchdog/check_groups/ssh_of_other_machines.yml connects to each
remote machine over the network once a minute. The second server listens
on port 2222. Each check has its own failing time and its own incident,
so an alert names the machine whose SSH server does not answer.
How it works
- MMWatchdog opens a TCP connection to the address. The address is written as host:port, and the check has no default port.
- MMWatchdog reads lines from the server until one starts with
SSH-. A server may send other lines before its greeting, and MMWatchdog keeps them for the body of the alert. MMWatchdog reads at most 16 KiB from the server. - When the greeting arrives, MMWatchdog sends
SSH-2.0-MMWatchdog, as an SSH client names itself, and closes the connection. So the server's log shows an ordinary connection that ended before a login, and no error.
MMWatchdog replaces every character that changes how text is shown, such as a control character, in the server's text.
OpenSSH from version 9.8 slows down an address that connects often and never logs in. A check every 30 seconds or every minute stays far below the connection rate that triggers the OpenSSH slowdown. Do not check an SSH server several times a second.
Errors and what they mean
The table below lists validation errors for the ssh input.
mmwatchdog validate and mmwatchdog reload print each validation error
after the configuration file's path, line number, and column number, such as
/etc/mmwatchdog/check_groups/ssh_server.yml:9:9:.
| The error | Meaning of the error, and what to do |
|---|---|
check_groups.ssh_server.checks.ssh_server_answers.ssh: address 127.0.0.1: missing port in address |
The address has no port. Write the address as host:port, such as 127.0.0.1:22. An empty value, ssh: "", gives check_groups.ssh_server.checks.ssh_server_answers.ssh: missing port in address. |
check_groups.ssh_server.checks.ssh_server_answers.ssh: port must be a number from 1 to 65535 |
The port is not an integer from 1 to 65535. |
check_groups.ssh_server.checks.ssh_server_answers.ssh: expected a string, found a mapping |
The check writes ssh as a mapping, such as ssh: {address: 127.0.0.1:22}. The value of ssh is the address itself: ssh: 127.0.0.1:22. |
Tested on
- Go tests:
TestSSHCheckininternal/checksandalerts/protocol_checks_test.goruns the check against a stand-in server: a greeting after other lines, the line with which MMWatchdog names itself, a server of another protocol, and a server that says nothing until the time limit. These tests run on the machine that runs the suites and on every target distribution. - On a node: the services scenarios (
tests/integration/service_scenarios.py) check the guest's own OpenSSH server, as the distribution installs it, on every target distribution. The healthy message holds the same greeting thatssh-keyscanshows, a closed port fails the check, and the SSH server's log shows no error for MMWatchdog's connections. The scenario also checks Postfix's SMTP port, which fails the check withsent no SSH greetingand shows Postfix's greeting in the body. The scenarios never stop the SSH server, because the test harness works through it.
The validation record has the latest results.
Why it works this way
- The check reads the greeting and logs in nowhere. The greeting shows that an SSH server, and not only some program, listens on the configured port. A check that needs no key leaves no credential on the node.
- MMWatchdog names itself before closing the TCP connection. A client that closes the connection without sending an SSH identification line causes an error in the server's log. MMWatchdog's connections must not fill the SSH server's log with errors.
- The address is the value of
ssh, as host:port, with no default port. A reader sees on each check where it connects (protocols and common server software). - MMWatchdog reads at most 16 KiB. A server that sends without end cannot make the check use all available memory on the node.
sshis a core module. Checking that an SSH server answers is common work, so the check is built intommwatchdogand tested with it.
This page is docs/reference/modules/ssh.md in the MMWatchdog repository.