tcp
tcp opens a TCP connection from the node to an address and port. The
check is healthy when the connection opens. When the tcp configuration is
written as a mapping, the check can also send a text and expect a reply.
The exchange is for a server that speaks a simple protocol of lines.
Choose tcp for a port that has to accept connections, such as the port of
a database server, and for a server of the team's own that answers a line of
text. A server that answers over UDP has udp. A program
that listens on a Unix socket, a file, has
unix_socket. A service whose protocol MMWatchdog knows has
a module of its own, which checks more than the connection: such as
http, tls, ssh, and
smtp.
Inputs
A check writes the module's inputs as a mapping below the module's name. In place of the mapping, it can write one value after the name:
- the value of
tcp(string): The address alone, as host:port, in place of the mapping of inputs.
The inputs of the mapping:
address(string; required): The address to connect to, as host:port.text_to_send(string; optional): A text that the check sends once the connection is open. Without it the check sends nothing.reply_contains(string; optional): A text that the server's reply must contain. Without a rule for the reply, the check reads nothing.reply_matches_regex(string; optional): A regular expression that the server's reply must match.
When the check fails, and when it cannot check
Without a rule for the reply, a run is healthy when the connection opens
and any configured text_to_send is sent,
with the message TCP connection accepted. The run fails when the
connection does not open. A connection can fail because no program listens
on the port or because a firewall refuses the connection. The run also
fails when the connection does not open within the check group's
time_limit_per_run. The message is the cause in the node's system's words,
such as dial tcp 127.0.0.1:5432: connect: connection refused, or, for a
connection that did not open in time, the connection to 127.0.0.1:5432 did not open within 5s (time_limit_per_run).
With reply_contains, reply_matches_regex, or both rules configured, a
run is healthy when the reply satisfies every configured rule.
The message is the reply holds what the check expects.
Besides a connection that does not open, the run fails when:
- the reply fails at least one configured rule when the server closes
the connection or MMWatchdog has read 64 KiB. The message names the first
unsatisfied rule:
the reply is not one that contains "READY", orthe reply is not one that matches the regular expression READY \d+ jobs; - the reply still fails at least one configured rule when
time_limit_per_runexpires: the message names the first unsatisfied rule, as in the preceding item, followed by; MMWatchdog waited until the time limit of the run; - the server sent nothing:
the server sent no reply, and the check expects one that contains "READY"; - the connection breaks in another way, such as a reset by the server, with the cause in Go's words.
The failed run shows what the server sent, in its details and in the
body of its alert, such
as the server sent: myapp 1.0 READY 3 jobs. In the displayed reply,
MMWatchdog replaces every control character and shortens the
reply to about 180 bytes.
A tcp check has no unknown result of its own: every answer of the server,
and every connection that does not open, is healthy or failed. Like every
check, a tcp check is unknown when the run does not end
(a run that does not end). A
failed run starts or continues the check's
failing time. tcp reports no
measurement, so a tcp check takes no limit.
Examples
A port accepts a connection
# /etc/mmwatchdog/check_groups/database_port.yml
check_groups:
database_port:
check_every: 30s
time_limit_per_run: 5s
alert_after_failing_for: 1m
recovered_after_healthy_for: 30s
tcp: 127.0.0.1:5432
alert_severity: sos
The check is written directly on the check group, with the address alone as
the value of tcp, so status shows it as database_port/tcp. Every 30
seconds, MMWatchdog on the node connects to port 5432 of the node itself and closes
the connection again. MMWatchdog sends nothing and reads nothing. When no
connection has opened for 1 minute, MMWatchdog sends the sos alert.
A server answers a text
# /etc/mmwatchdog/check_groups/myapp_ports.yml
check_groups:
myapp_ports:
check_every: 30s
time_limit_per_run: 5s
alert_after_failing_for: 1m
recovered_after_healthy_for: 30s
checks:
myapp_answers_ready_over_tcp:
tcp:
address: 127.0.0.1:7000
text_to_send: "STATUS\r\n"
reply_contains: READY
alert_severity: sos
myapp_greets_with_its_version:
tcp:
address: 127.0.0.1:7000
reply_matches_regex: '^myapp 1\.'
alert_severity: warn
In each run of the first check, MMWatchdog on the node connects to port 7000, sends
the line STATUS, and reads the reply until it contains READY. The run is
healthy as soon as the reply contains READY. MMWatchdog then closes the
connection. The second check sends nothing. MMWatchdog reads the server's
greeting for the second check. The second check is healthy when the
greeting starts with myapp 1..
How it works
- The address is host:port, with a port from 1 to 65535 and no default
port. A host name with several addresses, such as
localhostwith one of each IP version, is tried at each address until a connection opens. - Written as a value,
tcp: 127.0.0.1:5432, the check only connects. Written as a mapping, the check takesaddressand the three inputs below. text_to_sendis sent once, as soon as the connection is open. A text with a line break is written in double quotes, with\r\nor\nfor the break, as the example shows: most line protocols expect one.- Without a rule, the check reads nothing, also when it sends a text: it is healthy when the connection opens and any configured text is sent.
- With a rule, the check reads until the rules hold, and stops there, so
a server that keeps the connection open does not hold the check up.
MMWatchdog also stops reading when the server closes the connection,
after reading 64 KiB, or at
time_limit_per_run. - A check can state
reply_containsandreply_matches_regex. When a check statesreply_containsandreply_matches_regex, the reply must satisfy each rule.reply_containsis a plain text.reply_matches_regexhas the syntax of RE2 and matches anywhere in the reply unless it is anchored, as^myapp 1\.is. - The check sends and matches text, not binary protocol messages. A
protocol that needs non-text bytes, such as DNS, needs a protocol-specific
check (
dns) or a module written by your team for your deployment project.
Errors and what they mean
mmwatchdog validate and mmwatchdog reload print each error after the file, the line, and
the column, which the table leaves out. The examples are for a check group c with a
direct tcp check.
| Message | What the message means, and what to do |
|---|---|
check_groups.c.tcp.address: address 127.0.0.1: missing port in address |
The address has no port. Write the address as host:port, such as 127.0.0.1:5432 |
check_groups.c.tcp.address: port must be a number from 1 to 65535 |
The port is not a number from 1 to 65535 |
check_groups.c.tcp.address: missing port in address |
The mapping has no address, or it is empty |
check_groups.c.tcp.text_to_send: cannot be empty |
text_to_send is written with an empty text. Leave out text_to_send to send nothing |
check_groups.c.tcp.reply_contains: cannot be empty |
An empty reply_contains or reply_matches_regex is rejected |
check_groups.c.tcp.reply_matches_regex: is no valid regular expression: error parsing regexp: ... |
The regular expression is not valid RE2. The text after the colon says why |
check_groups.c.tcp.expect: unknown field "expect": the inputs of the module tcp are address, text_to_send, reply_contains, reply_matches_regex |
The mapping has an input that tcp does not have. Remove expect, or write an input that the message names |
check_groups.c.tcp: expected a mapping, or an address as host:port, found a list |
The check is written as a list. Write the address alone, such as tcp: 127.0.0.1:5432, or a mapping of the inputs |
Tested on
- Go tests in
internal/checksandalerts, each against a stand-in server on the test machine:TestTCPCheckSendsATextAndExpectsAReply(a reply in two parts, both rules, the greeting alone, a server that closes, one that says nothing until the time limit, a closed port, and validation),TestChecksOfServersThatDoNotAnswerAsExpected(a server that answers a part and waits, with what it sent in the details, and a port where nothing listens), andTestTCPExchangePrimitiveininternal/primitives. They ran on macOS ARM64 and, as the native suite, on the AMD64 guest of every target distribution. - The acceptance suite (
tests/integration/acceptance.py), on every target distribution: atcpcheck with the address alone, healthy against a test service, also with MMX absent at runtime. - The services suite (
tests/integration/service_scenarios.py), on every target distribution: a small TCP server of the test, with a check that sends a text and keeps both rules, one that reads the greeting, one that expects a text that never comes, with what the server sent in its details, and a refused connection once the server is stopped. - Not verified on a real node: a server that answers a part and then waits. Go tests cover a server that answers a part and then waits (validation record).
Why it works this way
-
A reply rule needs at least one received byte. A regular expression that matches empty text cannot make a silent server healthy.
-
No default port. A reader sees on each check where it connects (protocols and common server software).
-
The check reads until the rules hold, not until the server closes. A server of a line protocol often keeps the connection open, and the check must not wait for it until the time limit.
-
Without a reply rule, MMWatchdog reads nothing. A port that accepts connections is all that such a check asks, and a server that sends nothing first is still healthy.
-
What the server sent is in the alert. A reader sees at once whether the server said something else or nothing.
-
MMWatchdog reads at most 64 KiB of a reply, whatever the server sends.
This page is docs/reference/modules/tcp.md in the MMWatchdog repository.