tcp

tcp opens a TCP connection from the node to an address and port. The check is healthy when the connection opens. When the tcp configuration is written as a mapping, the check can also send a text and expect a reply. The exchange is for a server that speaks a simple protocol of lines.

Choose tcp for a port that has to accept connections, such as the port of a database server, and for a server of the team's own that answers a line of text. A server that answers over UDP has udp. A program that listens on a Unix socket, a file, has unix_socket. A service whose protocol MMWatchdog knows has a module of its own, which checks more than the connection: such as http, tls, ssh, and smtp.

Inputs

A check writes the module's inputs as a mapping below the module's name. In place of the mapping, it can write one value after the name:

  • the value of tcp (string): The address alone, as host:port, in place of the mapping of inputs.

The inputs of the mapping:

  • address (string; required): The address to connect to, as host:port.
  • text_to_send (string; optional): A text that the check sends once the connection is open. Without it the check sends nothing.
  • reply_contains (string; optional): A text that the server's reply must contain. Without a rule for the reply, the check reads nothing.
  • reply_matches_regex (string; optional): A regular expression that the server's reply must match.

When the check fails, and when it cannot check

Without a rule for the reply, a run is healthy when the connection opens and any configured text_to_send is sent, with the message TCP connection accepted. The run fails when the connection does not open. A connection can fail because no program listens on the port or because a firewall refuses the connection. The run also fails when the connection does not open within the check group's time_limit_per_run. The message is the cause in the node's system's words, such as dial tcp 127.0.0.1:5432: connect: connection refused, or, for a connection that did not open in time, the connection to 127.0.0.1:5432 did not open within 5s (time_limit_per_run).

With reply_contains, reply_matches_regex, or both rules configured, a run is healthy when the reply satisfies every configured rule. The message is the reply holds what the check expects. Besides a connection that does not open, the run fails when:

  • the reply fails at least one configured rule when the server closes the connection or MMWatchdog has read 64 KiB. The message names the first unsatisfied rule: the reply is not one that contains "READY", or the reply is not one that matches the regular expression READY \d+ jobs;
  • the reply still fails at least one configured rule when time_limit_per_run expires: the message names the first unsatisfied rule, as in the preceding item, followed by ; MMWatchdog waited until the time limit of the run;
  • the server sent nothing: the server sent no reply, and the check expects one that contains "READY";
  • the connection breaks in another way, such as a reset by the server, with the cause in Go's words.

The failed run shows what the server sent, in its details and in the body of its alert, such as the server sent: myapp 1.0 READY 3 jobs. In the displayed reply, MMWatchdog replaces every control character and shortens the reply to about 180 bytes.

A tcp check has no unknown result of its own: every answer of the server, and every connection that does not open, is healthy or failed. Like every check, a tcp check is unknown when the run does not end (a run that does not end). A failed run starts or continues the check's failing time. tcp reports no measurement, so a tcp check takes no limit.

Examples

A port accepts a connection

# /etc/mmwatchdog/check_groups/database_port.yml
check_groups:
  database_port:
    check_every: 30s
    time_limit_per_run: 5s
    alert_after_failing_for: 1m
    recovered_after_healthy_for: 30s
    tcp: 127.0.0.1:5432
    alert_severity: sos

The check is written directly on the check group, with the address alone as the value of tcp, so status shows it as database_port/tcp. Every 30 seconds, MMWatchdog on the node connects to port 5432 of the node itself and closes the connection again. MMWatchdog sends nothing and reads nothing. When no connection has opened for 1 minute, MMWatchdog sends the sos alert.

A server answers a text

# /etc/mmwatchdog/check_groups/myapp_ports.yml
check_groups:
  myapp_ports:
    check_every: 30s
    time_limit_per_run: 5s
    alert_after_failing_for: 1m
    recovered_after_healthy_for: 30s
    checks:
      myapp_answers_ready_over_tcp:
        tcp:
          address: 127.0.0.1:7000
          text_to_send: "STATUS\r\n"
          reply_contains: READY
        alert_severity: sos
      myapp_greets_with_its_version:
        tcp:
          address: 127.0.0.1:7000
          reply_matches_regex: '^myapp 1\.'
        alert_severity: warn

In each run of the first check, MMWatchdog on the node connects to port 7000, sends the line STATUS, and reads the reply until it contains READY. The run is healthy as soon as the reply contains READY. MMWatchdog then closes the connection. The second check sends nothing. MMWatchdog reads the server's greeting for the second check. The second check is healthy when the greeting starts with myapp 1..

How it works

  • The address is host:port, with a port from 1 to 65535 and no default port. A host name with several addresses, such as localhost with one of each IP version, is tried at each address until a connection opens.
  • Written as a value, tcp: 127.0.0.1:5432, the check only connects. Written as a mapping, the check takes address and the three inputs below.
  • text_to_send is sent once, as soon as the connection is open. A text with a line break is written in double quotes, with \r\n or \n for the break, as the example shows: most line protocols expect one.
  • Without a rule, the check reads nothing, also when it sends a text: it is healthy when the connection opens and any configured text is sent.
  • With a rule, the check reads until the rules hold, and stops there, so a server that keeps the connection open does not hold the check up. MMWatchdog also stops reading when the server closes the connection, after reading 64 KiB, or at time_limit_per_run.
  • A check can state reply_contains and reply_matches_regex. When a check states reply_contains and reply_matches_regex, the reply must satisfy each rule. reply_contains is a plain text. reply_matches_regex has the syntax of RE2 and matches anywhere in the reply unless it is anchored, as ^myapp 1\. is.
  • The check sends and matches text, not binary protocol messages. A protocol that needs non-text bytes, such as DNS, needs a protocol-specific check (dns) or a module written by your team for your deployment project.

Errors and what they mean

mmwatchdog validate and mmwatchdog reload print each error after the file, the line, and the column, which the table leaves out. The examples are for a check group c with a direct tcp check.

Message What the message means, and what to do
check_groups.c.tcp.address: address 127.0.0.1: missing port in address The address has no port. Write the address as host:port, such as 127.0.0.1:5432
check_groups.c.tcp.address: port must be a number from 1 to 65535 The port is not a number from 1 to 65535
check_groups.c.tcp.address: missing port in address The mapping has no address, or it is empty
check_groups.c.tcp.text_to_send: cannot be empty text_to_send is written with an empty text. Leave out text_to_send to send nothing
check_groups.c.tcp.reply_contains: cannot be empty An empty reply_contains or reply_matches_regex is rejected
check_groups.c.tcp.reply_matches_regex: is no valid regular expression: error parsing regexp: ... The regular expression is not valid RE2. The text after the colon says why
check_groups.c.tcp.expect: unknown field "expect": the inputs of the module tcp are address, text_to_send, reply_contains, reply_matches_regex The mapping has an input that tcp does not have. Remove expect, or write an input that the message names
check_groups.c.tcp: expected a mapping, or an address as host:port, found a list The check is written as a list. Write the address alone, such as tcp: 127.0.0.1:5432, or a mapping of the inputs

Tested on

  • Go tests in internal/checksandalerts, each against a stand-in server on the test machine: TestTCPCheckSendsATextAndExpectsAReply (a reply in two parts, both rules, the greeting alone, a server that closes, one that says nothing until the time limit, a closed port, and validation), TestChecksOfServersThatDoNotAnswerAsExpected (a server that answers a part and waits, with what it sent in the details, and a port where nothing listens), and TestTCPExchangePrimitive in internal/primitives. They ran on macOS ARM64 and, as the native suite, on the AMD64 guest of every target distribution.
  • The acceptance suite (tests/integration/acceptance.py), on every target distribution: a tcp check with the address alone, healthy against a test service, also with MMX absent at runtime.
  • The services suite (tests/integration/service_scenarios.py), on every target distribution: a small TCP server of the test, with a check that sends a text and keeps both rules, one that reads the greeting, one that expects a text that never comes, with what the server sent in its details, and a refused connection once the server is stopped.
  • Not verified on a real node: a server that answers a part and then waits. Go tests cover a server that answers a part and then waits (validation record).

Why it works this way

  • A reply rule needs at least one received byte. A regular expression that matches empty text cannot make a silent server healthy.

  • No default port. A reader sees on each check where it connects (protocols and common server software).

  • The check reads until the rules hold, not until the server closes. A server of a line protocol often keeps the connection open, and the check must not wait for it until the time limit.

  • Without a reply rule, MMWatchdog reads nothing. A port that accepts connections is all that such a check asks, and a server that sends nothing first is still healthy.

  • What the server sent is in the alert. A reader sees at once whether the server said something else or nothing.

  • MMWatchdog reads at most 64 KiB of a reply, whatever the server sends.

This page is docs/reference/modules/tcp.md in the MMWatchdog repository.