memcached
memcached asks a memcached server for its statistics, over TCP, in
memcached's own text protocol. The check is healthy when the server gives
the requested statistics. The check also measures the server's connections,
how many items the server holds, and how full the cache is. A get is a
request to retrieve a cached item.
For the interval since the previous run, the check measures gets that found an item and gets that found none. The check also measures the share of gets that found an item and counts items removed to make room. MMWatchdog speaks the protocol itself, so the node needs no separate memcached client program for the check.
The value of the module is the server's address, written as host:port, such
as 127.0.0.1:11211. The address has no default port, so a reader sees on
the check where it connects.
The check measures what the memcached server knows. Only the application knows what the cache saves the application, such as database work caused by a miss. A check of the application's own measures what the cache saves the application, for example through a module of the project's own.
Inputs
The module takes one value, which a check writes after the module's name, and no other input.
- the value of
memcached(string; required): The address of the memcached server, as host:port.
Measurements
current_connections(connections): The connections that are open to the server now.cached_items(items): The items that the server holds now.cache_used_percent(percent): The percentage of the server's configured cache memory used to store items.cache_hits_per_second(hits per second): The number of cache lookups per second since the previous run that found their item.cache_misses_per_second(misses per second): The number of cache lookups per second since the previous run that found no item, because the server never had the item, removed the item, or let the item expire.cache_hit_percent(percent): The percentage of cache lookups since the previous run that found an item, from 0 to 100. No reading is available without lookups.cache_evictions_per_second(evictions per second): The number of items per second removed before expiry to make room in the cache, measured since the previous run.
The module fails a check by itself, and it also reports these measurements. A check sets a limit on a measurement with fail_when_measurement_above or fail_when_measurement_below, and fails when a reading passes its limit. A check with the module always states an alert_severity. The configuration files reference explains limits in checks that measure a number.
When the check fails, and when it cannot check
A run sends stats to the server and reads its answer, which ends with a
line that says END.
-
The check fails when any of the following conditions occurs:
- the connection does not open;
- the server closes the connection before the answer is complete;
- the answer does not arrive within the check group's
time_limit_per_run; - the answer is not the statistics of a memcached server;
- a reading is above a
fail_when_measurement_abovelimit or below afail_when_measurement_belowlimit.
-
MMWatchdog cannot check when a limit names a measurement that this run could not make. The run is then unknown, with the reason:
- the server gave no
curr_connections,curr_items, orlimit_maxbytes; - the server gave no
get_hits,get_misses,evictions, oruptime, needed for hits, misses, hit ratio, and evictions; - no get reached the server since the run before, so there is no hit
ratio (
cache_hit_percent).
Without a limit on an unavailable measurement, the run is healthy and omits the unavailable measurement's reading.
- the server gave no
A failed run starts or continues the check's failing time. An unknown run opens a cannot-check incident of its own after the waiting times used for failure incidents, as the design's failing time describes.
Examples
The cache answers and has room
# /etc/mmwatchdog/check_groups/cache.yml
check_groups:
cache:
check_every: 1m
time_limit_per_run: 10s
alert_after_failing_for: 3m
recovered_after_healthy_for: 1m
checks:
cache_answers_and_has_room:
memcached: 127.0.0.1:11211
fail_when_measurement_above:
cache_used_percent: 95
alert_severity: warn
Every minute, MMWatchdog on the node opens a TCP connection to the memcached
server at 127.0.0.1:11211 and asks for statistics. The run is healthy when
the server answers and cached items occupy less than 95 percent of
memcached's configured capacity for storing items. The run's message is cache_used_percent is 12.5 (fails above 95); the memcached server at 127.0.0.1:11211 gave its statistics; hits, misses, and evictions over 60.000s. When the server does not answer, or the cache
is fuller than 95 percent, for 3 minutes, MMWatchdog sends a warn alert.
A memcached server whose cache is full is doing its job: it removes old
items to make room. So a limit on cache_used_percent says that the cache
is smaller than what the application would like to keep. The capacity concern is worth a
warn, not an sos. The measurements take a limit in the way that
every measurement does
(checks that measure a number).
Most gets find their item
# /etc/mmwatchdog/check_groups/cache.yml
check_groups:
cache:
check_every: 1m
time_limit_per_run: 10s
alert_after_failing_for: 10m
recovered_after_healthy_for: 5m
checks:
most_gets_find_their_item:
memcached: 127.0.0.1:11211
fail_when_measurement_below:
cache_hit_percent: 80 # of the gets in the minute since the run before
fail_when_measurement_above:
cache_evictions_per_second: 5 # items removed to make room, before they expired
alert_severity: warn
Every minute, MMWatchdog on the node reads the memcached server's counters
and compares the counters with the previous run's counters. The run fails
when fewer than 80 percent of gets since the previous run found an item.
The run also fails when the server removed more than 5 items a second to
make room. The run's message is then, for example,
cache_evictions_per_second is 0 (fails above 5); cache_hit_percent is 42.5 (fails below 80); the memcached server at 127.0.0.1:11211 gave its statistics; hits, misses, and evictions over 60.000s. After 10 minutes of such runs, MMWatchdog sends a warn alert.
Right after the server starts, its cache is cold: most gets find no item,
and cache_hit_percent is low. As the application fills the cache, the
share rises. The 10 minutes of alert_after_failing_for give a cache that
starts again time to warm up before anybody gets an alert. If the
application asks for nothing during the minute between runs, the check
cannot measure cache_hit_percent for the run ending the idle minute.
The limit on cache_hit_percent makes the run ending the idle minute
unknown, so set the hit-ratio limit on a cache the application uses all the
time.
How it works
| Measurement | Meaning |
|---|---|
current_connections |
The server's curr_connections: the connections that are open to it now |
cached_items |
The server's curr_items: the items that it holds now |
cache_used_percent |
Bytes occupied by cached items (bytes) as a percent of the configured capacity for storing items (limit_maxbytes), 0 to 100 |
cache_hits_per_second |
The increase in the server's get_hits: gets that found an item |
cache_misses_per_second |
The increase in the server's get_misses: gets that found no item. The server never had the item, removed the item, or let the item expire. |
cache_hit_percent |
The hits as a percent of all gets since the run before, 0 to 100 |
cache_evictions_per_second |
The increase in the server's evictions: items removed to make room for new items, before expiration |
-
The hits, the misses, the hit ratio, and the evictions cover the time since the check's run before. The server counts hits, misses, and evictions from the server's start. The check calculates rates and the hit ratio from counter increases between two runs. So the hit ratio of a cold cache differs from the hit ratio of a warm cache, whatever the server counted before.
Three cases need two requests within a run: the first run, a run after a gap of more than
2 × check_every + time_limit_per_run, and a run after the server started again. In each case, MMWatchdog asks the server a second time, 200 ms after the first request. The check calculates the rates and hit ratio from the counter increases between the two answers. So the check group'stime_limit_per_runmust be longer than200ms.The message ends with the interval covered by
cache_hits_per_second,cache_misses_per_second,cache_hit_percent, andcache_evictions_per_second, such asover 60.000s. The CPU sampling section of the configuration files reference gives the rules that every module that measures a rate shares. -
MMWatchdog sees that the server started again when one of its counters, or its
uptime, the seconds since its start, is lower than at the run before. -
A healthy run without a limit says
the memcached server at 127.0.0.1:11211 gave its statistics; hits, misses, and evictions over 60.000s. With a limit, the message names each reading that has a limit in front of the message about receiving statistics and the measurement interval. -
MMWatchdog reads at most 64 KiB of an answer. MMWatchdog shows the server's answer with every character replaced that changes how text is shown, as for every check of a common service (protocols and common server software).
Errors and what they mean
check_groups.cache.checks.cache_answers_and_has_room.memcached: address 127.0.0.1: missing port in address. The address has no port. Write thememcachedvalue as host:port, such as127.0.0.1:11211.... memcached: port must be a number from 1 to 65535. The port is not an integer from 1 to 65535.check_groups.cache.time_limit_per_run: must exceed 200ms for the initial sample of the memcached server's counters. The first run asks the server twice, 200 ms apart. Give the check group a longertime_limit_per_run.dial tcp 127.0.0.1:11211: connect: connection refused. No program listens at127.0.0.1:11211. Start memcached, or correct the address.the connection to 127.0.0.1:11211 did not open within the time limit of the run. The connection did not open before the time limit of the run ended it.the server at 127.0.0.1:11211 did not answer within the time limit of the run, or... ended the connection before its answer was complete. The server is overloaded or hangs. The alert's body shows what the server sent until the timeout or connection closure.the server at 127.0.0.1:11211 did not answer with the statistics of a memcached server. Another program listens at127.0.0.1:11211, such as a Redis server. The alert's body shows the listening program's response, afterthe server sent:.the memcached server gave no limit_maxbytes(orno curr_connections,no curr_items,no get_hits, and so on). The run is unknown because a limited measurement needs the missing counter named in the error. A server that only speaks memcached's protocol, such as a proxy, can leave numbers out.the memcached server answered no gets in the last 60.000s, so cache_hit_percent cannot be measured. The run is unknown because a limit names the hit ratio, and the application asked the server for nothing in the time that the message names.
Tested on
-
Go tests:
TestMemcachedCheckinprotocol_checks_test.goruns the check against a stand-in server on the test machine. The test covers:- statistics that provide every measurement;
- a limit that passes and a limit that fails;
- the first run, with two counter readings;
- a cold cache followed by a warm cache;
- an interval without gets;
- a server that started again;
- a restarted server whose counters already exceed the previous values;
- a server that gives no counters of gets;
- a server that answers
ERRORor another protocol.
TestTCPCheckSendsATextAndExpectsAReplycovers validation.TestChecksOfServersThatDoNotAnswerAsExpectedchecks a port where nothing listens. -
Real servers: the
servicessuite (service_scenarios.py) gets the statistics of memcached as each distribution installs it, with a limit oncache_used_percent, and finds the check failed while memcached is stopped and healthy after it starts again. The latest run passed on every target distribution (validation record). On Debian 13, the suite also passed checks ofcached_items,cache_hits_per_second,cache_misses_per_second, andcache_evictions_per_secondfrom the real server. The Debian 13 run verifiedcache_hit_percentof 0 when every get misses and 100 when every get hits. -
Not verified on a real node: a server that answers a part and then waits. A Go test covers a server that sends a partial answer and then waits.
Why it works this way
-
Without
bytes, there is nocache_used_percentreading. A missing counter does not mean zero use. A limit oncache_used_percentmakes the run unknown with the missing counter named. -
A full cache is not a failure by itself. memcached makes room by removing old items, so the check fails on the cache only with a limit chosen by the person writing the check configuration.
-
The hit ratio covers the time since the run before, not the time since the server started. A server that has run for weeks has a hit ratio since the server's start that hardly moves. Recent misses have too little effect on the lifetime ratio to clearly reveal a cold cache or an application change that causes misses all the time.
-
The hits, misses, and evictions are per second. A limit then means the maximum or minimum hits, misses, or evictions per second, whatever the check group's
check_everyis. -
A time without gets has no hit ratio, not one of 0 or 100. Reporting 0 percent or 100 percent would invent a hit ratio for an interval with no gets.
-
The address has no default port. A reader sees on each check where it connects.
-
The module is named for memcached, as every core module that checks a common service is named for the tool. The name helps a reader recognize the service being checked.
This page is docs/reference/modules/memcached.md in the MMWatchdog repository.