A subscription web service fleet
This example configures and deploys the servers of a whole web service with
MMX: two load balancers, a data server, two application servers, and a
worker server. MMX installs the packages, writes the configuration of every
service, deploys the applications from Git, and switches maintenance pages
on and off. The controller machine, where you run mmx, can be a laptop;
the nodes are the six servers that MMX manages.
The service is fictional. A Ruby on Rails application, the main
application, has web pages at www.example.com and an API at
api.example.com. Two small brand sites, gardening.example.net and
cooking.example.net, use separate deployments of a shared Rails
application. Each brand site is deployed from its own repository. The data
lives in MySQL, two Redis instances, memcached, Solr, and ClickHouse.
The fleet
| Node | Address | What the node runs |
|---|---|---|
lb1, lb2 |
192.0.2.11, 192.0.2.12 | nginx as the load balancer, and keepalived, which holds the virtual address 192.0.2.10 on lb1 or lb2 |
db1 |
192.0.2.21 | MySQL, the two Redis instances, memcached, Solr, and ClickHouse |
app1, app2 |
192.0.2.31, 192.0.2.32 | nginx with Passenger, which runs the main application's web pages (port 8081) and API (port 8080), and the brand sites (ports 8092 and 8093) |
worker1 |
192.0.2.41 | The main application's background workers and its cron jobs |
The playbooks never name a node. Each fleet stage, one step of a playbook that runs on a set of nodes, selects inventory groups:
| Inventory group | What nodes in the group run | Production nodes |
|---|---|---|
nginx_load_balancers |
nginx as the load balancer, keepalived | lb1, lb2 |
mysql_servers |
MySQL and its daily backup | db1 |
redis_and_memcached_servers |
Redis and memcached | db1 |
solr_servers |
Solr and its nightly backup | db1 |
clickhouse_servers |
ClickHouse and its hourly copy from MySQL | db1 |
main_application_servers |
The main application's web pages and API | app1, app2 |
brand_site_servers |
The brand sites | app1, app2 |
main_application_worker_servers |
The main application's workers | worker1 |
main_application_cron_job_servers |
The main application's cron jobs; exactly one node, because each job must run on one server only | worker1 |
Each part of the work is a role: a list of node tasks with declared inputs,
in roles/<role>/main.yml, with its files in roles/<role>/files/. A
role's header comment says what the role does and why.
The two inventories
inventories/production.ymllists the six servers. Its addresses are documentation addresses, and its Git repositories are placeholders. A real project writes its own, and adds the credentials that its Git host requires to thegittask of the roleapplication_release: an access token asusernameandpasswordfor anhttps://repository, orssh_private_keyandssh_known_hosts_linesfor one over SSH.inventories/test.ymlputs the same groups on two nodes for the guest suite:test_load_balancer_and_dataholds the load balancer and every data service, andtest_applicationholds the applications, the workers, and the cron jobs.
The test folds the groups onto two nodes, not six, for two reasons. The load
balancer and the application servers both run nginx and both own
/etc/nginx/nginx.conf, so they cannot share a machine. And a machine runs
one node job at a time: two inventory nodes on one machine in the same fleet
stage would collide.
inventories/production.yml and inventories/test.yml set the same
variables. The test values are smaller where a test machine is small, such
as memory sizes and process counts, and differ where a test must not reach
an outside service: the certificates are self-signed instead of Let's
Encrypt's, and the Git repositories are directories on the application
server.
default_inventory in mmx_project_settings.yml is test. A command on
the real servers must say --inventory production, so a forgotten option
reaches test machines only.
The playbooks
site: the whole server configuration
mmx dryrun site --inventory production
mmx run site --inventory production
playbooks/site.yml has one fleet stage for each part
that the team applies on its own, in this order: base_server_setup,
nftables_firewall, deploy_account, data_server_swap, mysql_servers,
redis_and_memcached_servers, solr_servers, clickhouse_servers,
nginx_load_balancers, keepalived, ruby_and_nodejs, main_application,
nginx_with_passenger, main_application_nginx_servers,
brand_site_nginx_servers, main_application_workers, and
main_application_cron_jobs. A run applies all of them. To apply some
parts, name their fleet stages:
mmx run site --inventory production --only-fleet-stages nginx_load_balancers,keepalived
mmx run site --inventory production --only-fleet-stages nftables_firewall
The fleet stages nginx_load_balancers, main_application_nginx_servers,
and brand_site_nginx_servers work on one node at a time
(max_nodes_at_once: 1), so a mistake that nginx -t cannot see reaches
one server first.
A restart of MySQL, of the Redis instances, or of Solr interrupts the service. If a routine run would restart MySQL, Redis, or Solr, the run fails with a message and leaves the configuration of the service requiring the restart unchanged. Run again off-peak with the approval:
mmx run site --inventory production --only-fleet-stages mysql_servers --var mysql_restart_approved=true
mmx run site --inventory production --only-fleet-stages redis_and_memcached_servers --var redis_restart_approved=true
mmx run site --inventory production --only-fleet-stages solr_servers --var solr_restart_approved=true
deploy: a release of the applications
mmx run deploy --inventory production --var release=v2
mmx run deploy --inventory production --var release=v2 --only-fleet-stages brand_sites
release is a Git tag or branch of each repository.
playbooks/deploy.yml deploys it in four fleet
stages:
main_application: the web pages and the API, on half of the application servers at a time, so the other half answers while the first restarts. Each server waits until its API answersapi v2 pongand its web pages answer with the release.main_application_workers: the same release on the worker server; the targetmain-application-workers.targetrestarts the workers.brand_sites: each brand site from its own repository, half of the servers at a time.check_through_load_balancer: each public name answers with the new release through the load balancer, over HTTPS.
A failure on one server stops the run (on_node_failure: stop_run), so the
other servers keep the release they run. Going back is another run with the
older release.
load_balancer_maintenance_pages: maintenance pages on and off
mmx run load_balancer_maintenance_pages --inventory production --var maintenance_pages_shown=true
mmx run load_balancer_maintenance_pages --inventory production --var maintenance_pages_shown=false
While a page is shown, the load balancers answer every request to that site
with the maintenance page and the HTTP status 503. maintenance_pages_shown
has no default: MMX refuses a run that does not say which of the two it
wants. The playbook has two fleet stages, brand_site_maintenance_pages and
main_application_maintenance_pages, so --only-fleet-stages shows or hides
the pages of one part only. Each fleet stage checks the HTTP status of every
site it changed.
server_details: addresses and disks for the team's notes
mmx run server_details --inventory production
Each server's network addresses and disks are saved on the controller
machine as server_details/<node>.txt in the project.
test_stand_in_applications: test scaffolding
mmx run test_stand_in_applications
Run test_stand_in_applications once on the test machines, after site.
The playbook puts tiny Rack applications, from
files/test_stand_in_applications/,
into bare Git repositories on the application server, with the tags v1 and
v2, so that deploy clones, installs gems, and switches releases for
real. The playbook also creates the MySQL table page_views with three
rows, which the application's migrations would create in production, so that
ClickHouse's hourly copy has something to copy. In production the
repositories are on the Git host, and test_stand_in_applications never
runs.
Secrets
The roles read five secrets: the recovery account's password hash, the
MySQL passwords of the main application and of ClickHouse's copy job, the
ClickHouse account reporting's password, and the Cloudflare API token that
certbot's DNS check uses. Each secret key, the name of one secret, is bound
to an environment variable on the controller machine in
mmx_project_settings.yml:
# mmx_project_settings.yml
secrets_from_env:
recovery_account_password_hash: SUBSCRIPTION_WEB_SERVICE_RECOVERY_ACCOUNT_PASSWORD_HASH
main_application_database_password: SUBSCRIPTION_WEB_SERVICE_MAIN_APPLICATION_DATABASE_PASSWORD
clickhouse_copy_database_password: SUBSCRIPTION_WEB_SERVICE_CLICKHOUSE_COPY_DATABASE_PASSWORD
clickhouse_reporting_password: SUBSCRIPTION_WEB_SERVICE_CLICKHOUSE_REPORTING_PASSWORD
cloudflare_api_token: SUBSCRIPTION_WEB_SERVICE_CLOUDFLARE_API_TOKEN
The guest suite sets those five variables to test values. A production
project keeps its secrets in the encrypted vault instead (mmx secrets edit)
and removes the bindings; see
Keep secrets.
Parts that exist only for tests
The inventory inventories/test.yml, and every name that starts with
test_, exist only for the test:
- the test inventory's nodes
test_load_balancer_and_dataandtest_application; - the playbook
test_stand_in_applicationsand its files infiles/test_stand_in_applications/, which become the directories/home/deploy/test_stand_in_application_sources/and/home/deploy/test_stand_in_git_repositories/on the application server; - the role
test_self_signed_certificates, which the fleet stagenginx_load_balancerscalls in place oflets_encrypt_certificateswhencertificates_from_lets_encryptis false. It writes its certificates to/etc/ssl/test-self-signed/<name>/, in the same form as certbot's.
The administrators' SSH keys in
roles/base_server_setup/files/public_keys/, the deployment key in
roles/deploy_account/files/deploy_public_keys/, and the company's internal
certificate authority in roles/base_server_setup/files/ are test keys and
a test certificate. A real project replaces the administrators' SSH keys,
deployment key, and internal certificate authority certificate with the
project's own keys and certificate.
Why it works this way
Settings left at their defaults
Every setting in a configuration file carries a one-line reason in a comment. Everything else stays at the program's default, because a default that its program's authors keep current beats a value copied from an old guide.
- Transparent huge pages stay at Ubuntu's kernel default,
madvise: only a program that asks for huge pages gets huge pages. Since Redis 6.2, Redis turns off transparent huge pages for itself (disable-thp yes). MySQL, Redis, Solr, and ClickHouse work withmadvise, so no service turns off the kernel's transparent huge pages. - Kernel network settings stay at the defaults.
net.core.somaxconn, the queue of connections that wait for a program to accept them, is 4096 since Linux 5.4, more than the old tuning guides asked for. Two kernel settings remain:vm.swappiness1 on the data servers (fleet stagedata_server_swap), andnet.ipv4.ip_nonlocal_bind1 on the load balancers, so nginx may bind the virtual address while keepalived holds it on the other load balancer. - Open-file limits are set for each service that needs more than
systemd's default, in its own unit: MySQL, nginx with Passenger, each
Redis instance, and Solr. nginx on the load balancers raises its workers'
limit itself (
worker_rlimit_nofile). No limit applies to every program on a server. - MySQL keeps
innodb_flush_log_at_trx_commitat 1. The value 2 is faster, but loses up to a second of committed transactions when the server crashes. The character set is MySQL 8's default,utf8mb4. - Redis sets only what differs for this fleet: the addresses it listens on, the memory limit and eviction policy of each instance, and the append-only log of the persisted instance.
- memcached keeps its default connection limit and thread count.
- nginx keeps its default timeouts and buffers, has no proxy cache, and sets no CORS headers: an application sets those where it needs them. The nginx package's daily log rotation stays.
- Passenger keeps its default start timeout and idle times.
- ClickHouse keeps its default connection and query limits, and rotates its own logs by size and count.
- Solr keeps the garbage collector settings of its start script.
One nftables file instead of ufw
The firewall of each server is one nftables file, /etc/fleet-firewall.nft,
written from a template that reads the inventory groups: a node in
nginx_load_balancers admits HTTP, HTTPS, and keepalived's VRRP packets,
and every node admits SSH, MMX's HTTPS port 7443, mosh, and everything from
the company's own networks (trusted_networks). The data services, such as
MySQL and ClickHouse, are reachable only from those networks. Loading the file replaces the table
fleet_firewall in one step, so the server is never without rules, and the
tables of other programs stay as they are. The service fleet-firewall
loads the file at boot. The ufw page
recommends this form for a firewall that follows the inventory.
One directory for each release
The role application_release checks out each release into its own
directory, <application>/releases/<release>, installs its gems there,
links the shared configuration files, and only then switches the link
<application>/current to the new release directory. The running release is
never changed in place. Passenger, or systemd for the workers, restarts the
application once for each new release, also when a run stopped between the
switch and the restart. The release directories
example shows the
pattern on its own.
Restarts that need an approval
A restart of MySQL drops every connection of the applications, a restart of
Redis empties its cache instance, and a restart of Solr stops search until
it has loaded its core. A node task block's checks_before_restart_or_reload
refuses such a restart unless the command line approves it, so a routine
site run never interrupts the service by surprise. See
Checks before a restart or reload.
Coming from Ansible
This example started as a real company's Ansible repository. The company's Ansible configuration was converted to MMX and made anonymous. The example's names, domains, addresses, and services are invented, and the parts that only served the company's own products were left out. These Ansible habits were replaced:
- One playbook for each kind of server, and tags became one
siteplaybook with a fleet stage for each part, chosen with--only-fleet-stages. ignore_errors: trueon account creation, which hid user ID clashes, became theusermodule, which fails with the reason.pauseprompts before restarts became restart approvals on the command line, such as--var redis_restart_approved=true.- A
pausethat asked to deploy the workers first became a condition: the workers' target starts only once a release exists (path_exists). lineinfileedits ofsshd_configand/etc/sudoersbecame drop-in files that MMX owns whole. The edits oflimits.confandsystem.confbecame each service's own limit in its unit, and the edit of/etc/timezonebecame thetime_zonemodule.- Overwriting a unit file of a package, the upgrade timer, became a drop-in.
git reset --hardandgit cleanbefore each pull into one checkout became one directory for each release.statchecks and marker files that kept shell installers from running twice became packages and modules that check the server themselves.chown -Ron every run of the Solr role became Solr's archive, unpacked and owned by root, with Solr writing only to its data, log, and backup directories.- A cron job that restarted a service when it did not answer became
systemd's
Restart=. - The swap file by
fallocate,mkswap, andswaponbecame theswap_filemodule. - A firewall script that flushed every rule and then added them again, which left a moment without rules, became the nftables file, which replaces its table in one step.
apt_keyfrom a keyserver becameapt_repositorywith the key pinned by its SHA-256.- Per-host
whenblocks such aswhen: "'lb03' in inventory_hostname"became inventory variables, such askeepalived_priority. - Private keys and passwords in the inventory, and an encrypted secrets file in an application's repository, became MMX secrets.
local_action: copyfor the server details becamesave_on_controller_machine.
Move from Ansible or Salt maps the rest of Ansible's vocabulary to MMX.
Run the executable example
The guest suite
tests/integration/subscription_web_service_example.py
runs every playbook on two test VMs, mmx-dev (the node
test_load_balancer_and_data) and mmx-peer (the node test_application).
Each step runs a dryrun, a run, a second run that must change nothing, and a
dryrun that must find nothing to change: site,
test_stand_in_applications, deploy with release=v1 and then v2,
load_balancer_maintenance_pages with true and then false, and
server_details. Then a check script on each node, as root, checks the
services, the releases, and the maintenance pages.
The suite is part of the catalog of guest suites that the
testing guide describes. It runs on Ubuntu 24.04
only, because the roles use Ubuntu's packages, such as mysql-server and
libmysqlclient-dev, and Phusion's repository for Ubuntu's release. It needs
VMs, not containers: the example sets kernel settings, a swap file, and time
synchronisation, which belong to the guest's own kernel and clock.
MMX_TEST_ROOT=$HOME/.cache/mmx-incus python3 tests/integration/incus.py \
--distribution ubuntu2404 --suite subscription_web_service_example.py
A run takes about 15 minutes. Package installation takes most of the run's elapsed time. About 4 minutes go to the first dryrun's download of Solr's archive from Apache's archive server, which sends the archive slowly. Later commands reuse the saved bytes.
This page is examples/subscription-web-service-fleet/README.md in the MMX repository.