Change SSH or network settings safely
Who acts: you, or your coding agent, in your MMX project on your
controller machine: the computer where you run mmx.
When to use this page: you change a server's network settings, such as its address, or sshd's settings, such as the SSH port or password login, and you want to keep your access to the server, and MMX's.
What this page does: you give web01 a fixed address in place of
DHCP, and see the server put the old network file back by itself after a
typo cuts the server off. Then you move SSH on web01 and web02 from port 22 to
port 2222 without a moment when MMX can't log in. The page also says what
MMX does not protect, and how to get back in when you are locked out.
Terms:
- A node is a server that MMX manages. A playbook lists fleet stages. Each fleet stage picks nodes, and each of those nodes runs the fleet stage's node tasks, in order.
- MMX connects to a node in two ways: SSH, through the server's sshd, and HTTPS on TCP port 7443, to MMX's own service on the node. Connect to your servers explains both.
- A reachability confirmation is a request that your running
mmxcommand sends to the node over a new SSH or HTTPS connection, to prove that the node can still be reached. - An owed restore is a record on the node that a network file must go back to its previous version unless reachability confirmations arrive in time. A systemd timer on the node carries out the restore, without your computer and without a connection.
Choose your route:
| You change | What MMX does | Route |
|---|---|---|
| A network file of systemd-networkd, Netplan, NetworkManager, ifupdown, or WireGuard, with MMX's network modules | MMX checks new connections before and after the change. Without the confirmations, the node puts the old file back by itself. | Change a network file |
| sshd's settings, such as the port, password login, or the accounts that may log in | No automatic restore. You keep a way in: add the new setting next to the old setting, check new connections, then remove the old setting. | Change sshd's settings |
| Firewall rules | ufw and firewalld refuse some rules that would close sshd's port. A firewall rule is not put back. |
What MMX does not protect |
| You are already locked out, or MMX is | Wait for the restore, or use the server's console. | Get back in |
Before you start
mmx runsucceeds on the servers you change. The examples useweb01at203.0.113.11, which runs Debian 13, andweb02at203.0.113.12, which runs Ubuntu 24.04. Both are in the groupweb_serversininventories/example.yml, and MMX logs in to both asroot.- You can open each server's console without SSH, such as the web console or rescue system of your hosting provider. When a wrong sshd setting or firewall rule shuts out both you and MMX, the console is the way back in. Find the console before you start, not after a lockout.
- Find the inputs of any module with
mmx describe-module NAME, such asmmx describe-module systemd_networkd_file.
Change a network file
MMX's network modules change one network file and protect the change with
an owed restore. A plain file node task that writes a network file gets
no such protection, so use the module for the server's network tool:
| The server's network tool | Module | Files |
|---|---|---|
| systemd-networkd | systemd_networkd_file |
/etc/systemd/network/<name>.network or .netdev |
| Netplan, with the systemd-networkd renderer | netplan_file |
/etc/netplan/<name>.yaml |
| NetworkManager | networkmanager_connection_file |
/etc/NetworkManager/system-connections/<name>.nmconnection |
| ifupdown, Debian's package | ifupdown_interfaces_file |
/etc/network/interfaces or a file in /etc/network/interfaces.d/ |
| WireGuard | wireguard_interface |
The interface's configuration file |
MMX does not install or switch a server's network tool. Two servers with the same distribution can use different tools: a hosting provider's image may use another tool than the distribution's installer.
-
Find the file that configures the network card. On
web01, which uses systemd-networkd, asknetworkctlfor the main card,eth0:ssh root@203.0.113.11 networkctl status eth0● 4459: eth0 Link File: /usr/lib/systemd/network/99-default.link Network File: /etc/systemd/network/10-cloud-init-eth0.network State: routable (configured)Check: the output names a
Network File:in/etc/systemd/network/. Ifnetworkctlis missing, or the card showsunmanaged, the server uses another tool.ls /etc/netplan /etc/NetworkManager/system-connections /etc/network/interfacesshows which tool's files exist. -
Write the whole file in a playbook.
web01gets its address from DHCP today. Createplaybooks/network_settings.yml, which givesweb01the same address as a fixed address:# playbooks/network_settings.yml fleet_stages: - id: network_settings nodes: web01 node_tasks: - id: main_network_card systemd_networkd_file: name: 10-cloud-init-eth0.network # the file from step 1 content: | [Match] Name=eth0 [Network] Address=203.0.113.11/24 Gateway=203.0.113.1 DNS=203.0.113.1 restore_unless_controller_confirms_within: 2mThe module takes the file's whole text in the tool's own format.
restore_unless_controller_confirms_within: 2mis how long the node waits for the confirmations. Without the confirmations, the node puts the old file back when the two minutes end.2mis the default. Choose a longer time when the network takes longer to come up, such as a bond that negotiates with a switch.Check:
mmx validate network_settingsprintsValid: mmx_example. -
Preview the change.
mmx dryrun network_settings --diff echo "exit code $?"network_settings / web01 / main_network_card: succeeded, would_change — Would replace /etc/systemd/network/10-cloud-init-eth0.network and activate it. Whether systemd-networkd accepts the proposed network configuration change and whether the node remains reachable after activation were not checked. network_settings / web01 / main_network_card: would update "/etc/systemd/network/10-cloud-init-eth0.network" 40 bytes, permissions_mode 0644, uid 998, gid 998 -> 89 bytes, permissions_mode 0644, uid 0, gid 0 --- before +++ after @@ -2,5 +2,6 @@ Name=eth0 [Network] -DHCP=ipv4 - +Address=203.0.113.11/24 +Gateway=203.0.113.1 +DNS=203.0.113.1 exit code 2Check: exit code 2, and the difference shows your change. A dryrun changes nothing on the server, so a dryrun can't tell whether the server stays reachable. Only the run can.
-
Run the change.
mmx run network_settingsnetwork_settings / web01: succeeded network_settings / web01 / main_network_card: succeeded, changed network_settings / web01 / main_network_card: updated "/etc/systemd/network/10-cloud-init-eth0.network" 40 bytes, permissions_mode 0644, uid 998, gid 998 -> 89 bytes, permissions_mode 0644, uid 0, gid 0 Run ad684c47: succeededThe run took about three seconds. New SSH and HTTPS connections reached
web01before and after the change, so the node kept the new file and removed the restore timer.Check: the run ends with
Run ...: succeeded, and the server has the fixed address:ssh root@203.0.113.11 ip -4 -brief address show eth0eth0@if4460 UP 203.0.113.11/24A second
mmx run network_settingsfinds the same file and waits for nothing:network_settings / web01 / main_network_card: succeeded, unchanged — Network configuration file /etc/systemd/network/10-cloud-init-eth0.network is unchanged; no activation or confirmation wait was needed. -
See the server put the old file back. This step makes a mistake on purpose, so you know what a cut-off looks like. Skip it on a server that serves users: the server is unreachable for two minutes. In
playbooks/network_settings.yml, change the address to203.0.113.17, a typo for203.0.113.11, and run:mmx run network_settings echo "exit code $?"network_settings / web01: failed main_network_card: No reachability confirmation arrived over a new HTTPS connection or a new SSH connection within 2m. The previous file /etc/systemd/network/10-cloud-init-eth0.network and its running configuration are restored. network_settings / web01 / main_network_card: failed, changed — No reachability confirmation arrived over a new HTTPS connection or a new SSH connection within 2m. The previous file /etc/systemd/network/10-cloud-init-eth0.network and its running configuration are restored. network_settings / web01 / main_network_card: requested update "/etc/systemd/network/10-cloud-init-eth0.network" 89 bytes, permissions_mode 0644, uid 0, gid 0 -> 89 bytes, permissions_mode 0644, uid 0, gid 0 network_settings / web01 / file change 46975be97b78541d: /etc/systemd/network/10-cloud-init-eth0.network was restored to its previous state why the change was not kept: No reachability confirmation arrived over a new HTTPS connection or a new SSH connection within 2m. Run a3c6997b: failed exit code 1What happened on
web01, seen from its console:Time web010 s The new file is in place, and eth0has the address203.0.113.17. New connections to203.0.113.11fail.0 s to 2 min The timer mmx-owed-restore-46975be97b78541d.timerwaits.2 min The timer starts mmx-owed-restore-46975be97b78541d.service. The service puts the previous file back and reloads systemd-networkd.eth0has203.0.113.11again, and the timer is gone.The run ended about two minutes after it started, and fails with
changed: the network was changed and then put back.Check:
ssh root@203.0.113.11 trueworks, and the file on the server is the one from step 4. Change the address back to203.0.113.11, and run again: the line saysunchanged.
What MMX checks, in order
For each network file that changes, MMX does these steps on the node before the next node task starts:
- Your
mmxcommand opens new SSH and HTTPS connections to the node. Each kind of connection that works now must work after the change too. When neither kind works, the node task fails, and the file is not changed. - The node saves the old file and starts the restore timer. The timer
runs out after
restore_unless_controller_confirms_within. - The node writes the new file and activates the file with the tool's
own command, such as
networkctl reload. When the tool reports an error, the restore starts at once. - Your
mmxcommand opens new SSH and HTTPS connections again. An open connection does not count, because a network change can keep open connections alive while new connections fail. - When every kind of connection from the first step confirms, the node keeps the new file and removes the timer. Otherwise the timer puts the old file back, the node activates the old file, and the node task fails.
The timer runs on the node, so the node restores an unconfirmed change
also when your computer sleeps, mmx ends, or you press Ctrl-C. The
node does not keep an unconfirmed change over a restart of the server: at
boot, the restore service puts the old file back at once. Run mmx status RUN-ID later
to collect the result: the lines are the same as in step 5. A node with
connection: ssh_only in the inventory needs only SSH confirmations.
A network change is protected one file at a time. When a playbook changes two files and the second file cuts the server off, the node restores only the second file. MMX does not follow a server to a new address: confirmations go to the address in the inventory, so a change that removes that address from the server is restored. To move a server to a new address, add the new address next to the old address, change the inventory, then remove the old address in a later run. When the server you change is also your controller machine, MMX also checks that your controller machine can still reach the other nodes; the execution design describes the check.
Change sshd's settings
A wrong sshd setting locks MMX out at the next new connection, and MMX does not put an sshd file back by itself. MMX's SSH connection logs in through sshd. MMX's HTTPS service on the node checks each new HTTPS connection against sshd's current settings for accounts and keys, as Login rules that only OpenSSH can check describes. So keep a way in at every step: open the new port next to the old port, check new connections, and close the old port only then.
-
Write sshd's settings in their own file, with both ports. Create
playbooks/ssh_settings.yml:# playbooks/ssh_settings.yml fleet_stages: - id: ssh_settings nodes: web_servers node_tasks: - task_block: id: sshd_settings changes: - id: write_sshd_settings file: path: /etc/ssh/sshd_config.d/10-shop.conf permissions_mode: "0644" content: | # Managed by MMX in playbooks/ssh_settings.yml Port 22 Port 2222 PasswordAuthentication no check_new_file: # check the new file before it replaces the live one shell: script: sshd -t -f "$NEW_FILE" dryrun: sshd -t -f "$NEW_FILE" env: NEW_FILE: "{{ .new_file.path }}" validate: - id: check_whole_sshd_configuration shell: script: sshd -t dryrun: sshd -t changed_when: false activate: # Ubuntu 24.04 and 26.04: ssh.socket holds the listening ports. - id: regenerate_ssh_socket_ports when: {equals: ["{{ .node.distribution.id }}", ubuntu]} shell: script: systemctl daemon-reload only_when_tracked_inputs_changed: true - id: restart_ssh_socket when: {equals: ["{{ .node.distribution.id }}", ubuntu]} systemd_service: name: ssh.socket state: running - id: reload_sshd systemd_service: name: ssh state: running refresh: reload verify: - id: check_new_port wait_for: tcp_port_accepts_connections: 2222 wait_at_most: 10sWhat each part does:
/etc/ssh/sshd_config.d/10-shop.confis a file of its own, which Debian's and Ubuntu's/etc/ssh/sshd_configreads through itsIncludeline. sshd uses the first value that it reads for most settings, and the name10-shop.confsorts before files such as50-cloud-init.confthat some cloud images add.Portis different: eachPortline adds a port.check_new_filerunssshd -t -fon the new file alone, before the new file replaces the live file. A file that sshd rejects never replaces/etc/ssh/sshd_config.d/10-shop.conf, so sshd's next start, such as after a reboot, can't fail because of the rejected file. Change a configuration and reload its service explainscheck_new_file.validaterunssshd -ton the whole configuration, with the new file in place. A file can passcheck_new_filealone and still break the whole configuration, for example a file that ends inside aMatchblock. A failedvalidatestops the node task block beforeactivate, so sshd is not reloaded. The new file stays on disk, though, so fix the file and run again before you restart sshd or the server.reload_sshdreloads sshd only when the file differs from the file that sshd had when MMX last started or reloaded sshd. Debian's and Ubuntu'ssshservice checks the whole configuration withsshd -tbefore the reload. A rejected configuration fails the reload, and the running sshd keeps its old settings.- On Ubuntu 24.04 and 26.04,
ssh.socketopens the ports, not sshd. A systemd generator makes the socket's ports from sshd'sPortlines, and the generator runs only atsystemctl daemon-reload.regenerate_ssh_socket_portsruns the reload, andrestart_ssh_socketthen restartsssh.socketwith the new ports. A restart ofsshalone does not open a new port. Run a command after a change explainsonly_when_tracked_inputs_changed. check_new_portchecks on the node that sshd accepts connections on port 2222.
Check:
mmx validate ssh_settingsprintsValid: mmx_example. -
Preview the change.
mmx dryrun ssh_settings echo "exit code $?"ssh_settings / web02 / sshd_settings/changes/write_sshd_settings: succeeded, would_change check_new_file: succeeded ssh_settings / web02 / sshd_settings/changes/write_sshd_settings: would create "/etc/ssh/sshd_config.d/10-shop.conf" absent -> 91 bytes, permissions_mode 0644, uid 0, gid 0 ssh_settings / web02 / sshd_settings/validate/check_whole_sshd_configuration: succeeded, unchanged ssh_settings / web02 / sshd_settings/activate/regenerate_ssh_socket_ports: succeeded, would_change — would run: tracked input /etc/ssh/sshd_config.d/10-shop.conf would change ssh_settings / web02 / sshd_settings/activate/restart_ssh_socket: succeeded, would_change — tracked input /etc/ssh/sshd_config.d/10-shop.conf changed after the service last started or reloaded ssh_settings / web02 / sshd_settings/activate/reload_sshd: succeeded, would_change — tracked input /etc/ssh/sshd_config.d/10-shop.conf changed after the service last started or reloaded ssh_settings / web02 / sshd_settings/verify/check_new_port: not_checked, unchanged — dial tcp 127.0.0.1:2222: connect: connection refused ssh_settings / web02 / sshd_settings: succeeded, would_change Run bdced5d8: succeeded exit code 3The lines for
web01are the same, except that the two Ubuntu node tasks sayskipped, unchanged — when is false.Check:
check_new_file: succeededon each server. Exit code 3 is expected: nothing listens on port 2222 before the run, so the dryrun can't checkcheck_new_port. A rejected file fails here already, as If something is different shows. -
Run the change. The playbook turns password login off. Before you run it, check which key MMX logs in with:
- If MMX logs in with your own key, go on.
- If the server took a password at the first contact, MMX made its own
key in
~/.ssh/mmx/(a server that takes only a password). MMX keeps that key working on the new port: at the first run after step 4, MMX logs in with it over SSH on port 2222, checks that port 2222 reaches the same machine, and copies the key to the new port (Other SSH settings). Port 2222 must therefore be open and its host key trusted before step 4, which this step's Check proves.
mmx run ssh_settingsssh_settings / web02 / sshd_settings/changes/write_sshd_settings: succeeded, changed ssh_settings / web02 / sshd_settings/changes/write_sshd_settings: created "/etc/ssh/sshd_config.d/10-shop.conf" absent -> 91 bytes, permissions_mode 0644, uid 0, gid 0 ssh_settings / web02 / sshd_settings/validate/check_whole_sshd_configuration: succeeded, unchanged ssh_settings / web02 / sshd_settings/activate/regenerate_ssh_socket_ports: succeeded, changed — ran because tracked input /etc/ssh/sshd_config.d/10-shop.conf changed ssh_settings / web02 / sshd_settings/activate/restart_ssh_socket: succeeded, changed — tracked input /etc/ssh/sshd_config.d/10-shop.conf changed after the service last started or reloaded ssh_settings / web02 / sshd_settings/activate/reload_sshd: succeeded, changed — tracked input /etc/ssh/sshd_config.d/10-shop.conf changed after the service last started or reloaded ssh_settings / web02 / sshd_settings/verify/check_new_port: succeeded, unchanged — TCP port 127.0.0.1:2222 accepts connections ssh_settings / web02 / sshd_settings: succeeded, changed Run eea95472: succeededCheck: the run ends with
Run ...: succeeded, and you can log in on the new port, from your computer:ssh -p 2222 root@203.0.113.11 true ssh -p 2222 root@203.0.113.12 trueBoth commands end without an error. Port 22 still works too.
OpenSSH files a host key under the address and the port, but a key that your computer already trusts for port 22 also counts on port 2222 when the server shows the same key: OpenSSH 10.0 asked no question for
ssh -p 2222here, and~/.ssh/known_hostsgot no new line. Both commands above need no answer then, and neither does step 4. If a command does ask about the host key, because your computer has no key for this server yet, compare the fingerprint and answeryes, as step 1 of Connect to your servers shows, or read the key withssh-keyscan -p 2222 203.0.113.11. MMX never accepts a host key, so step 4 fails withHost key verification faileduntil your computer trusts the server's key. -
Tell MMX the new port, and check SSH on the new port. In
inventories/example.yml, addssh_port: 2222to each server:# inventories/example.yml nodes: web01: address: 203.0.113.11 ssh_user: root ssh_port: 2222 web02: address: 203.0.113.12 ssh_user: root ssh_port: 2222 groups: web_servers: nodes: [web01, web02]Then run over SSH only:
mmx run ssh_settings --connection ssh_onlyReason: an ordinary run uses HTTPS, which does not go through sshd's port, so only
--connection ssh_onlyproves the SSH port (Other SSH settings). A node whose key MMX made itself needs the key from step 3 here.Check: the run ends with
Run ...: succeeded, and every node task saysunchanged.mmx results --jsonshows"connection_used": "ssh"for each server. The SSH port that MMX used is the node'sssh_portin the inventory, or 22 without one:mmx nodesdoes not print the port. -
Close the old port. In
playbooks/ssh_settings.yml, delete the linePort 22, then preview and run:mmx dryrun ssh_settings --diff mmx run ssh_settingsThe dryrun's difference removes
-Port 22. The first dryrun or run over HTTPS after you changedssh_portprints a line such as203.0.113.12: setting up MMX through verified SSHfor each server. Other SSH settings explains why.Check: the run ends with
Run ...: succeeded, port 22 is closed, andmmx run ssh_settings --connection ssh_onlysucceeds:ssh root@203.0.113.11 truessh: connect to host 203.0.113.11 port 22: Connection refused
After step 5, playbooks/ssh_settings.yml reads:
# playbooks/ssh_settings.yml
fleet_stages:
- id: ssh_settings
nodes: web_servers
node_tasks:
- task_block:
id: sshd_settings
changes:
- id: write_sshd_settings
file:
path: /etc/ssh/sshd_config.d/10-shop.conf
permissions_mode: "0644"
content: |
# Managed by MMX in playbooks/ssh_settings.yml
Port 2222
PasswordAuthentication no
check_new_file:
shell:
script: sshd -t -f "$NEW_FILE"
dryrun: sshd -t -f "$NEW_FILE"
env:
NEW_FILE: "{{ .new_file.path }}"
validate:
- id: check_whole_sshd_configuration
shell:
script: sshd -t
dryrun: sshd -t
changed_when: false
activate:
- id: regenerate_ssh_socket_ports
when: {equals: ["{{ .node.distribution.id }}", ubuntu]}
shell:
script: systemctl daemon-reload
only_when_tracked_inputs_changed: true
- id: restart_ssh_socket
when: {equals: ["{{ .node.distribution.id }}", ubuntu]}
systemd_service:
name: ssh.socket
state: running
- id: reload_sshd
systemd_service:
name: ssh
state: running
refresh: reload
verify:
- id: check_new_port
wait_for:
tcp_port_accepts_connections: 2222
wait_at_most: 10s
Settings that decide who may log in
AllowUsers, AllowGroups, DenyUsers, DenyGroups, and
PermitRootLogin decide which accounts sshd lets in. Each of these
settings must still let in the account in the inventory's ssh_user, or
MMX is locked out. The run that changes the setting still succeeds,
because the run's connections are already open. For example, with
AllowUsers deploy added while MMX logs in as root, the run that
changes the setting prints a warning on its housekeeping line:
ssh_settings / web01: succeeded
housekeeping: recording the last successful run on the node: Get "https://203.0.113.11:7443/v1/info": HTTPS 403: SSH account policy denies root
The next run fails:
ssh_settings / web01: failed
Get "https://203.0.113.11:7443/v1/info": HTTPS 403: SSH account policy denies root
To change such a setting safely:
- Put the inventory's account in the allowed list or group first, in an
earlier node task or an earlier run. For example, add
roottoAllowUsers, or add the account to the group inAllowGroupswith agroupnode task. - Run the change.
- Run
mmx run ssh_settings, thenmmx run ssh_settings --connection ssh_only. Both runs use new connections, so both runs prove that MMX can still log in over HTTPS and over SSH.
PasswordAuthentication no keeps MMX working, because MMX logs in with a
key: your own key, or the key that MMX made at the first contact with a
server that took a password. MMX carries its own key to a new SSH port after
SSH shows the same machine, but not to a new address
(Other SSH settings). Before you turn password
login off, check that you can log in with a key yourself.
On Fedora, Rocky Linux, and other Red Hat family servers
The sshd service is named sshd, not ssh, so use name: sshd in
reload_sshd. SELinux lets sshd listen only on ports of the type
ssh_port_t. Add a selinux_port node task
for port 2222 with selinux_type: ssh_port_t before the node task block.
The selinux_port module needs the package policycoreutils-python-utils,
which dnf_package installs.
A Fedora Server or Rocky Linux server usually runs firewalld with only the
ssh service open, so port 2222 stays closed and the ssh -p 2222
commands in step 3 time out. Open the port with a
firewalld node task before the node task
block. The zone is the zone of the server's network card;
firewall-cmd --get-active-zones on the server shows the zone:
# Before the task_block. Written from the firewalld module page; not run
# on a real Red Hat family server.
- id: allow_new_ssh_port
firewalld:
zone: public
rules:
- {action: allow, port: 2222, protocol: tcp}
rules_not_listed: keep
rules_not_listed: keep leaves the other rules of the zone as they are.
Keep the rule that allows the ssh service until step 5 has closed port 22.
What MMX does not protect
- sshd's files. No module puts an sshd file back. The order of steps in Change sshd's settings is the protection.
- Network settings that a module other than the network modules
writes, such as a
file,line, orshellnode task that edits/etc/network/interfacesor runsip. Use the module from the table in Change a network file. - Firewall rules.
ufwrefuses a change after which no rule lets sshd's port in from the address that MMX's connection came from.firewalldrefuses a change after which the zone no longer allows sshd's port, or rejects the port for every source.firewallddoes not check which sources the allow rule names. A rule that passes the check is not put back.firewalld_zone_membership, which moves a network card into another firewalld zone, is protected like a network file. Firewall rules from afileorshellnode task, such as nftables rules, are not checked at all. A firewall that closes port 7443 does not lock MMX out: MMX uses SSH instead, as A server whose port 7443 is closed says. - Several files together. Each network file is confirmed and restored by itself, as What MMX checks, in order says.
Get back in
| What happened | What to do |
|---|---|
| A network change cut the server off, and the run still waits | Wait. The node puts the old file back when restore_unless_controller_confirms_within ends, two minutes by default. The run then ends with was restored to its previous state. |
mmx ended during a network change, such as after Ctrl-C |
Wait for restore_unless_controller_confirms_within, two minutes by default. Then run mmx status RUN-ID, with the run ID that the run printed at its start, to collect the restore's result. |
The run ends with restore completion is unconfirmed, and the server stays unreachable |
The restore on the server failed. See A restore that did not finish. |
| An sshd setting shuts MMX and you out | Open the server's console. Fix the file, such as /etc/ssh/sshd_config.d/10-shop.conf, then run sshd -t && systemctl reload ssh there. On Ubuntu, a port fix also needs systemctl daemon-reload && systemctl restart ssh.socket. On the Red Hat family, the service is sshd. Fix the playbook the same way, and run the playbook. The next run reloads sshd again, because the file on disk differs from the file that MMX last reloaded sshd with: MMX does not see a reload that you run by hand. |
| A firewall rule shuts SSH out | Open the server's console, and remove the rule with ufw or firewall-cmd. Fix the playbook, and run the playbook. |
ssh: connect to host ... port 22: Connection refused on a run with --connection ssh_only, while ordinary runs succeed |
sshd moved to another port, and the inventory still names the old port. Set ssh_port in the inventory, as in step 4. |
A restore that did not finish
A restore can fail, such as when the network tool on the server is
broken. The server then stays cut off, so MMX can't learn what happened.
The node keeps the owed restore and the saved old file, and refuses a later
change of the same file until the restore succeeds. In this example,
networkctl was missing on web01 when the restore ran. After about ten
minutes of trying to reach web01, the run ended with:
network_settings / web01: unknown
cannot collect node job 6ab0f40b9fa9e681b66b10a7b467fa3feeb71a78f1a10755bad9fa25c052b2af after file change 2323d0d805429de0; restore completion is unconfirmed; run mmx status 20261115T073956.152627178-79c3a64532ba51f4 to inspect it again: context deadline exceeded
network_settings / web01 / file change 2323d0d805429de0: the restore of /etc/systemd/network/10-cloud-init-eth0.network has not started yet
restore service on the node: mmx-owed-restore-2323d0d805429de0.service
Run 79c3a645: running
The line has not started yet is what MMX knew when MMX lost contact.
The line names the restore service, which you need on the server.
-
Open the server's console, because the server is not reachable over the network.
-
Read why the restore failed. On the console:
journalctl -u mmx-owed-restore-2323d0d805429de0.servicemmx-runner[4610]: owed restore 2323d0d805429de0 failed: Reloading systemd-networkd: networkctl is not installed; install the package that provides it in an earlier node task: systemd (apt, pacman), systemd-networkd (dnf, zypper); correct the cause and run systemctl start mmx-owed-restore-2323d0d805429de0.serviceCheck:
systemctl status mmx-owed-restore-2323d0d805429de0.servicesaysfailed. -
Fix the cause that the message names, such as a missing program.
-
Start the restore again, with the command from the message:
systemctl start mmx-owed-restore-2323d0d805429de0.serviceCheck:
systemctl status mmx-owed-restore-2323d0d805429de0.servicesaysUnit mmx-owed-restore-2323d0d805429de0.service could not be found.The node removes the restore's units after a restore that succeeded.ssh root@203.0.113.11 trueworks again from your computer. -
Collect the result on your computer:
mmx status 79c3a645network_settings / web01: failed main_network_card: No reachability confirmation arrived over a new HTTPS connection or a new SSH connection within 2m. Restoring /etc/systemd/network/10-cloud-init-eth0.network failed: Reloading systemd-networkd: networkctl is not installed; install the package that provides it in an earlier node task: systemd (apt, pacman), systemd-networkd (dnf, zypper). The owed restore and saved previous file remain; the previous running configuration is not confirmed restored. ... network_settings / web01 / file change 2323d0d805429de0: /etc/systemd/network/10-cloud-init-eth0.network was restored to its previous state why the change was not kept: No reachability confirmation arrived over a new HTTPS connection or a new SSH connection within 2m. Run 79c3a645: failedCheck: the last
file changeline sayswas restored to its previous state. The node task's reason keeps the first restore's failure, and the node task staysfailed, because the change was not kept. Fix the playbook, and run the playbook again.
If something is different
| What you see | What it means and what to do |
|---|---|
No reachability confirmation arrived over a new HTTPS connection or a new SSH connection within 2m. The previous file ... and its running configuration are restored. |
The new network file cut the server off, and the node put the old file back. Fix the file in the playbook, and run again. |
No new SSH or HTTPS connection confirmed node reachability before the file change; ... was not changed. |
MMX could not open a new connection to the server before the change, so the node changed nothing. Check that mmx run reaches the server, as Connect to your servers says. |
Cannot change ...: node job ... still owes a restore of this file. Restore service: mmx-owed-restore-....service. |
An earlier change of the same file has a restore that is not finished. Wait for the restore, or repair a failed restore as A restore that did not finish says. |
the restore of ... has not started yet, on a change that failed or that MMX could not confirm |
The restore is waiting on the node. The next line, restore service on the node, names the systemd unit that runs it. See A restore that did not finish. |
check_new_file failed: exit status 255, with sshd's message, such as line 3: Bad configuration option: PasswordAuthentcation |
sshd rejected the new file, and the live file stays as it was. The message names the line. Fix the file in the playbook, and run again. A dryrun shows the same failure. |
housekeeping: recording the last successful run on the node: ... HTTPS 403: SSH account policy denies root |
An sshd setting no longer lets in the inventory's account. The next run fails. Fix the setting now, from the console if MMX can no longer log in, as Settings that decide who may log in says. |
HTTPS 403: SSH account policy denies root and the run fails |
An sshd setting no longer lets in the inventory's account. See Get back in. |
check_new_port: not_checked, unchanged — dial tcp 127.0.0.1:2222: connect: connection refused in a dryrun |
Nothing listens on the new port before the run. Run the change. |
ssh -p 2222 times out, or check_new_port fails, on Fedora or Rocky Linux |
firewalld keeps port 2222 closed. Add the firewalld node task from On Fedora, Rocky Linux, and other Red Hat family servers. |
Host key verification failed on the run of step 4 |
Your computer does not trust the server's host key. Trust the key as step 3 says. |
check_whole_sshd_configuration fails, while check_new_file passed |
The new file breaks the combined configuration. The new file stays on disk, so fix the file in the playbook and run again before sshd or the server restarts. |
check_new_port: failed in a run, on Ubuntu |
ssh.socket did not open the new port. Check that regenerate_ssh_socket_ports and restart_ssh_socket ran. |
Where to go next
- Connect to your servers: SSH settings in the inventory, host keys, and a server whose port 7443 is closed.
- Change a configuration and reload its service:
check_new_file,validate, and putting files back when a node task block fails. - The network module pages, such as
systemd_networkd_fileandnetplan_file: every input, the tool's own checks, and every error. - Roll out a change gradually: change one server first, before a risky change reaches the others.
This page is docs/how-to/change-ssh-or-network-settings-safely.md in the MMX repository.