Write a package module for your distribution
Who acts: you, or your coding agent, in your MMX project on your
controller machine: the computer where you run mmx. The checks run on a
test server that runs your distribution.
When to use this page: your servers run a distribution whose package
manager has no MMX module, and you want node tasks that install and remove
packages there. MMX's package modules are apt_package for apt on Debian
and Ubuntu, dnf_package for dnf on Fedora, Rocky Linux, and AlmaLinux,
and the official extra modules pacman_package for pacman on Arch Linux
and zypper_package for zypper on openSUSE. If one of these modules fits your
servers, use that module instead: Find the module for a job.
What this page does: you copy pacman_package into your project as
mytool_package, for a package manager called mytool. You replace
pacman's commands with your package manager's, one part of the script at
a time. Then you run the checks that show the new module behaves like
MMX's own package modules.
Terms:
- A node is a server that MMX manages. A playbook lists fleet stages. Each fleet stage picks nodes, and each of those nodes runs the fleet stage's node tasks, in order. A node task names one module, the code that does the work.
- A module of the project's own is a directory in your project's
external_modules/. This page uses a script in Starlark, a small language that reads like Python. The script reads and changes the node only through primitives, functions that MMX gives the script, such asrun_command. - An official extra module is a Starlark module that ships with MMX,
such as
pacman_package. Its files are inofficial_extra_modules/beside themmxprogram. - The package index is the package manager's local copy of the list of packages that the package servers offer. pacman calls the package index the package databases.
- A transaction is one run of the package manager that installs, upgrades, or removes a set of packages together.
- A node task block groups a change with what follows it, in phases
such as
changesandactivate. Asystemd_servicenode task inactivaterestarts its service when something inchangeschanged.
Before you start
- You know how a module of the project's own works: the script, the module description, and the statuses a script returns. Write your own module teaches them with a small example, and the module interface reference lists every primitive.
- You have a test node that runs your distribution with systemd, and that
nothing else depends on. The checks install and remove packages on the test node,
and hold the package manager's lock. The examples call the test node
test01, at203.0.113.31. - You test each step on a server of the distribution that the module is
for, which has its package manager.
mytooldoes not exist, so this page cannot show what your server prints. The outputs on this page come from a Debiantest01withoutmytool, where they show the program missing (steps 4 and 5). Where a step's result needs the real package manager, the step describes what you see, and shows no output. The outputs of pacman, in step 11 and in the checks, are from an Arch Linux node. - You know your package manager's commands for each job in this table.
pacman's commands are the ones that the copied script uses:
Job in the script pacman's command Where in module.starList the installed packages, one line with name and version each pacman -Qread_installedShow what an installation would do, without doing it pacman -Sup --needed <names> --print-format "%n %v"preview,preview_installationShow what a removal would do, without doing it pacman -Rcp <names> --print-format "%n %v"removeRefresh the package index pacman -Syrefresh_the_package_databasesInstall packages pacman -Su --needed <names>installRemove packages pacman -R <names>removeUpgrade every package pacman -Su, after the refreshupgrade_allAnswer every question with the default --noconfirmevery command that changes packages The lock that the package manager holds while it works the file /var/lib/pacman/db.lckLOCK_FILE,wait_for_the_lock
Steps
-
Copy
pacman_packageinto the project. In the project directory:mkdir -p external_modules cp -r ~/.local/lib/mmx/official_extra_modules/pacman_package external_modules/mytool_package~/.local/lib/mmxis where Get started installsmmxand its official extra modules. The directory name,mytool_package, becomes the module's name. Reason: a module of your own cannot keep the namepacman_package, because an official extra module has that name.Check:
ls external_modules/mytool_packagelistsREADME.md,module.star, andmodule_description.yml. -
Give the copy its new name. Two places still say
pacman_package:MODULEin the script, which starts every failure reason, and the examples in the module description.sed -i 's/^MODULE = "pacman_package"/MODULE = "mytool_package"/' external_modules/mytool_package/module.star sed -i 's/pacman_package:/mytool_package:/' external_modules/mytool_package/module_description.ymlOn macOS, write
sed -i ''instead ofsed -i.Check:
mmx describe-module mytool_packagestarts with:mytool_package: the project's own module, Starlark, in external_modules/mytool_package/If the command prints
examples.0.config: the example does not name the module mytool_package, an example inmodule_description.ymlstill namespacman_package. -
Add the test node and a test playbook. Add
test01toinventories/example.yml:# inventories/example.yml, below nodes: test01: address: 203.0.113.31 ssh_user: rootCreate
playbooks/test_mytool_package.yml. The test playbook installs nginx and starts nginx, so the checks see both the package and a service that depends on the package:# playbooks/test_mytool_package.yml fleet_stages: - id: install_nginx nodes: test01 node_tasks: - task_block: id: nginx changes: - id: install_nginx_package mytool_package: packages: [nginx] activate: - id: start_nginx systemd_service: name: nginx state: running start_at_boot: trueIf your distribution names the package differently, write that name.
Check:
mmx validate test_mytool_packageprintsValid: mmx_example.mmx validatechecks the playbook and the module description.mmx validatedoes not run the script, so a mistake inmodule.starshows only in a dryrun or a run. -
Replace the rules for the inputs.
check_inputsreturnsinvalid_inputsfor each rule that the module description cannot state, before the script reads anything on the node. pacman's rules are the characters of a package name, a name listed twice, and inputs that exclude each other. Put your package manager's characters intoNAME_CHARACTERS, and its rules intoname_problem.packagesinpacman_packageis a list of names, because pacman's package servers offer one version of each package. If your package manager can install a chosen version of a package, each entry needs anameand aversion, as inapt_package,dnf_package, andzypper_package. A module description has no type for a list of mappings, so givepackagesthe typeany, and check each entry incheck_inputs.Check: put a name that breaks your rule into the test playbook, and run
mmx dryrun test_mytool_package. The node task fails in the dryrun with your rule, and the exit code is 1. The reason is the text of your rule afterfailed, unchanged —. For pacman, the nameextra/nginxgives this reason:mytool_package.packages: extra/nginx may hold only letters, digits, and @ . _ + -; a repository in front of the name, such as extra/nginx, is not acceptedPut
nginxback afterwards. -
Replace the program and its message. In
run_node_task, changefind_program("pacman")tofind_program("mytool"), and rewrite themissing_prerequisitereason below it. On a node without the program, a run then fails without another try, and a dryrun reports the node tasknot_checked, because an earlier node task may install the program.pacman's reason tells the user to add a condition on the node fact
package_manager. A node fact is a value that MMX reads on the node. On a distribution outside the families that MMX knows,.node.package_manageris unavailable, and a node task that reads it fails. So your reason names the node factdistribution.id, theIDin the node's/etc/os-release. A playbook that runs on several distributions then reads:- when: equals: ["{{ .node.distribution.id }}", mydistro] mytool_package: packages: [nginx]For the checks below, the reason reads:
"reason": MODULE + ": the program mytool is not installed; mytool_package works only on a node whose distribution is mydistro; " + "give the node task a condition on the node fact distribution.id",Check: run
mmx dryrun test_mytool_packageontest01, which has nomytool. The dryrun no longer sayspacman_package works only on a node whose package manager is pacman. It reports the node tasknot_checkedwith your reason, and the exit code is 3:install_nginx / test01: succeeded install_nginx / test01 / nginx/changes/install_nginx_package: not_checked, unknown — mytool_package: the program mytool is not installed; mytool_package works only on a node whose distribution is mydistro; give the node task a condition on the node fact distribution.id install_nginx / test01 / nginx/activate/start_nginx: skipped, unknown — service state can be checked after the node task block's proposed changes are applied install_nginx / test01 / nginx: succeeded, unknown Run 55abc510: succeeded 1 finishedThen run
mmx run test_mytool_package. The run fails at once, with the failure kindmissing_prerequisiteand without another try, and the exit code is 1:install_nginx / test01: failed nginx: nginx/changes/install_nginx_package: mytool_package: the program mytool is not installed; mytool_package works only on a node whose distribution is mydistro; give the node task a condition on the node fact distribution.id install_nginx / test01 / nginx/changes/install_nginx_package: failed, unchanged — mytool_package: the program mytool is not installed; mytool_package works only on a node whose distribution is mydistro; give the node task a condition on the node fact distribution.id install_nginx / test01 / nginx: failed, unchanged — nginx/changes/install_nginx_package: mytool_package: the program mytool is not installed; mytool_package works only on a node whose distribution is mydistro; give the node task a condition on the node fact distribution.id Run afb21223: failed 1 failedmmx results afb21223 --jsonshowsfailure_kind: missing_prerequisiteandnode_job_attempt: 1. These outputs come from a Debiantest01, which has nomytool. On a node that hasmytool, the node task goes on to the next part of the script. -
Replace the list of installed packages.
read_installedrunspacman -Qand returns a dictionary of versions by name:def read_installed(): answer = run_command(["pacman", "-Q"]) if answer.error or answer.exit_code != 0: return None, message_of(answer) return read_lines_of_names_and_versions(answer.standard_output)Put in your package manager's command that prints one line for each installed package, with its name and version. If the lines have another form than
name version, changeread_lines_of_names_and_versionstoo.run_commandruns a command that only reads, also in a dryrun. Every later part compares with this dictionary.Also change
message_of, which picks pacman's error lines out of a failed command. Your package manager's errors start with other words.Check: on your test server, which has your package manager, run
mmx dryrun test_mytool_package. If the dryrun fails with a reason that startsmytool_package: reading the installed packages:, the command or the form of its lines is wrong; the reason ends with your package manager's own message. If that reason is gone, the node task reaches the next part that still calls pacman, and that part fails until you replace it in step 7. -
Replace the preview. Before any change, the script asks the package manager what a transaction would do, without doing it. pacman's dry run is
--print, written-p, inpreview:answer = run_command(["pacman"] + arguments + ["--noconfirm"] + PRINT_FORMAT)Change
previewto your package manager's dry run, and the arguments thatpreview_installationandremovepass to it. The preview answers three questions:- Which packages would change, with their versions. A dryrun
reports them, and a removal refuses packages outside
packagesunlessalso_remove_packages_that_depend_on_theseis true. - Which names the package index does not have. pacman prints
error: target not found: <name>for each, andpreviewcollects them. Change that text to your package manager's. - Which names the package manager installs under other names, such
as a group of packages or a name that another package provides, such
as
sh. The copied script refuses such a name, because the package manager installs and removes the name under other names, so the task would find the name missing on every run.
A removal needs the same check.
installed_under_other_namesasks pacman, for each name that is not installed under its own name, whether the name is a group with installed members or a name that an installed package provides. Write down the meaning of each exit code that the script relies on, as the copied script does:elif provided.exit_code != 127: # pacman -T exits with 127 for a name that nothing installed # provides. return None, message_of(provided)Check: on your test server, without nginx, run
mmx dryrun test_mytool_package --diff. The package node task reportswould_changewith nginx inchanged_packages, and the reason sayswould install nginxwith the version that your package manager's preview gave. The node task forstart_nginxisskipped, because the service can be checked only after the package is installed, and the dryrun's exit code is 3. Thewould run:line still names pacman until step 10. A name that your package index lacks isnot_checked, not a failure, because a run refreshes the index first. - Which packages would change, with their versions. A dryrun
reports them, and a removal refuses packages outside
-
Replace the lock. A package manager holds a lock while it changes packages or refreshes its index. Before each command that takes the lock,
wait_for_the_lockwaits for up toLOCK_WAIT_SECONDS, which is 60, and then fails with a reason that says what to do. Starlark has nowhile, so the wait is a loop with a fixed number of rounds:for _ in range(LOCK_WAIT_SECONDS): lock = inspect_path(LOCK_FILE) if lock.error: return {"status": "failed", "reason": MODULE + ": looking for pacman's lock " + LOCK_FILE + ": " + lock.error} if not lock.exists: return None run_command(["sleep", "1"])pacman's lock is a file that exists only while pacman runs. If your package manager also works this way, change
LOCK_FILEand the messages. Some package managers, such as dpkg, keep a lock file that always exists and lock the file while they run, so the file's existence says nothing. For such a package manager, use its own option to wait for the lock, if it has one. Reading the installed packages and the preview take no lock.Check: none yet. The lock is checked in step 13, in a run.
-
Replace the refresh of the package index. The refresh goes through the primitive
run_command_that_refreshes_a_package_index:refreshed = run_command_that_refreshes_a_package_index(["pacman", "-Sy", "--noconfirm"])Keep this primitive, and change only the command. The package index is a cache, so MMX does not count the refresh as a change of the node task. A whole-system upgrade with nothing newer then reports
unchanged. A dryrun refreshes nothing:refresh_the_package_databasesreturns at once whenis_dryrun()is true, before it waits for the lock.Keep the order that the copied
installuses: preview against the index that the node has, then refresh, then preview again. A name with a character that breaks your rules fails in step 4, before any refresh. A name that the node's index lacks may be in a newer index, so a run refreshes the index first, and fails withinvalid_inputsonly when the refreshed index still lacks the name.pacman has a rule of its own here: Arch Linux supports no partial upgrade, so each installation also upgrades the whole system, as
pacman -Syu <names>does, in two steps. Most package managers allow an installation without a whole-system upgrade. If yours does, the installation refreshes the index and installs only the named packages and their dependencies, andinstallcan drop the-upart.Check: none yet. The refresh is checked in step 13.
-
Replace the transaction.
run_transactionruns the command that changes packages:# In a dryrun MMX does not run the command. The call is still what makes # the dryrun report would_change and show "would run: pacman ..." in # the difference; a script never reports its change itself. command = run_command_that_changes_the_node(["pacman"] + command_arguments + ["--noconfirm"])Change
pacmanand the arguments thatinstall,remove, andupgrade_allpass. Run the package manager without questions, as--noconfirmdoes for pacman, because no terminal answers them.Call
run_command_that_changes_the_nodein a dryrun too. Do not wrap the call inif not is_dryrun(). In a dryrun, MMX runs nothing, and the call is how the node task reportswould_changeand how--diffshows the command.Check: on your test server, without nginx, run
mmx dryrun test_mytool_package --diff. Thewould run:line under the package node task now names your package manager's command, with the arguments you set, and no longer names pacman. -
Run the test playbook.
mmx run test_mytool_packageCheck: the package node task says
succeeded, changedand lists what the transaction changed, and nginx starts. With pacman, on an Arch Linux node:install_nginx / test01: succeeded install_nginx / test01 / nginx/changes/install_nginx_package: succeeded, changed — installed mailcap 2.1.54-3, nginx 1.30.6-3; upgraded dbus from 1.16.2-2 to 1.16.4-1, libp11-kit from 0.26.5-1 to 0.26.6-1, p11-kit from 0.26.5-1 to 0.26.6-1, python-jsonpatch from 1.45-1 to 1.46-1, python-jsonpointer from 3.8-1 to 3.12-1 changed_packages: [{"from":"1.16.2-2","name":"dbus","to":"1.16.4-1"},{"from":"0.26.5-1","name":"libp11-kit","to":"0.26.6-1"},{"name":"mailcap","to":"2.1.54-3"},{"name":"nginx","to":"1.30.6-3"},{"from":"0.26.5-1","name":"p11-kit","to":"0.26.6-1"},{"from":"1.45-1","name":"python-jsonpatch","to":"1.46-1"},{"from":"3.8-1","name":"python-jsonpointer","to":"3.12-1"}] install_nginx / test01 / nginx/activate/start_nginx: succeeded, changed — service is not running install_nginx / test01 / nginx: succeeded, changed Run 95aa8ded: succeededchanged_packageslists every package that the transaction changed, also dependencies and upgrades outsidepackages. The upgrades here come from pacman's rule of step 9. On the node, list the installed packages to see that the result is true:mmx run-command --nodes test01 -- pacman -Q nginxUse your package manager's listing command in place of
pacman -Q. If the run fails, its reason names the part of the script and the package manager's own message. -
Write the module's page. Replace
external_modules/mytool_package/README.md, which still describes pacman. Keep its sections, and write in each what is true for your package manager: what the module does, its inputs, and its examples. Write down where your package manager differs from the rules that every package module follows, aspacman_package's page does with pacman's own rules. Under "What a dryrun reports", write what your package manager's dry run does not check. pacman's dry run checks no conflict between packages, so a conflict shows only in a run.Also rewrite the descriptions in
module_description.yml, which still name pacman.mmx describe-module mytool_packageprints them.Check:
mmx describe-module mytool_packagenames your package manager, andgrep -n pacman external_modules/mytool_package/*finds nothing that is still about pacman. -
Run the checks on the test node. Each check in The checks shows one rule of MMX's package modules. Run each one before you use the module on other servers.
Check: each check gives the result that its row describes.
The checks
Each check runs on test01. The output shown is from mytool_package as
copied from pacman_package, on Arch Linux. Your module prints its own
package manager's names and messages in the same places.
| Check | What to do | The result you must see |
|---|---|---|
| A second run changes nothing | Run mmx run test_mytool_package twice. Before and after the second run, look at the modification times of the package index's files, such as with stat. |
The second run reports each node task unchanged, with changed_packages: []. The index's files keep their modification times: a node task with nothing to install refreshes nothing. |
| A dryrun changes nothing | Remove nginx by hand. List the installed packages, run mmx dryrun test_mytool_package --diff, and list them again. |
The dryrun reports would_change with nginx in changed_packages, and would run: with your command. Both lists of installed packages are the same, and the index's files keep their modification times. |
| A new version restarts the service | Install another version of nginx by hand, such as an older one. On Arch Linux: pacman -U https://archive.archlinux.org/packages/n/nginx/nginx-1.30.6-2-x86_64.pkg.tar.zst. Run the playbook twice. |
The first run reports start_nginx: succeeded, changed — tracked input module mytool_package changed after the service last started or reloaded. The second run reports start_nginx: succeeded, unchanged. |
| A misspelled name fails without another try | Write nginxx in the test playbook, and run it. |
The dryrun reports not_checked, because a run refreshes the index first. The run fails once with invalid_inputs, after the refresh: mytool_package.packages: pacman's package databases have no package nginxx; .... |
| A name that breaks a rule fails before the refresh | Write a name with a character that your rules refuse, such as extra/nginx for pacman, and run the playbook. |
The run fails with your rule, as in step 4, and the index's files keep their modification times. |
| A name of a group or a provided name is refused | Write a group's name, or a name that another package provides, such as sh, with state: absent too. |
mytool_package.packages: sh is a group or a name that another package provides, not a package of its own; name each package itself. If your module handles such names on purpose, its page says how. |
| A node without the program reports a missing prerequisite | Point the test playbook at a node of another distribution, such as nodes: web01 on Debian. |
The dryrun reports not_checked with your reason. The run fails once, without another try, with the same reason. |
| The lock makes the node task wait | Hold the package manager's lock on the node, remove nginx by hand, and run the playbook. For pacman: touch /var/lib/pacman/db.lck. Then release the lock and run again. |
The run fails after the wait: another pacman has held its lock /var/lib/pacman/db.lck for 60 seconds; .... MMX tries a failed node task three times by default, so the run took about three minutes. After the lock is gone, the next run installs nginx. |
| What the dry run does not check fails in a run | Do what your package manager's dry run does not check. For pacman, install a package that conflicts with an installed one, such as nginx-mainline beside nginx. |
The dryrun reports would_change. The run fails with the package manager's own message, and reports the node task failed, changed: mytool_package: installing nginx-mainline: nginx-mainline-1.31.8-2 and nginx-1.30.6-2 are in conflict; unresolvable package conflicts detected; failed to prepare transaction (conflicting dependencies). |
| The exit codes mean what the comments say | Run each command that the script tests for an exit code, with a name that exists and a name that does not. | The exit codes match the comments in module.star, such as pacman -T exits with 127 for a name that nothing installed provides. |
The last row's changed comes from a rule of MMX: each finished call of
run_command_that_changes_the_node counts as a change, also one that ends
with a nonzero exit code. The module interface reference
gives the reason.
Reinstalling the same version by hand restarts nothing: the module records each package's version, and the version is the same as before.
What the copied script already does
These parts of the copied script work for any package manager, so most modules keep them as they are:
-
Reading again after the transaction. A run reads the installed packages again and compares them with the first read.
changes_betweenreturns one{name, from, to}entry for each package whose version differs. The outputchanged_packagesholds the entries, and the reason says the same in words, with at most 10 packages of each kind, set inLISTED_IN_A_REASON. A dryrun has no second read, so it uses the versions that the preview predicted. -
A failure after the transaction stays a failure. If the second read fails, the result has
try_again: False. Another try would find the packages as the node task wants them, report success, and hide the failure. -
The fingerprint. A package module keeps no file through a changing primitive, so the script's own fingerprint is the whole fingerprint. The tracked-input rule decides what a changed fingerprint restarts. The fingerprint holds the version of each named package, or
absent:state = {} for name in sorted(names): state[name] = versions.get(name, "absent") result = {"outputs": {"changed_packages": changes}, "fingerprint": json.encode(state)}Each successful result carries the fingerprint, also an unchanged one. A dryrun's fingerprint uses the predicted versions, so the dryrun predicts the restart. A whole-system upgrade names no package and gives no versions to the fingerprint.
In a dryrun, the same run_node_task runs. The reads and the preview
run. The lock wait and the refresh do not. The transaction's call records
would run: ... and changes nothing. A name to install that the node's
package index lacks is not_checked in a dryrun, because a run refreshes
the index first, and an earlier node task may add the package server that
has the package. A missing program is not_checked too. Every other
failure, such as an input that breaks a rule or a removal that would take
other packages, fails in a dryrun as in a run.
What MMX's own package modules have that yours does not
A package module in Starlark gets what every script gets, and passes the
same checks. It lacks two checks that apt_package and dnf_package
have:
- The check across a fleet stage.
apt_packageanddnf_packagecheck on the controller machine that two node tasks of one fleet stage do not ask for conflicting versions of one package. Only modules built into MMX can make this check. - The compiler's checks. A mistake in a script shows only when that part of the script runs. So the checks above must reach each part of the script.
If something is different
| What you see | What it means and what to do |
|---|---|
mmx: external_modules/pacman_package: pacman_package is an official extra module of MMX; give the module another name |
The copy kept the name pacman_package. Rename the directory, as in step 1. |
examples.0.config: the example does not name the module mytool_package |
An example in module_description.yml still names pacman_package. Do step 2. |
the script of the module mytool_package ended with an error: module/mytool_package/module.star:29:25: got newline, want primary expression |
A mistake in module.star, at line 29, column 25. mmx validate does not run the script, so only a dryrun or a run finds the mistake. |
the program pacman is not installed on your distribution |
find_program still looks for pacman. Do step 5. |
would run: still names pacman |
The transaction still runs pacman. Do step 10. |
A second run reports changed again |
The listing of step 6 and the names in packages do not match, such as a name that the package manager installs under another name. Compare the listing with packages, and refuse such names, as step 7 says. |
A whole-system upgrade reports changed on each run |
The refresh goes through run_command_that_changes_the_node. Use run_command_that_refreshes_a_package_index, as step 9 says. |
| The service does not restart after a package changed | The package's node task is not in the same node task block's changes as the service's activate node task. |
Where to go next
- Write your own module: how a module of the project's own works, with a smaller example, and the general tests for any module, such as a repeat after a partial failure.
- Module interface reference: every primitive, status, and field that a script uses.
pacman_package's page and its script: the module that this page copies.zypper_package's script: another package module in Starlark, whosepackagesentries each take anameand aversion.- Reuse work with roles: one role that picks the package module for each distribution.
This page is docs/how-to/write-a-package-module.md in the MMX repository.