valis / Running a node
valis Operations Runbook
This is the hands-on companion to the host-deployment contract. The contract states what a host must provide and the fail-closed order a node comes up in; this runbook states what the operator does to install, run, observe, renew, back up, and restart a deployed node. Where the contract is the invariant, the runbook is the procedure.
Audience. The operator running a deployed unit, not the developer building valis from a REPL. For the inner development loop see DEVELOPMENT.org; for building your own system on the substrate see BUILDING.org.
The shape of a running node. One Type=notify systemd unit supervises the
fulcrum=+=valis process pair. The privileged fulcrum parent creates the network
namespace, binds the public sockets its launch configuration names (:53 when
:designated-ports includes 53, the TLS edge when the configuration names an
:edge-port, and the owner port when it sets :open-owner-port), loads the sk_lookup
director, and forks the unprivileged valis resident after a setpriv drop. The
director funnels every inbound TCP connection in the node's network namespace, loopback
included, into valis. valis adopts the inherited descriptors, brings its fabric and
edges up fail-closed, and emits READY=1 once every bring-up step has returned.
Readiness attests that the fabric, the edge, and the active modules came up without
error; it does not attest that anything has been served, and what keeps the unit alive
afterwards is a liveness check on those threads rather than a check that traffic is
moving. The unit is the supervisor of last resort: it restarts the whole image if the
image dies; the active-module seam inside the image supervises the modules.
1. Host filesystem layout
The shipped unit expresses an FHS layout the launcher threads into valis's XDG rooting. Provision these before first start:
| Path | Role | Ownership |
|---|---|---|
/opt/valis/bin/ |
the staged delivery binary (fulcrum-resident), world-traversable |
root, o+x chain |
/etc/valis/ |
the bootstrap EnvironmentFile seed only (read-only to service) |
root:valis 0750 |
/var/lib/valis/ |
StateDirectory: durable irreplaceable state (owner seed, store, ACME custody, the node's configuration under valis/config/) |
valis:valis 0700 |
/var/lib/valis-releases/ |
the release root: placed releases, the selection record, the release lifetime record, staging, and the release jail | valis:valis 0755, world-traversable |
/var/lib/valis-launch/ |
fulcrum's launch marks, one record of each trial's boot | root:root 0755, never the run account's |
/run/valis/ |
RuntimeDirectory: the AF_LOCAL control socket + pidfiles |
auto, ephemeral |
StateDirectory holds the irreplaceable durable state. systemd creates and chowns it
on first boot; the parent's CAP_CHOWN=/=CAP_FOWNER make the owner-seed write under
it succeed. Everything backup-critical lives here, and the valis --backup verb seals
the backup-critical subset out of it, see 16.
The release root and the launch-mark directory are the unit's second and third state
directories (StateDirectory=valis valis-releases valis-launch, valis first, since the
launcher takes the first as the data home). systemd hands every state directory back to
root at each start, and fulcrum then gives the release root and its contents back to the
run account and sets both directories to 0755. The release root must stay
world-traversable because fulcrum refuses a binary the run account cannot reach; the
mark directory stays root's, so the run account can never erase the record of a trial.
fulcrum finds both through its launcher configuration (see
Releasing a new binary to a running node).
The binary must be the serving-capable delivery core, not a plain :valis core: the
plain core lacks the additive DNS-serving stack and fails closed at boot with c3po's
serving codec … not available. Stage it from the versioned delivery tarball that
make dist produces (valis-serving-<version>.tar.gz); never rebuild the core on the
host. See the contract's
serving-capable resident binary section.
2. Installing the unit
The units ship under deploy/systemd/. Install them onto the host:
# The supervisor unit.
install -m 0644 deploy/systemd/valis.service /etc/systemd/system/valis.service
# The bootstrap environment seed. Copy the template, fill it, lock it down.
install -d -m 0750 -o root -g valis /etc/valis
install -m 0640 -o root -g valis deploy/systemd/valis.env.example /etc/valis/valis.env
# ... edit /etc/valis/valis.env (next section) ...
systemctl daemon-reload
The ExecStart is /opt/valis/bin/fulcrum-resident, the fulcrum delivery binary
that owns the whole bring-up. Stage that binary and the run account per the contract's
provisioning surface before enabling the unit.
3. The environment seam: /etc/valis/valis.env
/etc/valis/valis.env is the node's genesis seed. The node reads each value in it
once, at the first boot that considers that value's key, and keeps the result in its
configuration store under the data root: <StateDirectory>/valis/config/, which is
/var/lib/valis/valis/config/ under the shipped unit. From then on the store is the
node's configuration. Change a setting with valis config or through /config over the
owner session (see changing a setting). Editing valis.env
afterwards has no effect on a key the node has already considered, and the boot log
names each such key as superseded by the store, by name and never by value. A key that
a newer release adds is seeded from this file once, at that release's first boot.
The file is re-authored by provisioning, is 0640 root:valis, and never enters the
tracked repo. Its values travel nowhere: the node's configuration directory is what a
backup carries (see backup and restore).
valis apply checks the site facts a boot needs against the node's own configuration
first and this file second, the order the node reads them in. Without
--confirm-zone it confirms the node's own edge.domain, unless you have set one of
your own (see below). When apply cannot read the node's configuration,
it checks this file alone and says so; see letting apply read the node's settings.
Each variable below seeds the configuration key beside it:
| Variable | Configuration key | Meaning / effect when unset |
|---|---|---|
VALIS_PG_DSN |
operator-state.dsn |
operator-state Postgres URI over the local UNIX-domain socket under peer authentication, e.g. postgresql://valis@/valis_state?host=/var/run/postgresql. No password, no TCP listener: the kernel proves the run account's uid and Postgres maps it to the role. The configuration store refuses an address that carries a password, and says why, so no secret reaches it or a backup; use the socket form. An address with a password stays in the environment, is read from there at every boot, and the boot logs that the key is held in the environment because it carries a secret. |
VALIS_EDGE_DOMAIN |
edge.domain |
the DNS name the :443 edge serves a certificate for. Unset ⇒ :443 stays dark (fail-closed: no domain named). Naming it is necessary and not sufficient: the launcher config must also name an :edge-port, or no socket is bound for the edge and the firewall accepts none. |
VALIS_ACME_PRODUCTION |
acme.production |
truthy ⇒ the real Let's Encrypt production CA (strict rate ceiling). Unset/empty ⇒ staging (the fail-safe default; keeps dev/CI off production). |
VALIS_ACME_DIRECTORY_URL |
acme.directory-url |
override the ACME directory (a Pebble/CI or alternate staging endpoint). Honored only when the production opt-in is not set: production always wins. |
VALIS_ACME_STORE_PATH |
acme.store-path |
the custody store the :443 edge loads its cert from and the renewal manager writes into. In the shipped unit, under StateDirectory, e.g. /var/lib/valis/acme. Unset ⇒ the resident refuses to boot, on every boot including a genesis one: an unchosen store root would mint a fresh ACME account against the CA and spend registration budget nothing gives back, so it is required rather than defaulted. The shipped valis.env.example already carries a value. A first boot without it settles the key empty, so editing valis.env afterwards has no effect: once a boot has refused for this, set it with the node stopped, as the run account: sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis config set acme.store-path /var/lib/valis/acme. |
VALIS_ACME_ACCOUNT_CONTACT |
acme.account-contact |
the operator's ACME account contact, the address this node's ACME account is registered to. Unset/blank ⇒ automatic renewal refuses rather than ordering, and says so on the boot line: a contact is an identity, and one invented for you would register the account to an address you never chose. A first order can carry a contact typed at valis obtain --contact; a renewal falls due months later with nobody at the keyboard, and this is where it reads one. |
VALIS_MODULE_SOURCE_ROOT |
module.source-root |
where the bytes of every module this node admits are kept, one directory per module identity. In the shipped unit, under StateDirectory, e.g. /var/lib/valis/modules. Unset means the resident refuses to boot, and with it set the resident creates the directory at boot. The shipped valis.env.example already carries a value. A first boot without it settles the key empty, so editing valis.env afterwards has no effect: once a boot has refused for this, set it with the node stopped, as the run account: sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis config set module.source-root /var/lib/valis/modules. See the module source root. |
VALIS_EDGE_DOMAINS |
edge.domains |
further names declared primary for this node, comma or space separated. They add to edge.domain and never replace it, and the edge still needs that base name to open :443. Unset means the base name alone. |
VALIS_MAIL_LOCAL_DOMAINS |
mail.local-domains |
the domains this node accepts mail for at RCPT and delivers locally, comma or space separated. The development listener's notice goes to postmaster at the first. While mail.enabled is unset, naming a domain turns the mail switch on (see Mail on port 25). A running node takes a change at once. Unset means no local domain and the switch off by default. |
VALIS_MAIL_HOST_NAME |
mail.host-name |
the canonical name of the node's mail host: the name its greeting on :25 carries, its PTR record should answer, and its STARTTLS certificate is ordered for. Unset means mail. followed by the first local domain. A running node takes a change at its next connection. |
VALIS_MAIL_SESSION_CEILING |
mail.session-ceiling |
how many sessions :25 holds at once. Unset means 16. A whole number from 1 to 32, so mail never takes more than half the 64 connections every steered port shares. A connection over the ceiling is answered 421. A running node reads it at each connection, so a change applies to the next one. |
VALIS_MAIL_SECONDARY_FOR |
mail.secondary-for |
the domains this node is backup MX for, as domain=primary-host tokens. A list with a member naming no primary host is refused whole. Unset means the node is backup MX for none. |
VALIS_MAIL_ACCEPT_LOCALPARTS |
mail.accept-localparts |
the local parts accepted at RCPT, comma or space separated. postmaster and abuse are always accepted besides. Unset leaves the built-in accept list in force; it never widens acceptance to a catch-all. A running node takes a change at once. |
VALIS_MAIL_RESOLVE_DOT_HOST |
mail.resolve-dot-host |
the DNS-over-TLS upstream that outbound mail resolves recipient MX records through. Unset means every relay is deferred, because there is no smarthost to fall back to. |
VALIS_MAIL_RESOLVE_DOT_ADN |
mail.resolve-dot-adn |
the name that upstream's certificate is verified against, which is also the SNI sent to it. |
VALIS_MAIL_RESOLVE_CA_FILE |
mail.resolve-ca-file |
the CA bundle that verifies that upstream. Unset is an empty trust store and not a bypass: every peer is rejected, so mail defers until you name one. |
VALIS_TRANSFER_MASTER_ADDRESS |
names.transfer-master-address |
this node's public authoritative DNS address: the source of outbound NOTIFY, the address a secondary pulls from, and the address a new domain resolves to when you name no other. zone create, and zone secondary without --offline, ask the running node for it over the owner session when no flag names the address. Unset means those verbs refuse rather than pick an address. |
VALIS_SECONDARY_NS |
names.secondary-ns |
the nameserver name published beside this node's own in every domain's apex NS set. Unset means the apex names this node alone. |
VALIS_SECONDARY_TRANSFER_ADDRESS |
names.secondary-transfer-address |
the address the secondary pulls zone transfers from. zone create enrols that address when you pass no --peer. Unset means such a create refuses. |
VALIS_DEPENDENCY_DIST_URL |
dist.url |
the private dist valis dist register registers as the node's dependency source, when --location is not given. Unset means the verb refuses with exit 2. |
VALIS_DEPENDENCY_DIST_HOME |
dist.home |
where valis dist register records the registration, when --home is not given. Unset means valis/dependency-dist/ under the data home. |
VALIS_NET_DIAL_ALLOW |
net.dial-allow |
the special-purpose address ranges each client may reach when somebody other than you chose where it connects, as client=range members with one range to a member and a client repeated for several, e.g. delegation-probe=192.168.140.0/22. A range lying wholly inside the floor (the unspecified addresses, multicast and limited broadcast) is refused, because nothing ever reaches the floor. A range with a bit set past its prefix length is refused too. A list with one refused member is refused whole. An allowance stored under looser rules than the node now applies allows nothing, and the node logs config.allowance-refused once, naming net.dial-allow and why. Set it again under the current rules to restore it. Unset means no client is allowed any such range. A running node reads it as it makes each such connection, so a change applies to the next one. valis zone delegation reads it as well, for the client delegation-probe, unless you pass --allow-range, so the example lets that check ask nameservers in a lab's private network. A module you admitted is a client under its ASDF system name: example-client=10.0.0.0/8 lets the module admitted as example-client reach that network, and no other module. The allowance follows the system name, whichever vouched module carries it: a replacement, or another publisher's module admitted under the same name, inherits it. A module whose system name a client name cannot carry (anything but lower-case letters, digits, dots and hyphens, or a first character that is not a letter or a digit) can be allowed nothing, and neither can one named as one of the node's own clients (acme, delegation-probe, dist, secondary-forward, zone-notify). |
VALIS_OWNER_PORT_SESSIONS_PER_ORIGIN |
owner-port.sessions-per-origin |
how many connections the owner port holds at once from one origin, counted from the moment a connection reaches it, handshake included. Unset means 4. A whole number from 1 to 63. One origin is one IPv4 address or one IPv6 /64, and an IPv6 address carrying an IPv4 one counts as that IPv4 address, so machines behind one public IPv4 address count as one origin. A refused connection is logged at WARN at most once per origin per minute. A running node reads the setting at each connection, so a change applies to the next one. |
VALIS_OWNER_PORT_SESSION_CEILING |
owner-port.session-ceiling |
how many connections the owner port holds at once from every origin together. Unset means 16. A whole number from 1 to 63, which leaves at least one of the 64 connections every steered port shares to the other ports. A running node reads the setting at each connection, so a change applies to the next one. |
VALIS_MODULE_STANDING_GRANT_REVIEW_DAYS |
module.standing-grant-review-days |
how many days an active standing grant may go without a review before the owner's grant inventory, /cap/grants, marks it review due, counted from the later of its filing and its latest review. Unset means 90. A whole number from 1 to 3,650. The mark is advisory: nothing stops a module being admitted under a grant that is due. While any active standing grant is due, the node's status line ends by counting them and naming the interval. A running node reads it each time the inventory is read and each time the health loop refreshes the status line, every 15 seconds, so a change applies at the next of those. |
VALIS_RUNTIME_ROOT |
runtime.root |
the directory the node keeps its runtime files in, such as the development listener's socket. Unset means /run/valis/valis/, the run account's directory beneath the host agent's own /run/valis/. It must lie outside the data root, so a backup never captures what is in it; the configuration store refuses one beneath it. valis never creates it: the host agent makes it valis:valis 0750 at each start, so members of group valis may enter it and nobody else may, and the listener refuses to open under a root the run account does not own or that its group or others may write. A change takes effect at the next restart. |
VALIS_SLYNK_PORT |
development.listener-port |
Retired. The node no longer opens a TCP development listener, and it reads neither the variable nor the key. A node that still holds a stored port logs development-listener.retired-port-stored at WARN at each boot and opens nothing. Remove both once, as Retire the old listener port shows. |
The database DSN names a local UNIX-domain socket, so the resident reaches Postgres
by filesystem path under peer authentication rather than over the network: no password
travels in the environment or the configuration, and no TCP listener need be open.
Put the socket-path DSN in valis.env for the first boot, and change it afterwards
with valis config set operator-state.dsn in the form
changing a setting gives. (Moving the database onto a
dedicated address inside the netns is deferred hardening, and the node would refuse it
until the loopback-only rule is widened for it; the host is the root of trust for the
resident copy.)
A DSN naming a TCP host is accepted only when that host is loopback: an address in
127.0.0.0/8 (in either IPv4 or IPv4-mapped form), ::1, or localhost. A private-use, link-local, or unique-local
address is refused like any other, and a host name other than localhost is never
resolved. The startup check and every connection the node opens to its database check
this first, and refuse any other host with operator-state-host-not-local before
connecting, so a TCP connection to the database never leaves the node. The only TCP
database in use is the development one on 127.0.0.1; a deployed node uses the socket.
The tools you run read seven more variables, from the environment you run them in. They are not node configuration and have no configuration key: the node never
reads them, and they do not belong in valis.env.
| Variable | Meaning / effect when unset |
|---|---|
VALIS_KEYFILE |
the owner keyfile the owner-keyed verbs present, when --keyfile is not given. Unset means the keyfile in the data root of whoever runs the verb. A path only: the key bytes never cross the command line. The file must be a regular file, not a link, owned by the account running the verb, private to it, and exactly the 32 octets of a seed; anything else is refused. The key named is the node's own, so the verb accepts only that node as the responder: one that cannot prove it holds the key is refused. |
VALIS_MGMT_ENDPOINT |
the management endpoint, as HOST:PORT, the owner-keyed verbs connect to when --endpoint is not given. Unset means the endpoint file the resident writes. The port changes at every boot, so read it fresh rather than setting this once; see the break-glass route. |
VALIS_NODE |
the pinned node the owner-keyed verbs reach, by its name in the pin store, when --node is not given. Unset means no node is named, and the verbs use the keyfile and the endpoint. Under a node, VALIS_KEYFILE and VALIS_MGMT_ENDPOINT are not read, and a --keyfile or --endpoint flag is refused rather than dropping the pin. |
VALIS_KEY |
the device key presented to a pinned node, when --key is not given. Unset means this machine's device key, the one valis key generate writes under $XDG_CONFIG_HOME/valis/, or under ~/.config/valis/ when XDG_CONFIG_HOME is unset or not an absolute path. A device key is presented only to a pinned node, so VALIS_KEY or --key with no node named is refused. |
VALIS_APPLY_CONFIRM_SERVER |
the server valis apply queries to confirm the node is serving, when --confirm-server is not given. There is no default: with neither, the apply refuses before anything moves. --confirm-zone falls back to edge.domain in the configuration under the invoking account's own data root, or to VALIS_EDGE_DOMAIN in the invoking shell, and then to the node's own edge.domain; when your value and the node's disagree, the apply refuses and names both. |
VALIS_APPLY_TARGET |
the directory valis apply replaces the serving binary in, when --destination is not given. Unset means /opt/valis/bin/. |
VALIS_APPLY_UNIT |
the unit valis apply restarts, when --unit is not given. Unset means valis.service. |
One more is read only on the host that builds the binary, while it compiles.
No node reads it, and it does not belong in valis.env either.
| Variable | Meaning / effect when unset |
|---|---|
VALIS_BAKE_MANUAL_DIR |
the staged manual the binary carries, a directory make site site-stage wrote. make dist sets it for you, to the manual it built from the commit it releases, so a fresh node serves the whole manual before anything has been published to it. Unset means the binary carries no manual and a fresh node serves the landing page alone. A directory that is missing, holds no pages, holds a page whose name cannot be a publication slug, or holds no front door stops the build. |
4. Changing a setting
valis config reads and changes the node's settings and any module's. Against a
running node it goes through the node's owner session, so from your own machine you
name the node, as you do for every owner verb (see
Reaching a node's owner control plane):
valis --node NODE config list # every node key: path, origin, type, when it applies, value
valis --node NODE config list SYSTEM # a module's keys, and what an upgrade kept aside
valis --node NODE config get edge.domain
valis --node NODE config set edge.domain example.org
valis --node NODE config reset edge.domain # back to the shipped default
valis --node NODE config menu # walk the node's settings, one at a time
valis --node NODE config interview SYSTEM # a module's own questions; a blank answer keeps a value
valis --node NODE config set --module SYSTEM KEY VALUE # a module's key
valis --node NODE config purge SYSTEM # delete a removed module's kept settings
config get prints one line of five fields separated by tabs: the key, where its value
came from (default or operator), its type, when a change to it applies, and the
value, which is empty when the key is unset. A value set equal to the default reads
back with the origin default. A change answers with when it takes effect, described
below, then key KEY and origin ORIGIN. With a key the node has not enrolled, the
verb is refused with exit 2 and changes nothing.
From the node's host, the break-glass route. When you cannot use the route from your machine (the owner port closed, the pin or the device key lost, or the node not answering on the owner port), run the verb on the node's host the way you run the other owner-keyed verbs there, inside the node's namespace with its keyfile and endpoint named (see the break-glass route):
v() {
sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
/opt/valis/bin/valis config \
--keyfile /var/lib/valis/valis/keyfile \
--endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)" "$@"
}
v list
v get edge.domain
v set edge.domain example.org
On the host, reading the node's own settings does not need the owner session.
config list and config get KEY, run as the run account on the node's data root, read
its store directly, running or stopped. The read takes no lock and writes nothing, so it
never delays the node:
sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis config get edge.domain
I made reads work this way because a running node's owner session lives inside its
network namespace, out of reach of the host's, and reading a setting should not need
nsenter and root. A module's settings are still read over the owner session, since
only a node that has loaded the module can list them.
Against a stopped node, run it as the run account on the node's data root, and it changes the node's own settings directly:
sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis config set edge.domain example.org
A module's settings change only through the running node, since only a node that has
loaded the module can check a value for it. A secret is only ever shown as set or
unset. Run against a stopped node as any account but the one owning its data root,
the verb is refused with exit 3, as a configuration that could not be changed. Run
without --keyfile and --endpoint, it acts on the data root of the account running
it, so under sudo alone it reads and writes root's own store and not the node's.
A change answers with when it takes effect:
committed-applied: the value is stored and the running node is using it.committed-on-restart: the value is stored and the node takes it at the next restart. ⚠ This is a success, not a fault: some keys, such as the runtime root and the ACME store, only ever take effect at a restart, and every change to a stopped node answers this way.committed-rebind-failed: the value is stored, the running code refused it, and areasonline names the kind of condition. The next restart applies the value.
A purge answers purged or nothing-to-purge, and a purge of a loaded module also
answers when its return to the defaults takes effect, as one of the outcomes above. A refusal names what was wrong and
changes nothing. The exit status is 0 when the verb read or committed, 2 when the node
or the verb refused and nothing changed, which includes a key file that is missing or
one the verb will not present, and 3 when the node could not be reached or its configuration could not be
changed.
The configuration directory is part of every backup, so a restored node comes back with the settings it had.
4.1. The served set
The ports a node serves are one setting, edge.served-ports, and you read and change
them from your own machine like any other. Read the set first:
valis --node NODE config get edge.served-ports
The value is the ports in ascending order, separated by commas: 53,80 on a node that
kept the default core. Change it with config set and give the whole set. The value
you set replaces the set, so a port you leave out stops being served. To add 8099 to
that set:
valis --node NODE config set edge.served-ports 53,80,8099
valis --node NODE config get edge.served-ports
Keep every member you read unless you mean to stop serving it. A set without 53 stops
DNS, and an empty set is a choice the node keeps: it serves nothing, and a restart does
not bring the default core back. The change answers committed-applied and takes
effect without a restart. To put the set back, set it to exactly the value you read
first.
⚠ committed-applied says the node stored the set and took it into the running set.
It does not say the host agent's firewall admitted the port. Read that from a machine
other than the node's host with scripts/owner-port-reach.sh and both controls, as
From another machine describes, here with a
port the node admits and a port outside its set:
scripts/owner-port-reach.sh --address 192.0.2.10 --port 8099 \
--open-control 80 --closed-control 8100 --expect open
Read it with --expect closed before the change, with --expect open after it, and
with --expect closed again once you have put the set back. On a node whose director
is attached, a port in the set completes a connection even while nothing answers on
it, because the director carries every connection the firewall admits to the node's
one listener; the reading then shows the firewall admitting the port and nothing more.
On a node without the director, a port with nothing listening on it is refused
instead, and the probe reads it as refused, not open.
⚠ The gap. The node also takes one port at a time through /edge/served/ctl, which
declares or undeclares a single port and reports whether the host agent admits the
set, but the valis command line has no verb for that door. A 9P client composes the
request itself, and from the command line you change the whole set through config.
4.2. Mail on port 25
Whether the node takes mail from other mail servers on :25 is one owner setting, the
mail switch, stored as mail.enabled. 25 never appears in edge.served-ports: the node
refuses it there, and admits 25 only while its mail leg is answering on it. Turn the
switch with valis mail on and valis mail off, and read it with valis mail status,
from your own machine, naming the node as for every owner verb:
valis --node NODE mail status
valis --node NODE mail on
valis --node NODE mail off
Replies to on and off. Each prints one line:
| Reply | Meaning |
|---|---|
ok on |
the switch is stored on and the node is serving :25 |
ok off |
the switch is stored off and the node stopped accepting on :25; no session was open |
ok off draining N |
as ok off, with N sessions open at that moment, which are drained as described below |
error refused: no-operator-state: ... |
the node has no operator-state database, so the wire cannot open; nothing was stored |
error refused: recovery: ... |
the node runs in recovery, which never opens the mail wire; nothing was stored |
error not-serving: REASON: DETAIL |
the switch is stored on and the wire stayed closed; no-arrival means the node has no socket for :25 and no way to bind one |
error failed: DETAIL |
the change could not be made or read back |
The exit status is 0 for an ok reply; 2 for an error reply, printed on standard
error, a malformed command line, or a node that serves no /mail/ctl; 3 when the node
could not be reached; and 1 for a fault on your own machine.
What off does to open sessions. New connections stop at once. A session in the middle of a transaction finishes its message first, so a message whose data was being sent is not lost. A session waiting for its next command has that command answered, then gets 421, which tells the sender to try again later, and is closed. A session still open after three minutes is cut. Mail already queued is still delivered locally.
Reading the status. valis mail status prints one key: value line each:
| Key | Value |
|---|---|
queue-depth |
the mail still queued: incoming, active, deferred, and corrupt added together. Delivered and bounced mail is not counted |
incoming, active, corrupt |
the entries in each of those queue tiers |
deferred |
the entries waiting to be tried again, and nothing delivered or bounced |
drain |
always idle: the delivery loop does all draining, so read delivery instead |
delivered |
the messages the node has delivered into a Maildir |
bounced |
the messages that failed permanently |
delivery |
running while the standing delivery loop runs, else stopped REASON, with REASON not-started, stopped, no-mail-store, no-operator-state, or loop-still-running |
outbound |
open when relay mail may leave, held (no adapter) while no outbound client is registered, or held (switch off) while the switch is off |
switch |
on or off: the effective switch |
switch-origin |
owner when mail.enabled is stored, default when it is unset and the switch follows the local domains |
wire |
serving PORT, draining N while the leg the switch last stopped still holds N sessions, or not-serving REASON, with REASON switch-off, recovery, no-operator-state, no-arrival, failed, or not-started |
starttls |
offered NAME, or not-offered REASON, with REASON no-mail-name, no-certificate, or not-built when the certificate set has not been built yet |
host-name |
the canonical mail name, or none |
local-domains |
the local domains, or none |
accept-localparts |
the local parts accepted at RCPT, postmaster and abuse first |
sessions |
live sessions and the ceiling, as LIVE/CEILING |
accepted, refused-at-ceiling, starttls-upgraded, starttls-failed |
counts for the running leg, or for the leg last stopped |
Read the effective switch here, not from valis config get mail.enabled. The stored
value is empty while the owner has not set it, and the switch then follows the local
domains: on when mail.local-domains names a domain, off when it names none. A stored
on with wire: not-serving means the node wants mail and cannot serve it, and the
reason says why.
Delivery. The standing delivery loop delivers mail into your Maildir on every node
that has /mail and the operator-state database, whatever the switch says, so mail
already queued is delivered with the switch off. A landing wakes it at once, and it
also polls every 15 s. On a node without the database no loop runs: local mail is
written straight into your Maildir, and only /mail/inbox is mounted. Relay mail waits
in incoming while outbound is held, and never takes a slot local mail needs.
delivery: stopped no-operator-state on a node you expect to have a database means the
node could not engage it at boot.
Waking the loop. Write drain to /mail/ctl to wake the delivery loop now rather than
at its next poll. The reply is ok woken while the loop runs, or
error not-running: REASON, with the REASON delivery shows. A drain runs no
delivery of its own.
What the journal says about delivery. Three events, each content-free:
| Event | Level | Names | Never names |
|---|---|---|---|
mail.delivered |
INFO | the file written, as entry inbox/DID/new/ENTRY |
the recipient, the sender, the subject, or any byte of the message |
mail.delivery-held |
WARN | the queue entry, or cycle for a whole cycle, as entry, and the condition's type, as condition_type |
the condition's text, the recipient, the subject, or any byte of the message |
mail.delivery-loop |
INFO | state started, stopped, or skipped, and on a skip the reason: no-mail-store, no-operator-state, or loop-still-running |
anything about a message |
A held entry is logged once, and again only when its condition's type changes, so a destination that stays down does not fill the journal. A failed local delivery is tried again on the next cycle.
Turning it with config. valis --node NODE config set mail.enabled true and
false do what mail on and mail off do, with the change outcomes
changing a setting lists. One difference: with no
operator-state database, config set mail.enabled true stores the value and answers
committed-rebind-failed, where mail on refuses and stores nothing.
config reset mail.enabled returns the switch to its default. No variable in
valis.env seeds mail.enabled, so no environment file can open a node's mail port.
Local domains and the accepted local parts. Both change the running node at once:
valis --node NODE config set mail.local-domains example.org,example.net
valis --node NODE config set mail.accept-localparts fade,brian
valis --node NODE config reset mail.accept-localparts # back to the built-in list
The node accepts mail at RCPT only for the local parts in mail.accept-localparts at
its local domains, matched without any -extension, and refuses every other recipient
there with a 5xx. postmaster and abuse are always accepted, whatever the list holds.
Unset, the built-in list applies: fade, brian, postmaster, hostmaster, abuse,
and root. Every accepted name is delivered to your mailbox. While mail.enabled is
unset, naming a local domain turns the switch on, and removing the last one turns it
off. A change that turns it on and cannot open the wire answers
committed-rebind-failed with the reason; the domains are stored and in use all the
same.
The canonical mail name and its certificate. mail.host-name is the name the node's
greeting carries and its PTR record should answer; unset, it is mail. followed by the
first local domain. The node offers STARTTLS on :25 once custody holds a certificate
for that name, so for a zone the node holds, order one the way you order any other:
valis --node NODE config set mail.host-name mail.example.org
valis --node NODE obtain --domain mail.example.org --contact hostmaster@example.org
valis --node NODE mail status # starttls: offered mail.example.org
A sending host that names another name the node holds a certificate for by SNI is presented that certificate. STARTTLS is TLS 1.3 only and opportunistic: with no usable certificate the node offers no STARTTLS, takes mail in plaintext, and logs that once. A renewal reaches the next session with no restart.
The session ceiling. mail.session-ceiling is how many sessions :25 holds at once,
16 when unset, a whole number from 1 to 32. A connection over it is answered 421 and
counted in refused-at-ceiling. The node reads it at each connection.
Before a release that carries the switch. A node running an earlier release has no
mail.enabled key and no valis mail verb, so it cannot be told to stay off ahead of
time. On its first boot under the new release the switch takes its default, and a node
whose store names a local domain opens :25 if it has operator state and a way to
receive on 25. Check it before you run valis apply:
valis --node NODE config get mail.local-domains
An empty value means the switch comes up off and there is nothing to do. When it names
a domain and the node must not take mail yet, note the value, remove it with
config reset mail.local-domains, and apply the release. Once the node is up, turn the
switch off first and then put the domains back:
valis --node NODE mail off
valis --node NODE config set mail.local-domains example.org
valis --node NODE mail status # switch: off, switch-origin: owner
On a node already running a release that carries the switch, read valis mail status
before you apply, and run valis mail off first if the node must not take mail.
Not built yet.
:465,:587, and:993are not served, and no mail leaves the node over SMTP, sooutboundreadsheld (no adapter)on every node and relay mail stays inincoming.- A deferred entry is tried again every cycle with no backoff, and nothing fails an entry by age.
valis mailhas nodrainsubcommand; writedrainto/mail/ctlover the owner path.- No TLSA record is published for the mail name, so a sending host is never told to insist on TLS.
5. Enable, start, and check status
systemctl enable --now valis.service # enable at boot + start now
systemctl status valis.service # unit + MainPID (fulcrum) state
journalctl -u valis.service -f # follow the bring-up + serving log
Readiness is meaningful, and it is reported in two places that are not the same
place. The bring-up writes its account to the journal, and the resident separately
publishes a one-line summary to the supervisor, which systemctl status shows as
Status:. Neither carries the other's text, so a phrase you cannot find in the journal
is very likely a status-field phrase, and looking harder will not turn it up.
In the journal (journalctl -u valis.service), the bring-up says what it did:
site configuration complete: every declared site fact was answered, and the count of facts checked follows on the same line. A boot missing one stops here instead, naming the variable it wanted.adopting the DNS descriptor pair: the DNS handoff took; valis is serving authoritative DNS over fulcrum's inherited:53descriptors, binding nothing privileged itself.the launch handed down the DNS descriptor pair ... and 53 is not configured: the pair arrived, but this node's served set leaves DNS out, so valis closes it once the fabric is up. DNS does not answer on this node; this is not an adoption.:443 public edge opening — certificate in custody for <domain>: a cert was found for the configured domain and the edge is coming up.no usable certificate in custody for <domain>; :443 public edge cert-gated dark: the domain is named but no cert has been issued for it yet. This is what an operator sees between naming a domain and completing the first obtain, and it is the line that says the gate is the certificate rather than the configuration.no edge.domain configured; :443 public edge stays dark: no served domain named, so:443is intentionally dark by configuration.
In the supervisor's status field (systemctl status valis.service, the Status:
line), the resident publishes its serving state and refreshes it on every health-loop
pass. It reads <serving state>; cert <days>; store gen <n>, where the serving state is
one of:
serving :53 + :443: the public HTTPS edge is open on a usable base certificate, whether it was in custody at boot or arrived after.serving :53; :443 dark (no cert): a healthy genesis boot::53is up (which is exactly what dns-01 needs to obtain the first certificate), and:443is cert-gated dark until a cert is issued. This is not an error and does not withhold readiness. On a node whose launcher config names an:edge-port,:443opens when a usable base certificate arrives in custody, with no restart.recovery: development listener open, protocol services held: the host agent started the node in recovery after its starts kept failing; see Recovery mode.
A silently dark port would look like a fault; between the two surfaces you get why
:443 is or is not open, so you never have to guess.
6. Descriptor headroom
Nothing in the resident gauges descriptor headroom: there is no count in the journal, no field in the status line, and the liveness gate does not read it. A node that has exhausted its descriptors keeps reporting itself as serving and the supervisor is given no reason to restart it, so headroom is a reading the operator takes from outside the process, on the host:
pid=$(pgrep -o -x valis) # the resident is fulcrum's child, not MainPID
ls /proc/$pid/fd | wc -l # descriptors held right now
grep 'Max open files' /proc/$pid/limits # the ceiling this process actually has
Match the process name to whatever the launch configuration's :valis-command stages. Take
the reading as root or as the run account: another account's /proc/<pid>/fd is not
yours to list. The unit sets no descriptor limit of its own, so the ceiling is the
service manager's default on this host: read it rather than assume it.
One reading tells you little. Take two, some hours apart under comparable load: a busy node's count moves in both directions, while a count that only ever climbs is a leak, and the moment to act on it is while the number is still nowhere near the ceiling. Put that pair of readings into whatever monitoring you run against the node. Nothing on the node will raise it for you, and the failure it precedes is one every other instrument reports as healthy. Why that is, and what the liveness gate does and does not ask, is set out in backup-and-observability.org.
7. The certificate lifecycle
The :443 edge is obtained and renewed over the ACME/dns-01 spine (custody and the
ACME lifecycle are mercer's; valis is the edge that consumes the cert):
- Genesis (no cert). Boot with
edge.domainset but no cert yet::53comes up,:443is cert-gated dark. Serving:53is the precondition for dns-01: the authoritative answer proves domain control to the CA. - Issuance. Once a certificate is issued and lands in the custody store
(
acme.store-path), the node opens:443at its next read of custody, with no restart, when its launcher config names an edge port (:edge-port); a boot with the cert already in hand opens it at start. dns-01 is the only challenge a node answers today, and there is no fallback. Nothing answers an http-01 challenge on:80or a tls-alpn-01 challenge on:443, so an order for either has nothing to validate against. That is why serving:53is a precondition of issuance rather than a convenience. - Renewal (hot-swap, no restart). On a successful renewal the live
:443cert is reloaded in place through the shared credential cell: the edge swaps the new leaf without dropping the listener or restarting the unit. A renewal orders under the same account identity the first order used, and it reads that identity from the node'sacme.account-contactsetting: set it (valis config set acme.account-contact ADDRESS), or renewal refuses every attempt and backs off while the certificate you are serving walks toward its expiry. The boot line reports whether a contact is configured, so check it there rather than waiting for the backoff to tell you. - Staging vs production. Default is the ACME staging CA. Set
acme.productiontotrueonly when you want a real, publicly-trusted certificate: production carries a strict rate ceiling, so exhaust staging first. - Wildcards, and the name a certificate is filed under. Custody files an issued
certificate under one name, and a wildcard identifier such as
*.example.comis a matching rule rather than a name: no filesystem will hold it as a directory. So an order is placed with a host name leading, whatever order you asked for it in, and the certificate covers exactly the names you named. Ask for nothing but wildcards and the order is refused before it reaches the CA, because the alternative is to buy a certificate that has no name to be filed under. Every issuance counts against a rate limit whether or not you get to keep it. - A certificate per name, chosen per connection. Ask the node for a name it holds a
certificate for and you are presented that certificate. Ask for anything else and you
meet the credential sourced for
edge.domain, which is the fallback rather than a failure: an unknown name gets the base certificate, and your client then rejects it on a hostname mismatch as it should. Two names count as one only when they differ in case or in a single trailing root dot, because DNS and custody both file them that way. ⚠ Do not read the boot lineholds a certificate for N name(s)as any one certificate's subject names: that is the admission list, the names the edge will answer for. Which certificate each of them is served is decided per connection, from custody.
A domain whose DNS lives elsewhere. A node is meant to prove control of every name it
serves. For a domain you bring while its DNS stays with a third party, the proof is
tls-alpn-01, answered on the node's live :443 edge inside the TLS stack that already
serves it, once you enroll that domain through the control plane. That is not built
yet. Today such a domain cannot obtain a certificate, and its site does not serve:
valis obtain refuses it as a name the node does not serve, before any order reaches
the CA. The same holds for a node whose served set leaves out :53, since a node
answers dns-01 only from its own :53. To serve a domain today, delegate its DNS to
this node and hold its zone here.
Driving the first obtain by hand. The obtain verb runs the order on the process
that serves :53, which is what makes the dns-01 challenge answerable at all. Run it
from your own machine and name the node:
valis --node NODE config get acme.production
valis --node NODE obtain --domain example.org --contact hostmaster@example.org
Its whole form is:
valis obtain --domain <fqdn> --contact <email> [--profile <name>] [--directory-url <url>]
⚠ Read acme.production first. The first obtain does not refuse a node whose
acme.production is false: it orders from the staging authority, or from
acme.directory-url when that is set, and serves the certificate it gets, which no
browser trusts, on the node's :443 edge. On a node meant to serve the public, set it
before the first obtain with valis --node NODE config set acme.production true. A
renewal on a node with edge.domain set is refused while acme.production is
false, but the first obtain is not, so that read is yours to make.
The verb answers once the order is done. Its reply begins obtained, then a
domain NAME line for each name and a next-renewal line, then a line saying where
the new certificate is served: edge serving now when the :443 edge took it at once,
including on a node that started dark with an edge port and opened :443 for the base
certificate just obtained, or edge not open: serving from the next start on a node
whose edge cannot open yet: no edge port was handed down, or custody holds no usable
certificate for edge.domain.
A line for each name follows, saying whether a client asking for that name is handed
the certificate. With a key the node has not enrolled, the verb is refused with exit 2
and no order is placed.
To tell the new certificate from the one it replaced, compare the leaf's serial and its
notBefore. Its issuer may differ, since the authority can sign the new leaf from
another of its intermediates, and a production re-order did. The order also places its
dns-01 challenge records in the node's zone and takes them out again, and each change
moves the zone's SOA serial forward, so expect a higher serial after an obtain.
Both --domain and --contact are required; --domain repeats, or takes a comma-set.
⚠ Take a comma-set only for names that genuinely belong on one credential. Certificate
Transparency publishes the name set permanently, so packing several of your domains
into one certificate publishes that you hold them all, and nothing takes it back. One
certificate per registrable domain.
Like the other owner verbs it takes a pinned node through --node or VALIS_NODE,
and it stays on staging unless the node's acme.production setting is true. When
you cannot use the route from your machine (the owner port closed, the pin or the device
key lost, or the node not answering on the owner port), run it on the node's host as the
break-glass route, inside the node's namespace with --keyfile and --endpoint (see
the break-glass route):
sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
/opt/valis/bin/valis obtain --domain example.org --contact hostmaster@example.org \
--keyfile /var/lib/valis/valis/keyfile \
--endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"
A first-obtain can also be driven remotely by the authenticated owner over the
/acme/ctl 9P door: obtain <domain> <contact> starts the dns-01 order on a
background thread (single-flight: a second obtain while one runs is refused,
since every attempt burns CA rate-limit budget), and status reports
idle/running plus the last completed outcome. The door threads no CA selection
of its own; the staging-unless-opted-in default above applies unchanged.
Reading back what a node holds. I added valis cert list so you can see which
certificates a node holds, and whether its edge serves each one, without a shell on its
host. Name the node as you do for every owner verb:
valis --node NODE cert list
The verb changes nothing on the node. It prints one line per certificate the node holds, sorted by the name each is filed under, with seven fields separated by tabs:
- the name the certificate is filed under;
- its not-after instant;
- its next renewal;
- whether the node's
:443edge serves it:servedwhen the edge hands it out for every name it carries,partialfor only some of them,not-servedfor none,no-edgewhen the node has no edge listening, andunconfirmedwhen the node could not read what its edge serves; - the DNS names it carries, joined by commas;
- what the
:443edge presents for that name:<fingerprint>:<not-after>for the certificate it hands out,nonewhen it hands out none,no-edgewhen the node has no edge listening, orunknownwhen the node could not read it; - what
:25presents for that name after STARTTLS:<fingerprint>:<not-after>, ornot-offeredwhen the mail leg is not running or offers no STARTTLS, orunknown.
Both instants are whole numbers in the same encoding as the next-renewal line of an
obtain reply, or unknown when the node cannot read them from the certificate. A
fingerprint is the SHA-256 of the certificate in lowercase hex with no separators, as in
the cert reload reply; the cert.presented-changed and
cert.custody-held-inconsistent journal events write it with a colon between each pair
of digits. Fields 6 and 7 tell you
what the node is actually handing out, which can differ from custody for a moment after a
change and for longer when the custody watch is off. A node that predates them prints
five fields.
When the node cannot read its certificate custody at all, the list is the single line
custody-unreadable. An empty list therefore means custody holds no certificate, not
that it could not be read. No line carries key material. Only the node's owner can read the list: a key the node has not enrolled meets
an empty view and is refused before the list is read, as with every owner verb. On the
node's host it runs by the break-glass route, as obtain does above.
| Exit | Meaning |
|---|---|
| 0 | the node answered; its list is on standard output |
| 1 | a fault on this machine, not the node |
| 2 | refused, with the list not read: before dialling, for a malformed command line, no pin, or a key file that is absent, unsafe, or malformed; during the handshake and before your key is sent, for a node that does not match its pin; after the attach, for a key the node gave no owner view |
| 3 | the node could not be reached, its name did not resolve, or the session with it failed |
ubik's certificate panel reads the same list, and orders a certificate through the same
door valis obtain writes. On a node that predates cert list, the panel says that the
node does not list certificates and still offers the order.
The panel reads the first five fields and does not yet show the custody-unreadable
line, so a panel showing no certificates may mean custody could not be read; valis cert
list tells you which.
Making a node read its custody now. A node notices a change in its certificate
custody by itself, but I added valis cert reload for when you want it to happen now,
and to see what changed:
valis --node NODE cert reload
Run it after replacing a certificate in custody by hand, or when the node's STATUS line
says custody watch off. The watch is also what withdraws a certificate at its
not-after, so while it is off an expired certificate stays presented until you run this
or restart the node. The node reads custody in full and brings both legs to what a
start would present. On a node whose :443 is dark and whose launcher config names an
edge port, that includes opening :443 when custody holds a usable base certificate;
each name it then serves is reported added. Its reply is printed as the node sent it:
reloaded changed 443 www.example.org <old fingerprint> <new fingerprint> replaced changed 25 mail.example.org <old fingerprint> none removed
The first line is reloaded. Each changed line names the leg (443 or 25), the
name, the fingerprint presented before and after (none for no certificate), and why:
replaced, removed, expired, added, or held-inconsistent for a certificate and
key caught mid-write, where the certificate already in force stays. A line naming none
with additions-waited says new names waited for the node's operator state. When
nothing differed, the second line is unchanged. When custody could not be read, the
reply is the single line custody-unreadable, and neither leg changes beyond
withdrawing any certificate past its not-after.
| Exit | Meaning |
|---|---|
| 0 | the node answered, custody-unreadable included; its reply is on standard output |
| 1 | a fault on this machine, not the node |
| 2 | refused, with nothing read, for the same reasons as cert list |
| 3 | the node could not be reached, its name did not resolve, or the session with it failed, a re-read that failed on the node included |
8. Reaching a node's owner control plane
The owner-keyed verbs (zone, apart from load, secondary --offline and
delegation; obtain; publish; publisher; restart; key; and config against a
running node) reach the node's owner control plane. You run them from your own
machine and name the node:
valis --node NODE key list
valis --node NODE config get edge.served-ports
valis --node NODE zone export --origin example.org
That is the normal route, and it needs three things in place: the node pinned on your machine, a device key on your machine that the node has enrolled, and the node's owner port open. From another machine below shows how to set up all three. The client dials the address you pinned and refuses a responder that cannot prove the pinned DID before your key or any request reaches it. The node's answers print as they do on the host.
A key the node has not enrolled. The node still admits the session, but only to an
empty view, in which nothing manages the node. Every owner verb reads that view before
it sends a request, refuses with exit 2, and says that the node gave the key no owner
view: the key is not enrolled on that node, or was retired, and valis key list on
the node's host lists the keys it has enrolled. Nothing on the node changes.
The host route stays for when you cannot use the route from your machine: a shell on the node's host, and the verb run inside the node's network namespace with the node's own key. The break-glass route gives it and says when to reach for it.
8.1. From another machine: a pinned node
From a machine of your own you reach the node by name, with a device key the node has enrolled, and the client checks that the node answering is the one you pinned. The route, once:
- On the client,
valis key generatewrites this machine's device key and prints its DID (see On the client: a device key of its own). - On the node's host,
valis key enrol DID --label LABELadmits it (see From the host). - On the client, pin the node:
valis node pin NAME HOST:PORT DID, with the node's own Ed25519 DID, thedid:key:z6Mk...thatvalis key listprints first on the host. A node is pinned by that form alone:valis node pinrefuses any other DID, an X25519did:key:z6LS...included, with exit 2 and the store unchanged. A store that holds any other DID, or is written in any other form, is refused whole and never read as empty; move it aside and pin the node again.
Then every owner verb takes the name:
valis --node lab key list
valis zone export --origin example.org --node lab
VALIS_NODE=lab valis restart
--node may stand anywhere before a -- word, and --key FILE presents a device key
other than the default. Each is given once and needs a verb, as every owner flag does.
When the device key the verb would present is missing, the refusal gives the command
that writes it: valis key generate, with --out FILE when --key or VALIS_KEY named
a file other than the default. Nothing falls back to another key. --node and
VALIS_NODE may not be combined with --keyfile or
--endpoint: the verb refuses before reading a key or dialling, rather than letting one
replace the other, and --key or VALIS_KEY with no node named is refused for
the same reason. Under a node, VALIS_KEYFILE and VALIS_MGMT_ENDPOINT are not read. A
node that is not the one pinned is refused before your key or any request reaches it,
and the refusal names the command that pins it again. The node's answers print as they
do on the host.
The key you present under a node is never the node's own. A --key or VALIS_KEY whose
DID is the pinned node's DID, the node's keyfile named on its host for instance, is
refused with exit 2 before anything is dialled, and the refusal names the key file. I
made the client refuse that key so the node's seed never goes
out under a pinned name: under a node the verb presents an enrolled device key, and the
node's own key is presented on its host with --keyfile and no node.
Opening the owner port, after the pin. The node's owner port, 564 unless the host
agent's configuration names another, ships closed: the unit's firewall, which the host
agent installs, drops it, and the host agent binds no socket for it. Open it only once the owner's machine has pinned the node,
so the first attach from off the host already checks who answers. In this order, with
NODE the name you give the node and ADDRESS its public address:
On the node's host, read the node's DID inside its namespace, as the break-glass route shows:
sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \ /opt/valis/bin/valis key list \ --keyfile /var/lib/valis/valis/keyfile \ --endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"The first line names the node's own key, and its
did:key:z6Mk...is the DID to pin.- On the owner's machine, pin it:
valis node pin NODE ADDRESS:564 DID. Pin the address itself. The client dials an IPv4 or IPv6 address as given, with no reverse lookup first, so an address with no reverse DNS record connects like any other; a host name in a pin is resolved forward to its first address. Only then, on the node's host, open the port with the host agent's
owner-portverb, and restart the unit:sudo /opt/valis/bin/fulcrum-resident owner-port open sudo systemctl restart valis.service- On the owner's machine, attach:
valis --node NODE key list.
The owner-port verb sets :open-owner-port in the host agent's configuration store
and no other setting, though it rewrites the whole file, so hand comments in it are
lost. It restarts nothing and checks no pin, so the order above is yours to keep, and
the change takes effect at the next restart of valis.service. To close the
port again, run sudo /opt/valis/bin/fulcrum-resident owner-port close and restart the
unit; the pin stays on the owner's machine for the next open, and enrolled keys stay
enrolled across the restarts. owner-port status changes nothing. Each of the three
prints the store it read, the configured port, a configured: line with the store's
setting, and a live: line saying whether the unit's firewall carries the owner accept
line, so between a change and the restart the two lines disagree. open and close
exit 0 when the change was made or the store already held it, 1 when the verb refused
or failed, and 2 for an unrecognized argument.
An owner port other than 564. The resident picks the owner terminus's descriptor by
its port: 564, unless the launch names another with --owner-port N. N is a whole
number from 1 to 65535, written in digits alone, given once, and not a port another
service takes (25, 53, or 443); given any other value, the resident refuses the launch
and does not start. At boot it writes one line to the journal for each descriptor it was handed that
no service it runs takes:
valis: descriptor 9 (stream, port 2564) was handed down and answers for no service this node runs, so it serves nothing
A lone half of the DNS pair gets a line of its own, ending was handed down without the
other half of the DNS pair, so it serves nothing. The host agent passes --owner-port
beside the owner descriptor since fulcrum #60, so an owner port other than 564
configured on the host agent reaches the resident. A host agent older than fulcrum #60
passes no --owner-port, and the boot
then reports that port's descriptor with the line above. An owner port other than 564
has not yet been exercised on a node.
Checking the port from another machine. scripts/owner-port-reach.sh reads whether
one connection to an address and port completes or gets no answer. A firewall that drops
a port reads as no answer, so a reading means something only beside two controls on the
same address: a port the node admits and a port it drops. The probe reads both first and
gives no verdict when either reads otherwise:
scripts/owner-port-reach.sh --address ADDRESS --port 564 \
--open-control ADMITTED_PORT --closed-control DROPPED_PORT --expect open
It exits 0 when the port read as expected, 1 when it read otherwise, 2 for no verdict,
and 3 for a usage error, which includes --expect without both controls. Run
--self-test first to see the probe separate the cases on loopback.
To see which view the node gives a key, build owner-view-proof with
scripts/build-owner-view-proof.sh, which writes bin/owner-view-proof, and attach
with it: owner-view-proof --node NODE --key FILE --expect owner-frame. An enrolled
key gets owner-frame, a key the node has not enrolled gets empty, and
owner-view-proof --anonymous --endpoint ADDRESS:564 gets published-floor. It exits 0
when the view matched --expect, 1 when the node gave another, 2 when
owner-view-proof refused an argument or refused the attach before sending anything,
and 3 when the node could not be reached. It prints only a key's
public DID, and its --self-test shows it telling the three views apart on a scratch
fabric. With --node, --boot also prints boot: HEX, the node's boot identity, which
the node draws afresh at every start, so two equal readings bracket a span with no
restart in it. --pub SLUG prints pub: SLUG bytes N md5 HEX for the publication as
read through the attached view, or pub: SLUG absent. The length and digest are of the
bytes as published, so a file that ends in a newline counts that newline.
Owner-port connection caps. The owner port holds at most 4 connections at once from
one origin and 16 from all origins together, counting each from the moment it reaches
the port, handshake included. Change them with
valis config set owner-port.sessions-per-origin N and
valis config set owner-port.session-ceiling N, each a whole number from 1 to 63; the
node reads the new cap at the next connection. One origin is one IPv4 address or one
IPv6 /64, so machines behind one public IPv4 address share one origin and its 4
connections. A connection past either cap is closed before anything is read from it,
and the node logs the refusal at WARN at most once per origin per minute, in a line
that begins origin cap: refused a connection from. A session the node grants
nothing, with no enrolled key and no accepted capability, has 10 seconds before the
node closes it.
Where this stands. This route is proven from another machine against a deployed
node. The client refused an attach with no pin before dialling anything, the pin was
written before the port opened, and the pinned attach listed the keyring. The client
refused a pin naming another DID, and a key the node never enrolled got an empty view.
The port read closed, open, closed, and open again from off the host as owner-port
and a restart moved it, each reading beside both controls. The node's director carries
every connection to the owner port to the owner terminus: on that node a silent
connection to the port is cut at 10 seconds, and a fifth connection from an origin
already holding 4 is refused at accept. Only the terminus's claim on the port, where
the director carries those connections, arms that bound and that cap.
Managing a node over this route is proven on a deployed node that serves the public,
in one run from the owner's machine with no shell on the node's host. A zone record
was added and deleted, and was answered and then gone over UDP and TCP. A page was
published and revised, and each text was served at once. With one session from the
owner's machine already open, owner-port.sessions-per-origin was set to 2, and of two
more connections from that machine one was held and the other refused at accept; the
setting was then reset. Port 8099 was added to the served set, read open from off the host
beside both controls, and read closed again once the set was put back. The node's
boot identity read the same before and after those changes, so none of them restarted
it. A certificate for the names the node's certificate already covered was ordered
from the production authority, and the :443 edge served the new one at once, with no
restart. Last, the node was restarted and answered from a new boot identity, with the
device key still enrolled and both settings the run had changed reading as they did
before the run. Each change was read back after it took effect. A key the node never
enrolled got an empty view and was refused by every one of those verbs with exit 2,
and nothing on the node changed. A peer with no key got only the published floor, with
no door to the names, the configuration, or the node.
That run showed two gaps. The restart ended the owner's ubik session, and the ubik of that run had to be reconnected by hand; ubik now attaches again on its own, as Managing a node from ubik describes. The page stays published, since no verb retracts a publication.
⚠ The gaps. Opening or closing the port takes a restart of the unit. The first proof
on a node that serves the public used the valis command line and owner-view-proof;
ubik has since managed that node too, and
Managing a node from ubik names what ubik has not shown
there yet. Enough origins together can still fill the 16 and keep you out
of the port for as long as they keep reconnecting, and the caps are defaults I chose,
not measurements; your way in then is the host, by
the break-glass route.
One store, two clients. The pin store is $XDG_CONFIG_HOME/ubik/node-pins.sexp (or
~/.config/ubik/node-pins.sexp), shared with ubik, which reads and writes the same file.
Before you pin with valis on a machine where ubik also runs, make sure that ubik writes
the shared format, or one client may refuse the other's store. Two clients writing the
store at the same moment never leave it torn. Both clients change the store under its
lock, the file node-pins.sexp.lock beside it: valis node pin and valis node unpin
take it, and so does ubik when it records a pin, so neither client loses a change the
other makes. valis waits up to ten seconds for that lock. If another writer keeps it
longer, the verb exits 3, names the store, and changes nothing; run it again once the
other writer finishes.
8.1.1. Managing a node from ubik
ubik reaches a node the way the command line does: by a pin name in the same store, with a device key the node has enrolled. I made ubik present a key of its own, enrolled on each node under its own label, so you can retire the key ubik presents without touching the one your command line presents. Create it on your machine and enrol its DID on each node's host:
valis key generate --out ~/.config/ubik/owner.key --label ubik-laptop
ubik refuses to present the node's own key, by pin name or by any other route, and refuses a key file that grants anything to its group or to others; both refusals come before anything is dialled. A key the node has not enrolled meets an empty view, and ubik closes the connection.
Once ubik is attached, you manage the node from its panels: its configuration, the served set, a restart, its certificates, publication, and the keyring. The panels write the node's ordinary ctl doors, the ones the command line writes, and the served-ports panel writes the per-port door the command line has no verb for. Each panel reads the node again after every act, so what you see is what the node now holds. A disruptive act takes a second click on a confirmation that names its effect: a restart, taking a port out of the served set, retiring a key, or an order from the production authority. ubik never offers to take out the port it is attached through.
When the connection goes, whether ubik sent the restart or the node dropped off the
network, ubik attaches again on its own: at once, then after 1, 2, 4, 8, and 16 seconds,
and every 30 seconds after that. Every attempt checks the pin again. An attempt that
meets a node no longer matching its pin stops there, and ubik shows you the refusal
instead of trying again. A node that goes silent without closing the connection is
reported lost within 25 seconds of its last answer. Once ubik is attached again, every
panel reads the node again. ubik's own manual, docs/OWNER-MANAGEMENT.org in the ubik
repository, covers the panels in full.
Where this stands. On a test node, ubik attached with its own enrolled key and, when given the node's own key, refused it before dialling. Each panel changed the node, and each change read back through the valis command line or from off the host. ubik attached again after a restart it sent, after the node reset the connection, and after a silent drop, and it stopped when the pin was changed under it. A key the node never enrolled changed nothing.
Most of the same checks then ran on the deployed node that serves the public, the one
the command line was proven on, through ubik-manage, ubik's scripted runner, from the
owner's machine with no shell on the node's host. ubik attached with its own enrolled
key and read the same keyring listing as the command line, line for line. Given a
stand-in key pinned as the node's own (the node's own key never leaves its host), ubik
refused it on every route and dialled nothing.
With its own session open, ubik set owner-port.sessions-per-origin to 2. Of two more
connections from the owner's machine, the node held one open and refused the other at
accept. ubik then reset the setting to its default, as it was before. It added port 8099
to the served set; from off the host the port then read open, and ubik took it out
again. It revised a page already published and then restored the page's earlier text,
each text read over HTTP. It retired a key enrolled for the checks, which the command line
then read as retired. Its certificate list matched the command line's line for line, and
its sessions view matched the connections counted on the owner's machine.
ubik refused each disruptive act sent without its confirmation. Even with the confirmation given, it refused to take out the port it was attached through, a request its panels never offer. A restart sent through ubik brought the node back on a new boot identity. ubik attached again on its own about 3 seconds later, and every panel read the node again. A key the node never enrolled got an empty view, every act sent with it was refused, and nothing on the node changed. Every setting, port, and page the checks changed was put back afterwards. Every panel was then opened against that node and showed what the node holds, with the rough edges listed below.
Not yet shown on that node:
- A ubik session coming back on its own after something else, such as a deploy, restarts the node. The restart in those checks was sent through ubik, so it does not show a restart from elsewhere. The one other ubik session open at the time came from an older build of ubik, without automatic reattachment, and it stayed down.
- A lost connection without a restart, whether a reset or a silent drop. Causing one on that node needs a shell on its host, so ubik attaching again after one is shown on a test node only.
- A certificate ordered through ubik. On that node ubik listed the certificates and refused an order without its confirmation, and no confirmed order was sent; certificates ordered through ubik have been issued on a test node only.
ubik's interface has these rough edges today:
- The publication panel takes the page's name in ubik's input line at the foot of its window. The input line shows no prompt, and no cursor until you type. The editor above it, where you compose the page, shows a blank white field with no cursor.
- A divider under the publication panel's instructions looks like the box they mention, but is not one.
- Scrolling a pane redraws the whole pane with a visible flash.
- The configuration panel works, but its buttons are oversized and its layout is hard to read.
8.2. From the node's host: the break-glass route
Reach for the host when you cannot use the route from your machine:
- the owner port is closed, or the node does not answer on it;
- the pin, or the device key, is lost from your machine;
- the change is one only the node's own key may make, such as enrolling a device key (see Enrolling and retiring an owner device key).
On the host the verbs reach the resident over its loopback fabric. On a deployed node
that loopback belongs to the resident's network namespace, not to the host. The
fabric binds a fresh port at every boot, and the resident writes the address it bound
to mgmt-endpoint in its data root, which is /var/lib/valis/valis/mgmt-endpoint
under the shipped unit.
Run the verb on the node's host, inside that namespace, which you enter by the resident's pid:
sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
/opt/valis/bin/valis zone export --origin example.org \
--keyfile /var/lib/valis/valis/keyfile \
--endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"
- Read
mgmt-endpointevery time. The port is drawn afresh at each boot, so an address you saved is wrong after the next restart. - Enter by pid.
nsenter -nenters the network namespace and nothing else, so host paths resolve as usual, and the pid is always there to name it by. - Run the verb as the account that owns the keyfile. Entering the namespace needs
root, so
nsenterruns undersudo, and-Sand-Gthen drop the verb to the run account before it starts. Give them numbers, asid -uandid -gprint them:nsenterdoes not accept account names there. - The keyfile must be the run account's own. The verb presents the file named by
--keyfileorVALIS_KEYFILE, or the one in its data root, only when it is a regular file, not a symbolic link, owned by the account running the verb, and granting nothing to its group or to others. Anything else is refused before the key is read, with a message naming the file and what was wrong; for a file another account owns, it names both account numbers. Run as root, the verb refuses the run account's keyfile for exactly that reason. A keyfile that is missing, or that is not exactly the 32 octets of a seed, is refused the same way, and never reported as a node that could not be reached. Every owner verb (zone,obtain,publish,publisher,restart,key, andconfig) exits 2 on this refusal, as on any refusal made before anything is sent. - The client checks who answered. The keyfile names the node's own key, so the verb
presents that key to the node that holds it and to no other. A responder that cannot prove it
holds that key is refused before anything else is sent, so a process squatting on the
endpoint is refused rather than answered, and it never sees the attach. Every owner
verb exits 2 on that refusal too, and a
restartwaiting for the node to come back stops at the first such refusal. - Arguments the verb refuses also exit 2. Each owner flag (
--keyfile,--endpoint,--node, and--key) may stand anywhere before a--word, is given once, and needs a verb. A flag given twice, a command line that has an owner flag and no verb, or an owner flag beside--resident,--daemon,--dev,--backup, or--restoreis refused with exit 2 before any key is read or anything is opened, dialled, or started. - Pass both
--keyfileand--endpoint. The verb keeps root's environment, so its default data root holds neither the node's keyfile nor its endpoint, and the defaults find nothing. - An ssh forward reaches nothing. A forward lands on the host's loopback, which is a
different
127.0.0.1from the one the fabric is bound to, and nothing listens there for it. Off the host, use the route from your own machine, in Reaching a node's owner control plane.
9. Enrolling and retiring an owner device key
A node has one owner identity, its did:key, and one seed behind it, which never
leaves the host. I wanted the owner's other devices to manage a node without a copy of
that seed, so the node's factotum holds a keyring of device keys beside the seed. Each
is a device's own Ed25519 did:key, enrolled under the node's fixed owner identity.
Enrolling or retiring a device key never changes the node's identity and never moves,
rewrites, or copies the seed.
What an enrolled key reaches. A session presenting a live enrolled key is admitted to the owner frame: the node's services through their loaded modules, minting grants, signing with the node's own key, and backups. It is the frame an owner client such as ubik reaches over the node's capability system. The session keeps the device's own DID as its principal, so the session listing names the device rather than the owner.
The node's REPL is not in that frame. The owner's total reach is host authority: ssh to
the host, and from there the node's REPL. No device key reaches it, and the
ssh-reality proof, which mints a grant over the whole namespace, admits only the
pinned operator SSH key. Enrol only your own devices. Give another person a grant, or an instance of
their own, never a place on your keyring.
What a key carries. Each key records its DID, a label you choose, the key that enrolled it, and when. Retiring it records who retired it and when, and ends it. It has no expiry, so it stays admitted until you retire it. Read the listing from time to time and retire the key of any device you no longer hold.
Who may change the keyring. Only a session holding the node's own key enrols, so enrolment happens on the host. The node's own key or any live enrolled key may retire a key, so a lost device can be cut off from another device you still hold. The node's own key can never be retired, and a retired key cannot be enrolled again while the keyring records its retirement: a device whose key was retired mints a new one and is enrolled under that. Setting a refused keyring aside discards those records (see Healing a keyring the node refused).
9.1. On the client: a device key of its own
A device you manage the node from holds a key of its own, never a copy of the node's seed. I made the client mint that key itself, so the secret is created on the machine that uses it and never crosses a network or a command line. On the client, run:
valis key generate # writes ~/.config/valis/device.key
valis key generate --out FILE --label laptop
The key goes to $XDG_CONFIG_HOME/valis/device.key, or ~/.config/valis/device.key
when XDG_CONFIG_HOME is unset, empty, or not an absolute path, unless --out names
another file. A relative XDG_CONFIG_HOME is ignored, as the XDG Base Directory
specification asks, so the default path never depends on the directory you run the
verb from. valis key generate does not read VALIS_KEY: to write the key a later
session will look for at another path, name it with --out. The file holds the
32 octets of an Ed25519 seed, the same format as the node's own keyfile, at mode 0600
whatever your umask. When the valis directory does not exist it is created at 0700; a
directory that already exists keeps its mode. The verb never replaces a file and never
writes through a symbolic link: with anything already at the path it exits 2 and writes
nothing. It contacts no node, so --keyfile, --endpoint, --node, and --key are
refused.
Standard output carries the device's did:key and nothing else, and standard error
carries the file's path and the line to run on the node's host:
valis key enrol did:key:z6Mk... --label laptop
The label is --label when you give one, and otherwise this machine's short host name
with any character a label cannot carry replaced by a hyphen. A label is 1 to 64 of the
characters A-Z, a-z, 0-9, dot, underscore, and hyphen, the rule the node's keyring
applies at enrolment. A --label outside that rule is refused with exit 2 before
anything is written, so the line printed is always one the node accepts. Run that line
on the node's host, as the next section describes. The seed is never printed.
The client then presents the key to a node you have pinned, with --node, as
From another machine describes.
| Exit | Meaning |
|---|---|
| 0 | the key was written; its DID is on standard output |
| 1 | the key could not be written |
| 2 | refused, nothing written: something is already at the path, the label is one the node's keyring would refuse, or a flag that reaches a node was given |
9.2. From the host: valis key
Run valis key the way every owner verb runs on the host, inside the resident's network
namespace as the run account, with --keyfile naming the node's keyfile and
--endpoint read from mgmt-endpoint (see
the break-glass route). The session then presents the node's own
key, which is the only session that enrols. The verb has three subcommands, and the
wrapper below supplies the rest of each command line:
valis key enrol <did> --label <label>enrols a device key;valis key listprints the keyring;valis key retire <did>retires a device key.
k() {
sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
/opt/valis/bin/valis key \
--keyfile /var/lib/valis/valis/keyfile \
--endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)" "$@"
}
k enrol did:key:z6Mk... --label laptop # the device's Ed25519 did:key
k list # the node's own key, then every device key
k retire did:key:z6Mk... # either DID form of the device key
A label is 1 to 64 of the characters A-Z, a-z, 0-9, dot, underscore, and hyphen.
The listing prints the node's own key first, then one line per device key with its
Ed25519 DID, its X25519 transport DID, its label, when and by whom it was enrolled, and
either state=live or, for a retired key, when and by whom it was retired. Retired
keys stay in the listing.
The verb exits with one of four statuses:
| Exit | Meaning |
|---|---|
| 0 | the node enrolled or retired the key, or printed the listing; its reply is on standard output |
| 1 | a fault on this machine, not the node, such as no management endpoint found, or a malformed --endpoint; standard error names it. The request may have reached the node, so check with valis key list before you repeat an enrol or a retire |
| 2 | refused, and nothing changed: the node replied error <reason>, the arguments were refused before anything was sent, or the verb refused the node that answered or the key file it was given: absent, unsafe, not the 32 octets of a seed, or the node's own key presented under a node |
| 3 | the node could not be reached (including a connect that did not complete in time), its name did not resolve, or the session with it failed; standard error names it. The request may have reached the node, so check with valis key list before you repeat an enrol or a retire |
The reasons a node gives are not-seed-holder (an enrolment from anything but the
node's own key), not-owner, malformed, unsupported-proto, invalid-key,
invalid-label, seed-key (the node's own key named), already-enrolled,
already-retired, not-enrolled, and store (the keyring could not be written). With
a keyfile that is not the node's own, such as a device key, the verb exits 2 before
anything is sent: the keyfile route accepts only the node that holds the key it presents,
and the node answering does not hold that key.
After a successful enrolment the verb prints a notice on standard error, so the consequence is in front of you at the moment you grant it:
valis: key: <did> is now admitted to this node's owner management frame: its services and their modules, minting grants, signing, and backups. valis: key: it stays admitted, with no expiry, until you run valis key retire <did> valis: key: enrol only a device you hold.
9.3. Listing the owner sessions: valis session list
valis session list prints one line per owner session the node holds, so you can see
who is attached and under which key. Run it from any machine that reaches the node as
its owner, with --node like every owner verb, or from the host by the break-glass
route. Each line holds five fields separated by tabs:
- the session id;
- the principal: the X25519 transport DID of the key the session presented, which
valis key listprints beside each device key's Ed25519 DID; - when the node admitted the session, as a Unix time;
- when the session was last active, as a Unix time;
- the state:
live, orfenced@followed by the Unix time of the fence.
The verb opens a session of its own to ask, so its own session is always in the listing. It changes nothing on the node.
The last line states the development listener, its fields separated by tabs:
development-listener, then open with the socket, the route that opened it, and
when, or closed, or unknown when the node could not say. See
The development listener.
| Exit | Meaning |
|---|---|
| 0 | the node answered; the listing is on standard output |
| 1 | a fault on this machine, not the node |
| 2 | refused, with nothing read: a malformed command line, printed with the usage; a refusal before your key was sent; or a key the node gave no owner view |
| 3 | the node could not be reached (including a connect that did not complete in time), its name did not resolve, or the session with it failed |
9.4. From ubik: the keyring panel
ubik's keyring panel lists every key the node has enrolled, with its label and its state,
and under each key the sessions presenting it that are live now. It retires a key through
the same /id/ctl door valis key retire writes, after a second click on a
confirmation that names the effect. Retiring the key ubik itself presents takes a
confirmation of its own, saying that the session ends and that this ubik can no longer
attach to that node; ubik reports that retirement as applied, and the session as ended.
The node's own key is never offered for retirement.
For a new device, generate its key on that device, then paste its public did:key and a
label into the panel. The panel checks both and shows the command to run on the node's
host, valis key enrol DID --label LABEL, and sends nothing to the node: only a session
holding the node's own key can enrol a device key. The device's private key never leaves
the device.
The panel shows each key's sessions and does not fence them. Fencing a session from ubik waits for richer session control on the node.
9.5. From a REPL in the running image
The verb and the door call the same two functions, and a REPL attached to the running image can call them directly:
(valis/src/identity/keyring:enrol-owner-key
(valis/src/namespace/id:factotum-keyring) "did:key:z6Mk..." :label "laptop")
(valis/src/identity/keyring:retire-owner-key
(valis/src/namespace/id:factotum-keyring) "did:key:z6Mk...")
A refused change signals a condition naming its reason and writes nothing. Reach the running image through its development listener, as Reach the live image shows.
9.6. Over 9P: /id/ctl
The host verb writes one line to the node's /id/ctl over its sealed owner session, and
an owner client on the host writes the same line. Each open of /id/ctl is its own conversation: write one
command, then read its one-line reply on the same open.
key proto=noise key=<device ed25519 did:key> label=<label> delkey key=<did:key>
The reply is enrolled <did>, retired <did>, or error <reason> with the reasons
above, and a refused command changes nothing. A read before any command returns the
listing. Only the owner frame names /id/ctl: no grant, bearer name, or delegated
token reaches it.
9.7. Retiring a key cuts off its sessions
Retiring a key takes it from every session presenting it, at once: each such session's
view empties and every handle it holds is revoked, so a client in the middle of a read
gets an error rather than more data. A session that attaches with the key while the
retirement is in progress is cut off as it attaches. A device that retires its own
key is cut off the same way, before the reply reaches it; the retirement still stands,
and valis key list on the host shows it.
If the node cannot finish cutting off a session, the retirement still stands and the verb still reports it, and the journal names the step that failed. Restart the node to end every session at once.
9.8. Where the keyring lives, and when the node refuses it
The keyring is keyfile.keyring beside the keyfile in the data root,
/var/lib/valis/valis/keyfile.keyring under the shipped unit, at mode 0600 and owned by
the run account. Every change is one record appended to it and synced before the node
acts on it. A change the node cannot write leaves the file exactly as it was, and the
verb replies error store. If the node cannot even put the file back, the journal says
so and the next boot refuses the file; heal it as the next sections show. Do not edit
it by hand.
The node refuses to start on any of these keyrings:
- one it cannot read whole: a torn last record, a line that does not parse, a version it does not know, or a retirement of a key never enrolled;
- one reached through a symbolic link, one another account owns, or one granting anything to its group or to others;
- one written for another owner, such as a keyring copied from another node, or an old keyring left beside a seed minted afresh.
The node refuses to start in the same way on a keyfile it will not load: one reached through a symbolic link, one another account owns, one granting anything to its group or to others, or one that is not exactly the 32 octets of a seed. The keyfile is left as it was.
A refused boot exits 78, and the unit's RestartPreventExitStatus=78 keeps systemd from
restarting it into the same refusal. The journal names the file and the problem. A
refused node leaves no image and no listener to connect to, so heal it from the host as
the next section shows.
9.9. Healing a keyring the node refused
The standard heal is to recondense the node from a backup. Restore the backup into a fresh state directory as Backup and restore shows, then start the unit. The node comes back with the keyring that backup carried and everything else with it. A restore cannot know of a key retired after the backup was taken, so retire any such key again once the node is up.
On a node installed from the delivery archive alone, recondensing from a backup is the only heal today. The manual heal below needs SBCL, Quicklisp, and a valis checkout with its dependencies on the host, and the delivery archive carries neither a REPL entry nor the valis source.
The manual heal repairs the keyring alone and leaves the rest of the node as it stands. Run it on the host, as the run account, with the unit stopped (a refused node already is):
Start SBCL as the run account with the data root the unit uses, and load valis from a checkout the way the Makefile's
test-onerecipe does:sudo -u valis env XDG_DATA_HOME=/var/lib/valis sbcl(require :asdf) (load "<quicklisp>/setup.lisp") (push "<workspace>/" asdf:*central-registry*) (asdf:load-asd #p"<checkout>/valis.asd") (asdf:load-system :valis)Work in the SBCL debugger in your ssh terminal, which offers every restart the heal does. To attach your editor instead, open a development listener on a socket in a directory of the run account's own, since the stopped unit has no runtime root:
sudo -u valis mkdir -m 0750 /tmp/valis-heal(asdf:load-system :slynk) (valis/src/dev/slynk:start-development-socket-listener #p"/tmp/valis-heal/development.sock")Then, from your own machine, as a member of group
valison the host:ssh -N -L 4005:/tmp/valis-heal/development.sock HOST # then M-x sly-connect to localhost, port 4005Evaluate the heal:
(valis/src/keyring-heal:heal-owner-keyring :state-dir #p"/var/lib/valis/")It takes the node configuration's lock for the whole heal. It refuses while the node runs, and when no seed is at the keyfile path. It then reads the seed and the keyring exactly as the boot does. Either it reports that the boot's loader accepts the keyring, or it stops on the boot's own refusal with the restarts that apply to it.
Choose one restart:
keep-keyring, offered when the file was refused only for its mode or for the account owning it. For the mode, it sets0600on the same file, and it is offered only when the file grants no write permission to its group or to others: a keyring others could write may hold content the node never wrote, so recondense it from a backup or set it aside. For another account's file, it installs a copy the run account owns once the content parses whole under the node's owner, and sets the original aside.salvage-keyring, offered for a damaged file. It keeps the whole valid records before the damage, drops the rest, and retires every key a dropped line names.recondense-from-backup, offered for every refusal. Restore a backup into a scratch state directory first, then give the restart that directory; it takes only the keyring from there:sudo -u valis mkdir -m 0700 /tmp/keyring-restore sudo -u valis /opt/valis/bin/valis --restore --in /path/to/valis-backup.sealed \ --state-dir /tmp/keyring-restore --acme-store-dir /tmp/keyring-restore/acmeA key the damaged keyring enrolled and the backup's keyring lacks is not on the healed keyring. The heal report lists each such key; enrol it again. Recondensing applies the retirements the damaged keyring holds, even when the node refused that file for its mode or its owner. When the damaged keyring could not be read at all, the heal report says so and its retirements are not applied, so retire any such key again once the node is up.
set-keyring-aside, offered for every refusal. The node then starts with no enrolled keys, and you enrol your devices again.
After a restart acts, the heal checks the result with the boot's loader again and returns a report: every key live and retired, every key the heal retired and the line that named it, every line a salvage dropped, and every key the damaged file enrolled that a backup did not carry.
Quit the SBCL, start the unit and read the boot line, then remove the scratch restore, which holds a copy of the seed, and the listener's directory if you made one:
sudo systemctl start valis.service journalctl -u valis.service -n 20 sudo rm -rf /tmp/keyring-restore /tmp/valis-heal
What the manual heal guarantees:
- the damaged file is renamed aside as
keyfile.keyring.refused-<UTC time>and never removed; - the seed is never moved, rewritten, or created;
- only the node's own seed binds the result, so another owner's keyring, or a backup of another node, is refused;
- after
keep-keyring,salvage-keyring, orrecondense-from-backup, a key the damaged file retires, or that a damaged or torn line names, stays retired, so a device retired that way is enrolled again with a new key. A torn retirement that names no key retires every live key, because nothing on the line says which key it retired. This holds whenever the heal can read the damaged file; when it cannot, the heal report says so.
set-keyring-aside discards the retirement records. After a set-aside the node no longer knows
which keys were retired, so re-enrol only keys you hold. Enrolment still needs a session
proven with the node's own key.
9.10. The development listener
An enrolled device can open and close the node's development listener over the owner
path, as a session holding the node's own key can. Reaching the listener once it is
open takes an account on the node's host in group valis; see
The development listener.
9.11. Not built yet
- Opening or closing the owner port takes effect only at a restart of the unit; nothing changes it on a running node.
- The owner port's caps hold one origin to 4 connections and all origins together to 16, but enough origins together can still fill the 16 and keep the owner out of the port while they keep reconnecting. The figures are defaults, not measurements.
- ubik has managed a deployed node that serves the public and attached again after a restart it sent, but three things are not yet shown there: a ubik session coming back on its own after something else restarts the node, a lost connection without a restart, and a certificate ordered through ubik. The last two are shown on a test node. The publication panel, scrolling, and the configuration panel have the rough edges Managing a node from ubik lists.
- ubik shows each key's live sessions but does not fence one; fencing a session from ubik waits for richer session control on the node.
- A device key is enrolled only on the node's host, by design, since only a session
holding the node's own key can enrol one. ubik's keyring panel and
valis key generategive you the command to run there. - An enrolled key reaches the owner frame over the owner port, not over the
:443edge: the edge's owner proof accepts only a signature by the node's own key. - An enrolled key has no expiry. Until it does, the listing and
valis key retireare how a key you no longer hold is ended. - On a node installed from the delivery archive alone the manual heal is unavailable: the archive carries no REPL entry and no valis source.
- No verb retracts a publication; a page stays served until you revise it.
- The served set's per-port door,
/edge/served/ctl, has no command-line verb; ubik's served-ports panel composes its request one port at a time, and the command line changes the whole set throughconfig. - A transfer key for a zone's secondary is minted only on the node's host, never over a pinned node.
- The first obtain is not refused on a node whose
acme.productionisfalse, although a renewal there is, so read it before you order a certificate for a node that serves the public. valis mailturns and reads the mail switch over the owner path, and it has been driven against a node in the test suite, not yet against a deployed node.
10. The development listener
The development listener is a Slynk listener on a local socket, by default
/run/valis/valis/development.sock. Whatever connects to it evaluates code in the node
with the run account's authority. I kept it on every node because it is the route that
still works when the owner path is the thing that broke, and I made it closed on a
fresh node and loud whenever it is open, because an open socket hands that authority
to whoever reaches it.
The socket is owned by the run account and group valis, at mode 0660, in the runtime
root /run/valis/valis/, which the host agent makes valis:valis 0750 at each start. I
put it under /run rather than the data root so a backup never captures it, and a
reboot empties it. Membership of group valis on the node's host is the boundary of who
may reach the node's REPL: the run account, root, and every member of that group, and
nobody else. Read the members with getent group valis; the run account holds the
group as its primary group, so it is not listed there. Add a member with
sudo usermod -aG valis USER and remove one with sudo gpasswd -d USER valis. A change
reaches that account's next login, not a session already open.
The setting development.listener records whether the listener is open. Every open sets
it and every close resets it, so a node restarted with the listener open opens it again
at boot, before anything else binds. The one exception is the node's own shutdown,
which closes the socket and leaves the setting on. development.socket-path names
another socket path when you need one; it applies the next time the listener opens. A
resident opens no TCP listener; valis --dev on a development host still opens one on
loopback.
You can tell when it is open in four places:
- the journal logs
development-listener.openedat WARN with the route and the actor that opened it,development-listener.still-openevery hour while it stays open, anddevelopment-listener.closedwhen it closes; development statusprints the status text, and the last key ofrelease reportand the last line ofsession liststate it (see [BROKEN LINK: reading-the-release-report] and Listing the owner sessions);- the host file
/run/valis/valis/development.status, which the node rewrites as it changes, at mode0640, so a member of groupvalisreads it withcat; - a mail notice to the node's owner, sent when it opens. See Not built yet for where the notice does not reach you today.
10.1. Open, read, and close it
Three routes open the listener: the owner verb, the host route, and the host agent's recovery start (see Recovery mode). The first two close it.
Over the owner path, from your own machine, as for every owner verb:
valis --node NODE development start # prints: ok open PATH
valis --node NODE development status # prints the status text the node serves
valis --node NODE development stop # prints: ok closed
The node's own key and every enrolled device key may open and close it. The status
records the actor as owner for the node's own key and as the device's label for an
enrolled key.
From a shell on the node's host, without the owner path, as the run account:
sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis development start --host
sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis development status --host
sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis development stop --host
cat /run/valis/valis/development.status # as any member of group valis
With the node running, start and stop hand it a request through the runtime root and
print its reply, waiting up to 15 seconds for it. With no node running, they set or
reset development.listener in the node's configuration, so the listener opens, or stays
closed, when the node next starts. Name the data root as shown: without it, the verb looks
for the node's configuration in the run account's home directory, where there is none.
It then refuses with exit 2 and prints the data root it looked in and the command in this form,
rather than changing a store no node reads. A data root holds a node when it holds the
owner seed (keyfile) or a running node holds its configuration.
Name the binary by its full path too, since sudo searches only its own secure_path.
When the verb prints a command for this route, it prints it in this form, with the
node's own data root and binary, so you can paste it as given. The listener's mail
notice names the installed binary, /opt/valis/bin/valis, so a kept notice still
works after the next apply.
The runtime root is --runtime-root DIR when you give it, else the node's stored
runtime.root, which VALIS_RUNTIME_ROOT seeds only at the first boot that considers
it, else /run/valis/valis/.
Both routes exit 0 when done; 2 when refused, when the node replied with an error, or, on the host route, when the data root holds no node (nothing changed in any of these); 3 when no running node answered, or none has written a status file; and 1 for a fault on this machine.
The status text opens with valis-development-status 1, then one key and value per line:
state, socket, route, actor, since, mode (normal or recovery), and
persisted, then the recovery analysis described under
Recovery mode, whose keys read none on a node in normal mode.
10.2. Reach the live image
From your own machine, as a member of group valis on the node's host, forward a local
port to the socket and attach Sly to it:
ssh -N -L 4005:/run/valis/valis/development.sock HOST
Then, in Emacs, M-x sly-connect, host localhost, port 4005.
ssh binds the local end to loopback unless you name another address; never name one.
The forward needs the host's sshd to allow it (AllowStreamLocalForwarding, yes by
default) and needs your key's line in authorized_keys to carry no restrict and no
no-port-forwarding. The node's run account has no .sly-secret, so keep none in your
own home: Sly sends one first when it finds it, and the node is not expecting it.
10.2.1. When the forward is refused: a relay as the run account
When the host refuses the forward, or your account is not in group valis, relay the
socket as the run account through sudo instead. Run this on your own machine; it
listens on 127.0.0.1:4005 and opens one relay over ssh for each connection:
socat TCP-LISTEN:4005,bind=127.0.0.1,reuseaddr,fork EXEC:'ssh HOST sudo -n -u valis socat STDIO UNIX-CONNECT\:/run/valis/valis/development.sock'
Then M-x sly-connect to localhost, port 4005, as above. The ssh inside needs a key
that logs in without a prompt, and sudo -n needs a rule that lets your account run
commands as valis without a password. On a deployed node, socat 1.8.0.3 ran this
way under sudo -n -u valis with no terminal. Where the host has no socat, OpenBSD
nc does the far end: replace the inner socat STDIO UNIX-CONNECT\:PATH with
nc -U PATH.
10.3. Recovery mode
When a node's starts keep failing, the host agent starts it in recovery: the development socket open and every protocol service held, so you can attach and repair the image that will not boot. I added it because a node that cannot start has no owner path, and the listener is the one route left.
How a node enters it. The host agent counts consecutive failed starts. A start fails when its run ends before the node reports ready, whatever ended it, or ends in a failure before it has served for the host agent's settle window, 300 seconds by default. A run that serves past the settle window resets the count, and so does a clean exit. A run whose end the host agent never saw, such as one the watchdog killed before the settle window passed, counts as failed. At three failed starts, by default, the next start is a recovery start.
A run that exits 78, the status valis gives when it refuses a start that only a host act can repair, sets the count to the threshold at once. systemd does not restart the unit after 78, so the next start you make is a recovery start. The exception is a trial release's 78, which counts as one ordinary failed start.
The journal says so: recovery start: N consecutive failed starts (threshold 3);
launching RELEASE in recovery mode.
What it holds and what it starts. In order:
- it reads its configuration;
- it assembles the recovery analysis from the last run's fault record and the host agent's count;
- it opens the development socket, with route
recoveryand actorfulcrum, and setsdevelopment.listener; - it holds a session with the host agent, used only to ask for a restart;
- on a node whose owner port is open, it serves a status-only owner door that holds
/node/development/statusand/node/release/reportfor the node's own key and enrolled keys, and nothing else; - it brings up the mail transport: the operator-state database and local delivery;
- it sends the listener's mail notice;
- it reports ready, with the status line
recovery: development listener open, protocol services held.
DNS, the :443 edge, :80, mail on :25, modules, and every other owner door stay held.
The host agent refuses admission and the DNS bind to a recovering node and answers its
restart request. The journal logs each step as recovery.rung. A step after the socket
that fails logs at WARN, naming the missing piece, and the run goes on.
A recovery that cannot start. When the socket itself cannot open, the journal logs
development-listener.refused and recovery.rung at ERROR, and the node exits 78 before
it contacts the host agent. The unit then stays down until you act on the host: read
the journal, fix what it names, and start the unit, which runs as an ordinary start.
What you see.
systemctl status valis.serviceshows the recovery status line;- the journal shows the host agent's recovery line and the
recovery.runglines; /run/valis/valis/development.statusreadsmode recoveryand carries the analysis;- the mail notice is queued for the owner's Maildir when the mail transport came up. ⚠ Gap: in recovery the notice is queued but not yet delivered; on a test guest it was still in the queue 150 s after the listener opened, so do not wait on it;
- from your own machine, over the owner door,
valis --node NODE development statusprints the same status text andvalis --node NODE release reportprints the release report, each exiting 0. Every other owner verb is refused after the attach with exit 2, and its reason says the node is in recovery. A key the node has not enrolled still reads as a key given no owner view.
The analysis is the rest of the status text:
failure-condition,failure-message,failure-rung- the condition that ended the
last failed run, its message with any secret setting's value withheld, and the
bring-up step it ended in. When the last run left no fault record and an earlier run
of the same streak did, the condition and step read
unknown, no traceback follows, and the message says how the last run ended and names the earlier run's fault as earlier; launch-reason,failed-starts,run-ends- why this release was launched, the
count, and up to five ends, each as
exit:N@S,signal:N@S, orunrecorded@-for an end the host agent never saw,Sthe run's length in seconds; release-running,failed-trial,jailed- as in
release report; changed-since-good- the settings, modules, and release that changed since the last boot that passed its health gate;
next-step- one suggested repair;
traceback-lines- the number of lines after it, each a function on the failing stack, innermost first, named without its arguments.
The node keeps the fault record and the last good boot's record in its data root, as
/var/lib/valis/valis/boot-fault.record and /var/lib/valis/valis/last-good-boot.record,
where they outlive the reboot that empties the runtime root. A boot that passes its
health gate removes the fault record. Each normal start also writes
/var/lib/valis/valis/run-start.record, naming the run, and the fault record names the
run that wrote it, so a fault an earlier run left is never reported as the cause of the
last run's end. A condition that ends a recovery run is recorded in
/var/lib/valis/valis/recovery-fault.record, in the same form, and leaves the boot's
fault record as the failed boot left it.
How you leave it. Attach as Reach the live image shows and make the repair the analysis points at. Then ask the host agent for a restart from the REPL:
(valis/src/edge/control-client:request-host-restart)
(:accepted SERVICE) or (:probably-accepted) means the restart is under way. Or restart
the unit on the host:
sudo systemctl restart valis.service
Every end of a recovery run resets the count, so the next start is an ordinary one.
The listener stays open after it, because recovery set development.listener: close it
with development stop once the node is healthy. A close during recovery is refused:
the host route answers error refused: the listener stays open in recovery; restart the
node to leave recovery, and the owner door refuses it as a verb recovery does not serve.
The host agent's launcher configuration sets the threshold and the settle window; a threshold of 0 turns recovery off. See the host deployment contract's provisioning section.
10.4. A node that wedges after it has been running
Recovery does not start for a node that served past the settle window and then wedged:
that run already reset the failed-start count. While the node still answers the host
route, development start --host opens the listener in place. When it does not, the
verb gives up after 15 seconds and names this route: stop the node, set the listener
for its next start, and start it again.
sudo systemctl stop valis
sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis development start --host
sudo systemctl start valis
With the node stopped, the host route finds the configuration unheld and sets
development.listener, the same change as
sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis config set development.listener true.
The next boot opens the socket before anything else binds, and you attach as
Reach the live image shows.
10.5. Retire the old listener port
A node that once had a TCP development listener may still hold its port. Do this once on such a node: reset the stored key and remove the variable from the environment file.
valis --node NODE config reset development.listener-port
sudo sed -i '/^VALIS_SLYNK_PORT=/d' /etc/valis/valis.env
Until the key is reset, each boot logs development-listener.retired-port-stored at
WARN, naming the port. Neither leftover opens anything.
10.6. Not built yet
- The mail notice goes to
postmasterat the first local domain, filed under the owner; no setting names another owner address. Every boot bindsmail.local-domainsas the node's local domains, so the notice routes locally like any other mail for them. Withmail.local-domainsempty, the journal logsdevelopment-listener.notice-skippedwithno-local-domain. On a node running the delivery loop with no local domains bound, the notice is skipped withno-local-routing, because that loop would send it out as relay mail. In recovery the node still tries the operator-state database first; only when it cannot engage it is the notice written straight into your Maildir. - ubik shows a banner while the listener is open. You cannot dismiss it, and its Close the listener button closes the listener. You need a ubik release that includes it.
- Recovery covers only a node whose starts fail. A node that wedges after the settle window needs the host act above.
11. Managing standing grants
A module loads only with two signatures: its publisher's vouch and your grant. You give
a grant for one module, or once for a publisher as a standing grant, which covers
everything that publisher vouches. You file publishers and grants by writing commands to
/cap/admit in your owner session; grant-publisher PUBLISHER-DID mints a standing
grant that lasts a century and files it in one step. A grant you mint yourself and file
with file-standing-grant ends at its own expiry, which admission checks. Because a
standing grant can stand for a century, I made the node keep a record of every change to your trust and ask you to look
at each standing grant again from time to time.
Two read-only files sit beside /cap/admit in the owner's /cap, and nowhere else: no
grant, bearer name, or delegated token reaches them. Each is rendered when you open it,
so everything you read from one open is one state.
/cap/grants- the inventory: one stanza for each grant you hold, then one for each
publisher you have filed, each a run of
KEY VALUElines, stanzas separated by a blank line. A standing grant's stanza carriesgrant(its hash),kind,state(active,withdrawn,backed-out, orrevoked),publisher,filed-at,reviewed-at(ornever),review(dueorok),last-admission,modules, and oneadmittedline for each module it admitted. Times are Unix times. No stanza carries the text of a grant or a vouch. /cap/audit- the grant audit trail, one entry a line, oldest first:
UNIX-TIME ACT SUBJECT ACTOR. ACT is one offiled-publisher,filed-vouch,filed-list,filed-owner-grant,cleared-owner-grant,filed-standing-grant,granted-publisher,withdrew,backed-out,revoked,reviewed, andrestored. SUBJECT is a grant hash, a designation, or a did:key, and ACTOR is the DID of the session that acted, orlocal.
On a node in service the trail only grows, and a restore whose backup carries a trail
replaces it whole. It records changes to trust, so a refused act, or one that changes
nothing, leaves no entry. When an entry cannot be written the act still stands, and its
reply ends with a line saying the trail did not record it. A torn final line, left by a
crash during an append, is skipped. Any other line that is not an entry makes
/cap/audit an error beginning cap audit unreadable: line N. An unreadable or unsafe
trail gives cap audit unreadable: and the reason, with no line number. Either way
/cap/grants, which takes its filing and review times from the trail, is an error
beginning cap grants unreadable. Backups carry the trail (see
Backup and restore).
Reviewing a standing grant. An active standing grant is due a review once more than
module.standing-grant-review-days days have passed since the later of its filing and
its latest review: 90 unless you set it, a whole number from 1 to 3,650 (see
the environment seam). An active grant with neither on record is
due at once. Write reviewed GRANT-HASH to /cap/admit to stamp it, and read back:
reviewed standing grant <hash> at <unix-time>; the review is advisory and admits nothing
The stamp is the trail's reviewed entry and nothing else, so a review whose entry cannot
be written is refused and nothing is stamped. Only an active grant you hold takes one: a
withdrawn, backed-out, or revoked grant is refused. The review is advisory: nothing
refuses a module under a grant that is due. While any active standing grant is due, the
node's status line (systemctl status valis.service) ends by counting them:
; 2 standing grants due a review (90-day interval)
Previewing before you withdraw or back out. Write preview withdraw GRANT-HASH or
preview back-out GRANT-HASH and read what the act would do, with nothing changed. The
first line is preview ACT HASH: nothing was changed, the second state and the grant's
state, and then what the act would do: for a withdrawal, the modules that stay and what a
fresh install from that publisher would then need; for a back-out, the modules admitted
under the grant, every one refused from then on and jailed at the next full restart. When the act would change nothing, one
line says so. A back-out preview is refused wherever the back-out itself would be.
Withdrawing and backing out. withdraw-standing-grant GRANT-HASH ends the grant for
anything new: it admits nothing more, and the modules it admitted stay, across restarts
too. back-out-standing-grant GRANT-HASH withdraws it and revokes its hash, so every
module it admitted is refused from then on. Withdrawal is one way, and there is no
reinstate: to trust that publisher again, file a new standing grant, which carries a new
hash and its own filing time. A publisher holds one active standing grant at a time, so
filing a different one while one is active is refused, naming the active grant's hash.
⚠ Where this stands. The command line has no verb for /cap/admit, /cap/grants, or
/cap/audit. ubik's standing grants panel reads the grants and the audit trail, and
marks a grant reviewed, withdraws it, or backs it out through /cap/admit; filing a
publisher, a list, or a grant still takes a 9P client in your owner session. A back-out refuses the modules it names from then on, but nothing
removes them from a running node: they run until a restart, when a full boot moves each to
the module jail (a resident-only boot leaves them for the next full one), takes it out of the module manifest, names it as not running, and
comes up serving the rest (see Reading the module set).
12. DNS zone management
valis is authoritative for the SOA zones it serves on :53. The zone data itself is
operator state: a node comes up serving whatever zones are in its operator-state
store, and comes up with none on a genesis boot. Loading and maintaining that data
is the valis zone verb family: a headless, owner-authenticated client the deploy
recipe drives non-interactively, and the operator drives by hand.
How it authenticates. From your own machine you name a pinned node with
--node NAME or VALIS_NODE, and each verb presents your enrolled device key and
reaches the node's owner-gated management axis through its owner port, as
From another machine describes:
valis --node NODE zone record add --origin example.org --owner www --ttl 300 --type A --rdata 192.0.2.10
valis --node NODE zone export --origin example.org
valis --node NODE zone record delete --origin example.org --owner www --type A --rdata 192.0.2.10
An added record appears in the export and is answered over UDP and TCP; a deleted one is gone from both. Only a key the node holds in its keyring is given the management axis, so an absent or wrong key fails closed: the management axis never opens to an anonymous caller, and a key the node has not enrolled is refused with exit 2 and changes nothing.
From the node's host, the break-glass route. When you cannot use the route from your
machine (the owner port closed, the pin or the device key lost, or the node not answering
on the owner port), or to mint a transfer key for a secondary (see secondary below),
run the verb on the node's host. There it reaches the management axis over the
resident's loopback fabric, inside the resident's network namespace, at the endpoint
the resident wrote at boot, and presents the owner key custody holds in the
StateDirectory keyfile. The key is supplied as a path, resolved
--keyfile PATH > VALIS_KEYFILE > the StateDirectory keyfile; the key bytes never
cross the command line. The break-glass route
gives the command that gets it there:
sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
/opt/valis/bin/valis zone export --origin example.org \
--keyfile /var/lib/valis/valis/keyfile \
--endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"
A transfer key is kept under the data root of the process that mints it, and the
recipe above keeps root's environment, whose data root is not the node's. To mint one
the node can use, name the node's data root as well, so the key lands in the node's
custody under /var/lib/valis/valis/tsig/:
sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
/usr/bin/env XDG_DATA_HOME=/var/lib/valis \
/opt/valis/bin/valis zone secondary --origin example.org --peer 198.51.100.2 \
--keyfile /var/lib/valis/valis/keyfile \
--endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"
Either way, a verb that refuses before anything is sent exits 2: for a key file it will not present, a responder that is not the expected node, or arguments it refuses.
The verbs.
valis zone create --origin <fqdn> [--peer <addr>] [--secondary-ns <name>] valis zone record add --origin <fqdn> --owner <name> --ttl <s> --type <type> --rdata <value> valis zone record delete --origin <fqdn> --owner <name> --type <type> --rdata <value> valis zone import --origin <fqdn> --file <master-file> valis zone export --origin <fqdn> [--out <file>] [--full] valis zone delete --origin <fqdn> valis zone delegation --origin <fqdn> [--wait] [--timeout <s>] [--interval <s>] [--allow-range <client=range,...>] valis zone secondary --origin <fqdn> --peer <addr> [--notify <ref>] [--key-name <name>] [--master-ip <addr>] valis zone secondary --origin <fqdn> --peer <addr> --key-name <name> --existing-key # Straight to operator state, for a node that is not serving yet: valis zone load --origin <fqdn> --file <master-file> [--migrate] valis zone secondary --offline --origin <fqdn> --peer <addr> [--notify <ref>] [--key-name <name>]
- create brings a whole domain up in one move: it mints the apex records from the domain template, enrols the secondary when you name a peer, and prints the block you paste at the registrar. Reach for this before reaching for import on a new domain.
- record add and record delete publish or retract one typed record and leave the
rest of the zone alone. This is the editing path. An
--ownerending in a dot is absolute; without one it is relative to the zone. The master-file text format is a boundary format for the secondary, never the format you edit in. - delegation asks the parent zone's nameservers whether the delegation has settled,
and exits non-zero until it has. It needs no resident, no keyfile and no database,
so you can run it from anywhere while you wait on a registrar.
--waitpolls. It asks nameservers on port 53 and nowhere else. The zones it walks are somebody else's, so a nameserver whose address lies in a private or other special-purpose range is skipped unlessnet.dial-allowallows that range for the clientdelegation-probe.--allow-rangenames the allowance for one run instead, in the sameclient=rangeform: a lab whose simulated internet sits in 192.168.140.0/22 passes--allow-range delegation-probe=192.168.140.0/22. The check holds that value to thenet.dial-allowrules before it asks any nameserver: a member they refuse stops the run with exit status 2 and an error naming--allow-range, the member and why. - import loads an RFC-1035 zone master file as the named zone, committing it atomically; a malformed master is refused with a non-zero exit and changes nothing, so a bad file never half-lands. Re-importing an origin replaces its zone (advance the SOA serial).
- export writes the zone's master text (to
--out, or standard output). It defaults to the durable, re-importable master: the published records, the faithful thing to archive or re-load. Pass--fullfor the whole serving set, which also includes any transient turn-up records (e.g. ACME challenge records) present at that moment. - delete removes the whole zone (apex-SOA removal is whole-zone removal).
- secondary enrols a downstream secondary for a zone: it lands the allowlist row for
the address the secondary pulls from, mints a TSIG key, and prints the secret and a
BIND snippet to paste on the secondary. Over the owner session the key is minted only
after the node has accepted the enrolment, so a refused, unserved, or unreachable
node leaves no secret behind. With
--offlineit is minted only after operator state has recorded the row, so a write that fails leaves the key already in custody under that name as it was.--existing-keymints nothing: it lands the allowlist row for a key already installed on both sides, and prints no secret. Over a pinned node (--nodeorVALIS_NODE) only--existing-keyapplies. The minting form and--offlineare both refused with exit 2, before any setting is read, anything is minted, or anything is dialled. A key minted on your machine stays in your machine's custody, where the node cannot use it to authenticate a transfer, and--offlinewrites your machine's operator state, never the node's. To mint a new key, run the verb on the node's host. ⚠ The gap. No door on the node mints a transfer key for an existing zone yet, so a new key for a pinned node is minted only on its host. - load and secondary –offline are the two routes that do not need a running
resident. They write operator state directly, reading no keyfile and no management
endpoint, with the connection coming from the node's
operator-state.dsnsetting (or, when the address carries a password, fromVALIS_PG_DSN) and never from the command line. Reach for them on a node that is fresh or recovered and not answering yet. ⚠--offlineis what selects that route. secondary without it drives a running resident over the fabric, exactly like the owner-keyed verbs above, so on a node that is not serving yet it refuses and tells you to add the flag. load takes no such flag and is always direct. Either waysecondaryrecords the allowlist row; without--existing-keyit also mints the TSIG key and prints the BIND snippet to paste on the secondary. Both are walked through in first boot and operator moves.
A committed change is picked up by the running resident without a restart: the serving side re-reads the store, so the node begins answering the new data authoritatively.
At go-live. After a fresh node is up and healthy (serving :53, restore-proven per
the go-public gate above) the deploy recipe imports the node's own zone before the
address is cut over: whichever domain that node serves, with its apex SOA, its apex
NS, and the ns1 address record. The recipe runs valis zone import as
the account that can read the 0600 keyfile, then confirms the node answers the zone
over loopback (dig against :53) and that valis zone export round-trips before
declaring the node live. Zone master files are operator data staged onto the host, not
part of the delivery binary.
13. Publishing to a node, and updating one
Neither of these takes the node down, and neither wants you editing files on the host.
Publishing a page. publish places a publication into /pub on a running node,
owner-keyed like the zone verbs. It is an ordinary operator act at any time, not a
deploy-time one, and you run it from your own machine and name the node:
valis --node NODE publish --slug notes --file notes.html --content-type text/html
Its whole form is:
valis publish --slug <name> --file <path> --content-type <type> [--plaintext-file <path>]
The first three are required. The slug is lower-case letters, digits, and hyphens, and
--file names a file on the machine you run the verb on, placed on the node byte for
byte. A slug that already exists is revised, not refused, and the node serves the
new text at once. With a key the node has not enrolled, the verb is refused with exit 2
and nothing is placed.
--plaintext-file is optional. It names a UTF-8 plain-text rendering of the page, and
a Gopher client selecting the page is served that text in place of the body, with the
page listed as a text item. make publish-site supplies one for every manual page, an
Org text export of the same source as its HTML. Publish a page again without the flag
and the node keeps the text it already holds.
⚠ The gap. No verb retracts a publication. A page you publish stays served until you revise it, so publish only what you mean to leave up.
From the node's host, the break-glass route. When you cannot use the route from your
machine (the owner port closed, the pin or the device key lost, or the node not answering
on the owner port), run publish on the node's host, inside the node's namespace with
its keyfile and endpoint (see the break-glass route).
The file must then be on the host, readable by the run account:
sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
/opt/valis/bin/valis publish --slug notes --file /var/lib/valis/notes.html \
--content-type text/html \
--keyfile /var/lib/valis/valis/keyfile \
--endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"
Updating the binary. On a node that keeps releases, run apply from your own machine
and name the node: valis --node NODE apply carries the release to the node over the
owner path, with no shell on the host, as
Releasing a new binary to a running node describes. Without
--node, apply is the host-local mode, for a host with no owner path: it takes a
delivery archive that is already on the host, puts it in place, restarts the serving
unit, and confirms the node still answers:
valis apply --artifact <path> --confirm-server <addr> --confirm-zone <fqdn>
It transfers nothing, so move the archive onto the host first. Without
--confirm-server it reads VALIS_APPLY_CONFIRM_SERVER; without --confirm-zone it
reads edge.domain in the configuration under your own account's data root, or
VALIS_EDGE_DOMAIN in your shell, and when neither is set, the node's own
edge.domain. When yours and the node's name different domains, the apply refuses and
names both: that disagreement is exactly the wrong-node case confirmation is there to
catch. Pass --confirm-zone to settle it; the flag always wins, since a node may
confirm for a zone other than its edge domain. The report names the zone it confirmed
and where that zone came from. There is no loopback default: name them or the apply refuses before anything moves. That refusal
is the point. Exit 2 means it refused and nothing moved; exit 3 means it moved and
could not confirm, which is the case that wants you looking. --dry-run walks it
without touching the host.
By default apply places the serving binary alone. --with-resident and --with-steer
are the explicit ask for the wider set, so the privileged parts of the delivery never
move because you forgot they were in the archive.
An apply leaves the host agent's configuration store as it is, so a node whose owner port was open answers on it again once the unit is back, under the same DID, and you go on managing it from your own machine with the pin you already hold.
13.1. Letting apply read the node's settings
Before it moves anything, apply reads the node's settings so it can judge what the
node will actually boot on. It does this by running its own binary as config list
under the node's run account, the owner of the unit's StateDirectory, through
sudo -n. It runs its own binary, never the one standing at the destination, because
an older binary there may not know the config verb and would start a node instead.
The read takes no lock and writes nothing.
Grant the deploy account that command in a file under /etc/sudoers.d/ (edit it with
visudo -f):
<deploy-account> ALL=(valis) NOPASSWD: /usr/bin/env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis config list
sudo matches the command word for word, so the grant fits only when:
- you run apply as
/opt/valis/bin/valis(a symbolic link to it is fine, since the binary names itself by its resolved path); - the unit's
StateDirectoryisvalis, which apply turns into/var/lib/valiswith no trailing slash; - the run account is
valis.
Understand what this grant is before you add it. It is not a narrow read permission.
apply needs /opt/valis/bin writable by the deploy account so it can replace the
binary, so the deploy account can put any binary it likes at /opt/valis/bin/valis and
run it as valis through this grant at once: no restart, no unit and no apply log
stand in the way. In effect the grant lets the deploy account run code it chose as the
node's run account, with everything that account can read and write, the node's store
included. That is the same trust the deploy account already holds through
sudo -n systemctl restart valis.service, which apply runs to restart the node, since
a restart runs whatever stands in /opt/valis/bin, but this one takes effect
immediately. Give it only to an account you already trust to deploy the node.
Through the grant the deploy account also sees every setting config list shows.
A secret shows only as set or unset, but a personal setting shows its value: today
that is acme.account-contact, the address the node's ACME account is registered to.
Without the grant, or when any of the three conditions above does not hold, the read
is refused, apply checks the flags and the unit's environment alone, as it did
before it could read the node, and its report says why the node's settings were not
read. A refusal for a missing site fact says so too.
One gap remains: config list does not show which keys the node's first boot already
settled from valis.env. A required key reset in the store after that boot reads as
unset to the node, but passes apply's check when valis.env still names it. The check
is never weaker than the environment-only check it replaced.
14. Releasing a new binary to a running node
A new valis binary reaches a running node as a release: you build it on your own
machine, and valis --node NODE apply carries it to the node over the owner path, has it
signed and admitted on the same two signatures a module needs, and makes it the next
boot's trial. The trial gets one boot. When that boot passes the node's health gate the
release becomes the node's default; any other end leaves the node on the default it ran
before, and you read why. ARCHITECTURE.org
gives the design.
Only the valis binary moves this way. fulcrum-resident and the director it loads run as
root, and the owner path writes only as the run account, so both still change only by a
host act. The delivery kit that copies binaries onto a host over ssh is interim: it is how
a host gets those two and its first release root, and it is not the procedure for the
valis binary.
14.1. Before the first release
Each of these is done once per node.
A release root. The unit must declare the release root and the launch-mark directory (the host layout), and
fulcrum's launcher configuration must name them. On the host, as root:fulcrum-resident release-root set /var/lib/valis-releases/ /var/lib/valis-launch/ fulcrum-resident release-root statusThe change takes effect at the next restart of
valis.service.statusprints the two paths and, for each directory, its owner and mode and whether they are what a launch requires.release-root clearremoves both keys, and the node then launches its configured binary as it did before. A node whose host agent names no release root refuses an upload withunsupported: this node has no release root.A delegation that covers releases. The node's publisher signing key must be delegated over
/releaseas well as/module. A node provisioned before releases existed has a delegation over/moduleonly, and its sign door refuses a release withsigning refused: not signed: the delegation to the signing key covers no /release; delegate the signing key again. Delegate it once before the first release:valis --node NODE publisher delegateModule vouches signed under the old delegation still verify.
- Your standing grant to the publisher. A release is admitted on the publisher's vouch and your consent, and the standing grant you filed for DeepSky supplies the consent (Managing standing grants). A grant over one module admits no release.
- A fresh publisher list. Each apply is judged as a fresh install, so the node must hold a list from the publisher issued in the last seven days (Renewing the publisher list).
14.2. Building and applying a release
Build the delivery archive as for any host:
make dist
The archive, dist/valis-serving-<version>.tar.gz, carries the serving binary and,
beside it, opt/valis/bin/release-manifest: the binary's version, commit, compiler, and
the store format it reads and writes. valis release-manifest prints the manifest of
the valis binary you run it with. Then apply it from your own machine:
valis --node NODE apply --artifact dist/valis-serving-<version>.tar.gz \
--confirm-zone FQDN [--confirm-server ADDRESS]
--confirm-zone is required, and --confirm-server also reads
VALIS_APPLY_CONFIRM_SERVER. --vouch FILE gives a vouch made elsewhere instead of
asking the node's sign door. --wait SECONDS bounds the wait for the new boot and
--trial-wait SECONDS the wait for the trial's verdict, 180 seconds each by default.
The host-local flags (--destination, --unit, --with-resident, --with-steer,
--dry-run) are refused beside --node.
Before it connects, the verb reads the archive and mints the release's identity, so a
malformed archive or manifest is refused with nothing sent. Over one owner session it
reads the node's boot identity and release report, uploads the binary and its manifest,
has the node's sign door sign the staged release, and applies it. The node places the
release, records it as the next boot's trial, and asks fulcrum for the unit restart,
the same restart valis restart asks for. The verb then waits for a new boot identity,
reads the release report until it shows a verdict, and asks the server for the zone's
SOA. The lines it prints include:
valis: apply: sending /release/<hex> (version <version>, commit <commit>) to NODE valis: apply: the node recorded /release/<hex> as its next boot's trial: ok restarting valis.service valis: apply: the node is back, boot <boot identity> valis: apply: /release/<hex> passed its trial and is the default
then the previous and new release, version, commit, and compiler, the zone answer, and the report. The upload is about 37.5 MB in writes of under 8 KiB, each waiting for its reply, so it takes longer the further you are from the node. Its time has not been measured yet.
Exit statuses.
- 0
- the release report names the release as the running default, and the server answers authoritatively for the zone.
- 1
- a fault on this machine: I/O, the scratch directory the archive is unpacked into, or a condition outside the classes below.
- 2
- refused, and the node runs what it ran: a malformed command line, an archive with no serving binary or no release manifest, an unreadable vouch file, an owner key file that is not safe to use, a refusal at the handshake, the upload, the stage, the sign door, or apply (the jail entry is printed when the bytes went to the jail), or an apply the node recorded but took no restart for, whose trial then runs at the node's next restart.
- 3
- not reached or not confirmed: the node could not be reached before apply was
sent, no new boot came within
--wait, no verdict came within--trial-wait, another node answered on the pin during a wait, or the server gave no authoritative answer. What was read last is printed. - 7
- the trial ran and the node fell back to its default; the failed trial is printed as the report gives it, then the report.
What the node refuses at apply. A release whose bytes mint an identity other than the
one named, or whose signatures the gate refuses, goes to the release jail under
jail/ in the release root, and the verb prints the entry. While a trial's boot is
under way another apply is refused with error trial-under-way. A release whose trial
failed on this node is refused for good, with error trial-failed /release/<hex> <end>.
⚠ Any restart spends a pending trial's boot. Once apply records the trial, the next start of the unit launches it, whatever caused that start.
14.3. Reading the release report
valis --node NODE release report
prints what the node serves at /node/release/report, exactly as served, and changes
nothing. One key and value per line, every key present, in this order:
release-root- where the node keeps its releases, or
none. running- what
fulcrumlaunched: a release's hex, orbasefor the configured binary. launch-reasondefault,trial,fallback, orfloor.floormeans a record or a digest could not be verified, andfulcrumlaunched the configured binary.version,commit,compiler- read from the running image.
default- the selection record's default;
baseon a node with no release yet. previous- the default the last promotion replaced.
trial- the pending trial, or
none. failed-trial- the last trial
fulcrumreported failed, and how it ended:exit:N,signal:N, orunconfirmed. jailed- the number of entries in the release jail.
store-format- the highest store format any boot on this node has recorded.
development-listeneropen SOCKET ROUTE SINCEwhile the development listener is open, naming its socket, the route that opened it, and when; otherwiseclosed. See The development listener. A node older than this key leaves it out.
none says nothing of that kind is held, and unknown says the fact could not be read.
running and launch-reason read unknown on a binary fulcrum launched without its
launch-time selector. Exit 0 read; 2 refused, with nothing read; 3 the node could not be
reached; 1 a fault on this machine.
14.4. A failed trial
A trial that dies, exits, or hangs before its boot passes the health gate is recorded as
failed, and the next start runs the default. fulcrum turns a trial's exit 78, which
would otherwise hold the unit down, into a restart, so a release that refuses its
configuration costs one restart, never the node. A trial that hangs before it reports ready is
ended by systemd's 90-second start timeout, and after that by the 30-second watchdog;
its end then reads unconfirmed. apply exits 7 and prints, for example:
valis: apply: the trial of /release/<hex> failed and the node fell back to its default failed-trial <hex> exit:78
then the report, which reads running with the default, launch-reason fallback,
trial none, and the failed-trial line. The journal carries fulcrum's messages
launching valis base (fallback) trial <hex> failed (exit:78); the next start runs base
and valis's release.trial-failed event, naming the release, its end, and what the
fallback boot launched.
Fix forward. A release whose trial failed is never retried, by apply or by rollback, so you build a corrected release, which mints a new identity, and apply that. The node is already serving its default. Putting an older binary back by a host act is a tool for a node the owner path cannot reach, not the recovery plan.
14.5. Rolling back to a binary the node ran
A rollback is for a release that passed its trial and is wrong. It returns the node to a
release it ran, or to base, through the same one-boot trial:
valis --node NODE release rollback [IDENTITY|base] [--override-downgrade] \
[--confirm-zone FQDN [--confirm-server ADDRESS]] [--wait SECONDS] [--trial-wait SECONDS]
With no IDENTITY the node returns to the default its last promotion replaced (the
report's previous). IDENTITY is /release/<hex> or the hex alone. The verb waits as
apply does, and confirms the zone only when you pass --confirm-zone.
The node refuses, with no override, a target that is already the default, a release that never ran on the node, failed its trial, is in the release jail or no longer retained, or whose bytes no longer mint its identity, and one its signatures no longer admit. It judges the signatures as a restart judges a module's: a revocation refuses, and expiry is judged at the release's recorded admission time, or now when none was recorded.
The store-format bound. Each release declares the store format it reads and writes, and
the node records the highest any boot on it has run with. A rollback to a release below
that, or to one whose format cannot be read, is refused as a downgrade, because an older
binary cannot be relied on to read what a newer one wrote and refuses some records
whole. base has a store format only once it has booted under the selector; until then a
rollback to base is a downgrade. --override-downgrade commits it anyway; the node
records the override and logs release.rollback-override.
Exit statuses. 0: the target is the default (and the zone answers, with
--confirm-zone). 2: refused, a downgrade included, or recorded with no restart taken.
3: not reached or not confirmed, including a reply lost with no IDENTITY named. 7: the
trial fell back. 1: a fault on this machine.
14.6. Signing releases and renewing the publisher list
On a node that holds the publisher's keys, apply has the node's sign door sign each release, so you sign nothing by hand. To sign a staged release yourself, name its absolute path on the node:
valis --node NODE publisher sign-release PATH
The publisher list's seven days. A release is admitted only while the node holds a list
from its publisher issued in the last seven days. When the list is older, apply is
refused with a reason ending older than the 604800 a fresh install allows, and the verb
prints the renewal. Sign a new list on the node:
valis --node NODE publisher sign-list
The reply's first line is publisher-list-signed, followed by issuer, sequence, and
list with the signed list's text. Signing does not file it. File it by writing
file-list and that text, as one write, to /cap/admit in your owner session; the node
answers filed publisher list from <did> at sequence <n>. Then apply again.
⚠ The gap. No command-line verb and no ubik panel files a publisher list, so renewing it takes a 9P client in your owner session, as filing a publisher or a grant does.
14.7. Where this stands
- Not yet read on a running node. The release path is built and tested in the image on
both sides, valis and
fulcrum. It has not yet run on a test guest or a deployed node, so none of the replies, report lines, or journal lines above has been captured from one, and the upload's time is unmeasured. - fulcrum-resident and the director move only by a host act. They run as root, and the owner path writes only as the run account.
- Upload is the only source. A release reaches the node only by upload over the owner session. A fetch by git tag, over HTTPS, or from the DHT would be a further source behind the same door; none is built.
- A release carries no source. The release tree holds the binary and its manifest. The node's own source does not travel with it yet.
- On the origin node both signatures are local. The publisher's creating identity is derived from the node's own seed, and any enrolled owner key can ask its sign door to sign. There the gate proves the release's integrity and records your consent; it is no defence against a compromised image, which holds both keys.
- Staging is not swept. A session that drops without closing the release door, or an
apply refused at the sign door after the stage, leaves its bytes under
staging/in the release root. Nothing removes them yet. - The report shows one failed trial.
failed-trialis the last one; the release lifetime record keeps every one.
15. The module source root
module.source-root (seeded from VALIS_MODULE_SOURCE_ROOT) names the directory
holding the bytes of every module the node admits. Each module lives in a directory named for the identity of its bytes,
written once and never changed, and a new version of a module is a new directory
beside the old one. Never edit files under it. Keep it apart from the directory that
holds the owner keyfile, and keep it, and every directory above it, unwritable by
other users: an install refuses a root that is not.
I made the root a fact you state rather than a place valis picks, because it is where the code a node compiles into itself comes from. It is a boot-site requirement, so it is checked at three points:
- A resident whose configuration names no root refuses to boot, and the refusal names the key, and the variable that seeds it at a first boot. With a root named, the resident creates the directory at boot.
valis applyreads the node's settings, and the unit's environment, before it moves anything. When the root, or any other fact the resident needs to boot, is missing from both, the apply refuses with exit 2 and the node keeps serving what it served. Set it as the run account withvalis config set module.source-root ...with the node stopped, or, before the node's first boot, add the line to/etc/valis/valis.env, then apply.- Provisioning renders the env file from the same requirements, so a provisioned node carries the root.
What a node does with its modules. A module in the set the node boots by default is admitted again from its identity directory at every boot, and its publisher's and owner's signatures are checked again then. A credential that expired after the module was admitted still admits it; a revocation recorded since stops it from loading. A node that serves real names, because it declares an edge domain or is authoritative for zones held on someone's behalf, takes modules as any other node does: after three boots running die inside one module, the next boot that stands the installed modules up leaves it out, moves it to the module jail, and comes up serving everything else. A module that cannot be admitted again at boot does not run: the node logs it by name with the reason, comes up serving everything else, and names it on its status line. A backup carries no module bytes, so a node restored with modules recorded comes up without each one whose identity directory is not back under its module source root. Put those directories back and restart the unit to bring them up.
An install is tentative until it is made permanent. An install or a supersede in a
running node takes effect at once and is recorded as a new generation of the node's
module set, marked tentative. The set the node boots by default does not change, so a
restart before that generation is made permanent stands up the boot default, without
the change. I wanted a change that breaks the node to cost one restart, never the node.
Making the generation permanent moves the boot default to it, and the next restart
stands the change up. The node does that on its own once the change has passed every
health check for a whole health window. Each pass of the health loop, every 15 seconds,
checks that what the running generation bound is still served, and one failed check
starts the window again. The window is 900 seconds unless you set
module.health-window-seconds, from 60 to 86,400. When you would rather decide
yourself, set module.bless-required, and nothing is made permanent until you bless it:
valis --node NODE config set module.health-window-seconds 1800
valis --node NODE config set module.bless-required true
Both apply at the next pass. The health loop runs only under the unit's supervisor, so
a development image makes a change permanent only when you bless it, and so does a boot
that stood up only its resident set. Each promotion is logged as
module.generation-promoted, naming the generation and the window.
To make a change permanent yourself, bless its generation with
valis --node NODE module bless N, N being the number in the install reply's
generation line. Only the generation the node is running can be blessed, and only
while it is tentative. Module lifetime below covers the verbs,
the reads, and the journal.
Rejecting a change. To keep a change that is not yet permanent from ever becoming
permanent, reject its generation with valis --node NODE module reject N, logged as
module.generation-rejected. A change recorded on top of a rejected one becomes
permanent only when you bless it. When the node runs the generation you rejected, its
image is marked diverged until you restart the unit, and the restart returns the node
to its boot default. Only a generation that is not yet permanent, and not already
rejected, can be rejected.
When the node degrades. When two passes of the health loop in a row find something
the running generation bound no longer served, the node logs module.degraded at
ERROR, once, naming each failing check. It logs nothing more for that episode until a
pass finds every check passing, which it logs as module.recovered. The node restarts
itself only when you set module.restart-when-degraded; otherwise valis module
repair is yours to run.
To install a module into a running node, or to supersede the version it carries, use
valis module install (Installing a module as the owner
below). Two verbs let you check a module first, on the node's host. Neither binds a port
or writes the node's stores, so either may run beside the running node:
valis module try-install /module/HEXinstalls the module into a process of its own, through the same gate a running node applies, reading the node's keys and stores, and records nothing. Run it as the run account on the node's data root, so it reads the node's configuration:sudo -u valis env XDG_DATA_HOME=/var/lib/valis \ /opt/valis/bin/valis module try-install /module/<hex>Exit 0 means the module was admitted and loaded. Exit 2 means you gave the verb no designation or more than one; 3, the argument is not a module designation; 4, no identity directory holds that module; 5, the module was refused before any of its code ran; 6, the module source root or the node state the verb needs is missing or unusable, or something else stopped the verb, such as a scratch directory it could not make; 7, both signatures admitted the module and its load then failed, so some of its code may have run. The verb's last line on standard output begins
valis-dry-run:and carries the same JSON verdicttry-supersedeprints, withphaseset tonew. That line is what a node reads when it tries an install first: before installing a module, a running node runs this verb in a child process of its own binary and goes on only when the verdict says the module loaded.valis module try-supersede /module/OLD /module/NEWloads the old version as a restart would and then the new one as an install would, again in a process of its own. It prints one line beginningvalis-dry-run:, carrying a JSON verdict that names what the new version redefined, and exits with the try-install status for the version named in the verdict'sphasefield. The node's log can write to standard output as well, so take the line with that prefix. A node runs this same check in a child process before it supersedes a running module.
Where a node compiles modules. A node compiles each module beneath module-fasls/
in its data root (/var/lib/valis/module-fasls/ on a packaged node), in a directory
named for the module's identity, and compiles a library a module needs that the binary
does not carry beneath module-fasls/deps/. Nothing is compiled into the run account's
home. A node upgraded from an earlier release keeps that release's compiled output under
module-fasls/, filed by each source's full path (.../var/lib/...); nothing reads it
any more. To reclaim the space, stop the unit, remove module-fasls/, and start it: the
next boot compiles its modules again.
A supersede is always tentative. A change which a running image cannot make in place is refused before any of it is compiled: a changed structure, macro, constant, type, or inline or ftype declaration, a removed file, a changed grouping form, or a change inside a top-level form the comparison compares only whole. The new version must then pass the child dry run and the node's own gate. Even then the old version's code can stay live: a closure that captured it keeps running it, and the scan for leftover code can miss some. So the node records its image as diverged from what a restart would build. Restart the unit when you need the running image to match its record. That is apart from the generation the supersede records, which is not yet permanent, like any install's: making it permanent sets what a restart stands up, and only the restart ends the divergence.
15.1. Installing a module as the owner
valis module install asks a running node, as its owner, to install a module, or to
supersede in place the version of it the node already runs. You name the module by its
designation and say where its bytes are:
valis --node NODE module install /module/<hex>
valis --node NODE module install /module/<hex> --dist /path/to/release
valis --node NODE module install /module/<hex> --git REPOSITORY TAG
With no source, the node installs the bytes already in the module's identity directory
under its module source root. --dist names a release directory on the node by its
absolute path, and it must lie apart from the node's keys and data. --git names a
third party's tagged release in a repository the node reaches over https or ssh. Bytes
from either source land only when they are the module the designation names. The
request carries no system name, vouch, or signature: you file publishers and grants
through /cap/admit (Managing standing grants), and the
module loads only when the node's two-signature gate admits it. I kept trust out of the
request, so nothing a request carries can admit the bytes it names.
Run the verb from your own machine with --node, like every owner verb, or on the
node's host by the break-glass route.
What the node does, in order. Each step that refuses has loaded nothing.
- When the node already carries the identity you named (loaded from its identity
directory, recorded in its module manifest, and admitted in its vouch store), it
answers
already-currentand does nothing else. - It refuses the request when a newer instance of the node holds the instance fence
(
fenced). - It fetches or copies the release you named, giving a fetch 300 seconds, and lands it only when its bytes are the identity you named.
- It takes the module change lock, so one change runs at a time; a request that waits
more than 5 seconds for it is refused (
change-in-progress). - It tries the change in a child process of its own binary within 300 seconds: for a
new module the
try-installcheck above; for a module it already runs, thetry-supersedecheck, once a comparison of the two versions finds no change a running image cannot make in place. It goes on only when the child loaded the module. - It checks the fence again, then its gate checks the publisher's vouch and your grant and loads the module into the running image. It records the change as a tentative generation of the module set, so a restart stands it up again only once that generation is made permanent, and puts any edge adapter a new module registered in front of the running edge, with no restart.
The reply. valis module install prints the node's reply as it came: a token on the
first line, then KEY VALUE lines, system and designation first.
| Token | What it means | What to do next |
|---|---|---|
installed-serving |
The module was admitted, loaded, and recorded in a tentative generation. Each adapter line names an edge adapter answering now, then its ports. |
Nothing. valis module show SYSTEM reads it back. |
installed-unserved |
Admitted, loaded, and recorded in a tentative generation, with nothing answering on the edge. The reason line says why: no-edge-adapter (the module registered none), edge-not-running (the edge binds it when it next starts), or attach-refused (the edge would not attach it). |
For attach-refused, read the journal line naming the adapter, below. Otherwise nothing. |
installed-unrecorded |
Admitted and loaded, so it runs now, but its record could not be published. The reason line says why, and runs-now yes and after-restart absent say that a restart does not stand it up again. |
Put right what reason names, then restart the unit before you ask again. Until a restart the node refuses another request for that module as not-carried. |
superseded |
The version the node ran, previous, was replaced in place, tentative until-restart. recorded says whether a tentative generation names the new version. residue counts the old version's code the scan found still alive, or reads unscanned. Each residue-code line names one item: its file within the directory the old version was loaded from, then each of its function names after a tab, since a closure's name holds spaces. Past 32 items, residue-code-more says how many more the scan found. |
Restart the unit when you need the image to match its record. With recorded no, or before the generation is made permanent, a restart returns to the version the boot default names. |
already-current |
The node already carries, records, and admitted that identity. Nothing changed. | Nothing. |
After installed-unrecorded the verb also writes
valis: the module runs now and will not return after a restart on standard error.
installed-serving, installed-unserved, and a recorded superseded reply each carry a
generation N tentative line: N is the generation the change recorded, which is not
yet permanent until it is made so, as described under
The module source root. On superseded it comes
after the residue lines and before tentative until-restart, which says something
else: the image stays diverged from its record until a restart.
What a supersede leaves alive. Supersede an edge module whose protocol object holds a
closure and you should expect a residue-code line naming the old version's constructor
and the closure it made, from the file that defines them. Supersede leaves the bound
port, its listening socket, and the protocol object the old version built in place. That
object keeps whatever the old constructor captured, so a reply built from those values
still carries the old version's text until a restart builds the protocol again from the
new version. I keep this by design: replacing the protocol object would mean rebinding a
port somebody may be connected to. The scan can also miss that item, so residue 0
after a supersede of an edge module does not show the old version gone. Only a restart
into the new version clears the old closure.
Exit statuses.
- 0: the node answered with one of the five tokens.
- 1: a fault on this machine, or a reply this client cannot read.
- 2: refused with nothing installed: a command line the verb does not take (printed
with its usage), a refusal before your key was sent, a key the node gave no owner
view, a node with no
/modulesdoor, or a node refusal, printed asvalis: the node refused:and the node's text. - 3: the node could not be reached (including a connect that did not complete in time), or the session with it failed.
- 7: the install failed and code of the module may have run: the gate admitted it and its load failed, or the node's text does not say how far it got.
When the session failed or a fault on this machine stopped the verb after the request
was sent, the reply was lost, and a second line says
valis: module: the request may have reached the node; run valis module list to see what it holds before you ask again.
A reply this client cannot read prints only its own line.
Refusals. A refusal is an error beginning module install refused, and nothing of the
module was loaded. Apart from a malformed request or a caller who is not the owner,
neither of which the verb sends, the reason follows in parentheses:
not-a-designation: the request names no module.not-a-source: the request names no source an owner may name.no-source-root: the node has no module source root.fenced: a newer instance of the node has taken over.fetch-timed-out: the fetch ran out of time.not-the-requested-identity: the release holds other bytes.staging-refused: the release could not be fetched or staged.conflict: its identity directory already holds other bytes.vouch-refused: the node refused the vouch the release ships.bytes-not-found: no identity directory holds the module.not-one-system: the identity directory holds no system definition, or more than one.change-in-progress: another module change held the lock for more than 5 seconds.dry-run-unavailable: the child could not be asked, or did not answer within 300 seconds.dry-run-refused: the child answered that the module did not load.install-refused: the two-signature gate refused it, and the text after the reason says why.- For a module the node already runs:
not-carried(the running version was never admitted as a module, or an unrecorded change awaits a restart),same-identity,publisher-changed(the new version comes from another publisher),restart-required(a change a running image cannot make in place), anddry-run-failed.
A load that failed after the gate admitted the module is not a refusal: its error begins
module install failed after loading, the image is recorded as diverged, and a later
request for that module is refused as restart-required until you restart the unit.
The journal. The node logs each outcome, so journalctl -u valis.service shows what
it did with a request:
module /module/<hex> landed at <module-source-root>/<hex> module <system> installed as /module/<hex> module <system> (/module/<hex>) installed for the owner: installed-serving module <system> (/module/<hex>) is already current for the owner module <system> superseded, tentatively until a restart, from /module/<old> to /module/<new> module <system> superseded for the owner, from /module/<old> to /module/<new>
and at WARN, for what did not go as asked:
module <system> not installed from <directory>: <reason> module <system> installed from <directory> and running, not recorded: <reason> module <system> (/module/<hex>) installed for the owner and not recorded: <reason> the request for /module/<hex> was refused (<reason>): <cause> the module change for /module/<hex> was refused (<reason>): <detail>; nothing was loaded the module change for /module/<hex> failed after the gate admitted it (load-failed): <detail>; code of the module may have run in this image module adapter <name> was admitted and does not answer: <reason>
A supersede whose record could not be published logs its line with , not recorded
at the end.
How long an install holds its session. The verb waits for the node's reply, and the node answers only once the change is done: up to 5 seconds for the module change lock, up to 300 seconds for a fetch, up to 300 seconds for the dry run, and then the compile and load, which have no bound. Keep the session open until the reply arrives.
From a 9P client. The door is /modules/install in the owner's session. Write the
request as newline-framed lines, install, the designation, and then optionally the
source (here, dist and the absolute directory on the next line, or git and the
repository and then the tag), then read the reply, then clunk. The first read runs the
request and the fid keeps its outcome. A clunk before any read still runs it, but the
reply is lost. A session that is not the owner's finds no /modules.
⚠ Where this stands. Nothing rolls an installed module back on a running node; you can take one out of service, as Disabling a module describes. ubik's modules panel writes this door too. An install from a second machine is proven against a test guest and against a node that serves real names (see Checks on the module path).
15.2. Disabling a module
valis module disable asks a running node, as its owner, to take a module out of
service now and leave it out from the next restart on:
valis --node NODE module disable /module/<hex>
You name the module by the designation it was installed from. At once, everything the
node can still attribute to the module stops: its active parts are stopped through their own :stop, its
edge and DNS adapters are retired with every port and serve loop they bound, its mail
adapters are unregistered, and everything the node handed its parts is taken back. In
the same act the node records a new generation of the module set without it and makes
that generation the boot default at once, so the next restart does not stand it up. A
disable is never tentative. When an install is tentative at the time, the node records
the running tentative generation again without the disabled module, still tentative,
so making it permanent later cannot bring the disabled module back. A boot that leaves a module out and jails its
bytes records a generation the same way. The bytes stay in their identity directory. A request to install
the module again is refused until the unit restarts; after that, valis module install
with the same designation installs it from those bytes.
The module's code stays in the running image until that restart. I left it there because code cannot be removed from a running Lisp image reliably, and another module may still use its packages or call its functions. The node marks its image as diverged, and the owner view shows the module as disabled and still resident.
The reply. The verb prints the node's reply as it came: a token on the first line,
then system, designation, and retired, the count of the module's admissions taken
out of service.
| Token | What it means | What to do next |
|---|---|---|
disabled-still-resident |
The module's record is gone, and recorded no says so. Every admission the module was loaded under since the node started is out of service, and retired counts them. threads-left counts the module's threads still running, which the status line names with how to clear them. |
Nothing, or restart the unit to clear its code and any thread it left. |
disabled-still-recorded |
Every admission the module was loaded under is out of service, as retired counts, but its record could not be removed. The reason line says why, on one line. |
Put right what reason names and disable it again, or a restart stands the module up again. |
After disabled-still-recorded the verb also writes
valis: the module is out of service, and its record is kept, so a restart stands the module up again
on standard error.
Refusals. A refusal is an error beginning module disable refused, with the kind in
parentheses, and the verb prints it as valis: the node refused: and the node's text:
fenced: a newer instance of the node has taken over. Nothing changed.not-a-designation: the request names no module. Nothing changed.change-in-progress: another module change held the module change lock for more than 5 seconds. Nothing changed.tentative: the version you named is recorded only by the running generation, which is not yet permanent, as after an install or a supersede you have not blessed. The text names that generation. Reject it instead withvalis module reject N(Module lifetime), and restart the unit to return to the boot default, then disable the module if you still want it out. A supersede you have blessed is permanent and is not refused here, although its reply still readstentative until-restart, which is about the image, not the generation. Nothing changed.rejected: the version you named is recorded only by a generation you have already rejected. The text names that generation. A restart drops that version and returns the node to its boot default. Nothing changed.not-carried: the running image loaded no module from that designation's identity directory, or the node has no module source root. Nothing changed.retire-incomplete: a part of the module could not be released, and the text names each step still held. Whatever could be released was released, the record stays, so a restart stands the module up again, and the module's state shows the divergenceretire-incomplete. Restart the unit to clear what is held.
Exit statuses. 0 for either token; 1 for a fault on this machine or a reply this
client cannot read; 2 for a command line the verb does not take (printed with its
usage, before any key or pin is read), a refusal before your key was sent, a node with
no /modules/disable door, a node refusal, or any other error the node answers; 3 when
the node could not be reached or the session with it failed. As for an install, a reply
lost after the request was sent adds a line telling you to check valis module list.
Reading it back. valis module show SYSTEM shows the disabled module as long as its
code is loaded: its identity reads none, and its state reads recorded none,
loaded at its designation, divergence with disabled, disabled still-resident, and
diverged yes. After a restart it is gone from the listing.
The journal. Each disable is logged as the event module.disabled, naming the system,
the designation, how many admissions were retired, whether the record is kept, and how
many threads were left running, and
each refusal as module.disable-refused with its kind. A module that left threads
running is also named in a WARN line, retired <system>; threads left running: ....
From a 9P client. The door is /modules/disable in the owner's session. Write two
lines, disable and the designation, then read the reply, then clunk. The first read
runs the request and the fid keeps its outcome. A session that is not the owner's finds
no /modules.
⚠ Where this stands. ubik's modules panel writes this door too, after a confirmation
of its own. A version recorded only by a generation that is not yet permanent is
refused as tentative, and you reject that generation instead. Disabling a resident module, one compiled into the binary, is refused as
not-carried. Every admission the module was loaded under since the node started is
retired, including one whose every registration another owner has taken over and one
left holding nothing but threads or grants, and the reply's retired counts each of
them: a module superseded in place counts two.
15.3. Reading the module set
valis module list prints one tab-separated line per module the node lists: system,
recorded, loaded, divergence, and boot. valis module show SYSTEM prints the five files
the node holds for that module, identity, generation, files, explain, and
state, each name on its own line and its lines indented beneath it. Both read and
change nothing, take --node like every owner verb, and exit as install does, without
valis module jaillists what the node refused and kept, andvalis module parole
lets a module out of the jail, both as described under the module jail below.
The node serves these as read-only files under /modules/manifest/SYSTEM/, for the
owner alone. There is one directory for each module the running generation records,
resident modules included, and one for each module the image loaded with no record
naming it, as installed-unrecorded leaves one.
identity- the recorded designation,
resident, ornone. generation- the generation the manifest pinned when it was published. It describes; nothing checks it.
files- the record's relative paths, one a line.
explain- which two signatures admitted the module, and when:
publisher(the did:key its vouch is rooted at),owner-credential(per-module-grantorstanding-grant),grant(the grant's hash), andadmitted-at(a Unix time), eachnonewhen unknown, andpublisher unreadablefor a vouch that does not decode; orresident yes. state- the running image beside the record:
recorded;boot-default,yeswhen the boot default holds the same record, orno;loaded, the designation the module was loaded from,resident,unknown, orno;divergence, the reasons the image's divergence record holds, ornone;boot,ok,refused, ormanifest-unreadable;disabled,still-residentfor a module you disabled whose code
stays until a restart, or no; and diverged, yes or no.
diverged yes means the running image differs from the record the running generation
holds, or the boot left the module out; divergence and boot say which. What a
restart would stand up is what boot-default says. A boot refusal shows its kind
alone, and its text stays in the journal and on the status line, since it can name a
host, an address, a key, or a path. No file carries vouch text, grant text, or key
material beyond the publisher's did:key. A manifest the node cannot read is an error
beginning module manifest unreadable and naming the kind of failure, never an empty
listing.
/modules is the module set: which modules run, from which bytes, and why. One
module's settings are somewhere else, under /config/modules/SYSTEM/, which
valis config reads and changes with --module SYSTEM (see
Changing a setting). /modules sits in the owner's own
session and never under the node root, so no grant, bearer name, or delegated token
reaches it. It is one letter from /module/, the prefix designations are written
under, and the two never meet: no directory is ever made at /module/.
The module jail. Bytes the node refuses are kept, never loaded, and never deleted.
Bytes that land and are refused before any of their code runs, the modules of a
standing grant you backed out, and a module the node died in on three boots running
each go to the jail, the .jail directory under the module source root on the host,
unless the jail refuses them, which the node logs. That holds for a supersede as well
as an install: a change you offer for a module the node carries goes to the jail when
it is refused for another publisher's vouch, a missing or refused signature, or a dry
run that refused it before any of its code ran. A supersede refused because the
running image cannot take the change in place leaves the bytes where they landed, so
a restart can still take them. Bytes that were on the node before you asked are never
moved.
A backed-out module goes there at the next ordinary or evacuative boot, and the node
takes it out of the module manifest, names it as not running, and comes up serving the
rest. Each jail
entry is a directory of its own, 0700 and owned by the node's account, holding the
moved directory as bytes/ and a plain-text why file: the kind of jailing, the
designation, the system, the time, the reason, and for a back-out the grant hash and
the back-out line from the grant audit trail. Read it as root, or as the node's
account, on the host.
The node lists the jail for you as read-only files under /modules/jail/ENTRY/, one
directory per entry, oldest first, each holding designation, kind
(refused-landing, backed-out, or failed-boots), and jailed-at (a Unix time).
These carry no reason, path, grant hash, or key material, since a reason can name a
path; read the why file on the host for those. Only an owner session reads
/modules/jail, whether the command line or ubik's modules panel opens it.
valis module jail reads it for you from any machine that reaches the node as its
owner, and prints one tab-separated line per entry, oldest first: the designation, the
kind, and when it was jailed. A field the node does not know reads none. It reads
and changes nothing, takes --node like every owner verb, and exits as valis module
list does: 0 with the listing; 2 for a refusal (including a key the node gave no
owner view); 3 when the node could not be reached or the session with it failed; and 1
for a fault on this machine.
Paroling a module. Security is not the only reason a module is in the jail: a grant you backed out, a module the node died in three times, and a landing refused for a fault you have since put right are all operational. Let such a module out with
valis module parole /module/HEX
from any machine that reaches the node as its owner, with --node as every owner verb
takes it. The command writes parole and the designation, one a line, to
/modules/parole, which an owner session can also write itself. The bytes of the
newest jail entry for that designation that still holds them move back, by one rename
that never replaces anything, to the identity directory the designation names under
the module source root. The entry stays on the host with its why file and a new
parole file (paroled-at, a Unix time, and by owner), and it leaves
/modules/jail and valis module jail, which list only the entries that still hold
their bytes.
The reply is printed as the node sends it:
paroled designation /module/1220... kind backed-out jailed-at 1793001234 paroled-at 1793004567 loaded no
A parole loads, compiles, registers, and records nothing, and writes no generation.
The command adds a line on standard error saying so. To run the module, install it as you would any bytes already on the node, with
valis module install /module/HEX and no source: that goes through both signatures
and records a tentative generation, as every install does. A module jailed for a grant
you backed out is refused by that install, its bytes left where the parole put them,
until you file a fresh grant over it. Once paroled, a module is no longer jailed for a
rollback, and retention leaves its bytes alone until a generation recorded after the
parole names them and is itself dropped.
The node refuses a parole and moves nothing when the request is not parole and one
well-formed designation (not-a-designation), when no module source root is set or it
is refused (no-source-root), on an instance a newer one has taken over from
(fenced), while another module change holds the lock past five seconds
(change-in-progress), when the jail cannot be read (jail-unreadable), when no
entry names the designation (not-jailed), when no entry for it still holds its bytes
(bytes-gone), when its identity directory is already there (already-present), when
the module manifest cannot be read (manifest-unreadable) or still records the
identity, so the next boot would stand it up without an install (still-recorded),
when the jailed bytes are not the identity the designation names (not-the-identity),
and when the move itself fails (move-failed). Each refusal is an error beginning
module parole refused with the kind in parentheses; a caller who is not the owner is
refused without a kind. A fault after the bytes moved is an error beginning module
parole failed after the move, which says the bytes are back in their identity
directory and nothing of the module is loaded.
valis module parole exits 0 when the module is paroled; 2 for a refusal, a key the
node gave no owner view, or a malformed command line (no designation, more than one,
one not beginning /module/, or a flag that is not an owner flag), which prints
usage: valis module parole /module/HEX before anything is dialled; 3 when the node
could not be reached or the session with it failed; and 1 for a fault on this machine.
The journal logs each parole as module.paroled at warning level, naming the kind and
the designation, and each refusal as module.parole-refused, naming the kind. Neither
names a path.
15.4. Module lifetime
Each change to a node's module set is recorded as a numbered generation: an install, a supersede, a disable, a boot that jails a module, and a rollback. I kept the record so that a change that breaks the node costs one restart, and so that you can return to a set the node ran before. The design behind it is in the architecture document's module lifetime section; this one is about reading it and acting on it.
Reading the lifetime. valis module generations prints the node's lifetime lines
as the node serves them, then one tab-separated line per generation the node keeps,
oldest first: number, cause, status, ran, parent, recorded-at (a Unix time), binary,
overrides, and the modules, separated by spaces. A node that has had one module
installed and blessed, then restarted, reads like this (the designation is cut short
here):
$ valis --node NODE module generations booted 2 running 2 boot-default 2 trial none trial-attempted no last-failed-trial none promotion not-applicable degraded no stamp matches bless-required off default health-window-seconds 900 default restart-when-degraded off default 1 baseline permanent yes none 1793940000 unknown none gopher http 2 install permanent yes 1 1793951014 0.1.0 6d1f2a90c4e7 none gopher http /module/1220c70f...
| Line | What it says |
|---|---|
booted |
The generation this boot stood up, or none before a boot adopts one. |
running |
The generation the running image carries: the last change made in it, else the boot default. |
boot-default |
The generation a restart stands up. |
trial |
A rollback waiting for the next boot, or none. |
trial-attempted |
yes once that rollback's one boot has begun. |
last-failed-trial |
none, or the generation of the last rollback that failed, the Unix time it failed, and why. |
promotion |
What the health loop's last pass found: promoted, not-applicable (the image runs a permanent generation), waiting SECONDS left in the health window, or blocked and why: trial, rejected, resident-only, or bless-required. unchecked before the first pass, and always on a node with no supervisor, where the health loop does not run. |
degraded |
no, or each failing check by kind and subject. |
stamp |
How the booted generation's binary stamp compares with the running binary: matches, differs, or unknown. It reads unknown before a boot adopts a generation, and when either side has no version or no commit. |
bless-required, health-window-seconds, restart-when-degraded |
Each setting's value, then where it came from: operator for one you set, seeded for one the environment seeded, or default. |
On a generation's line, status is tentative, permanent, failed, or
rejected, and ran is yes once the node has run it. binary is the version and
commit of the binary that recorded it, each unknown when not known, or unknown
alone. overrides names withdrawn and downgrade when a rollback used them. A
resident module is listed by its system name, any other by its designation. The verb
reads and changes nothing, takes --node like every owner verb, and exits as valis
module list does.
Acting on generations. Four verbs change the lifetime, each through the node's
/modules/generation door:
valis --node NODE module bless N
valis --node NODE module reject N
valis --node NODE module rollback N [--override-withdrawn] [--override-downgrade]
valis --node NODE module repair
bless Nmakes generation N, the running tentative generation, permanent and the boot default. Bless a rollback trial after its boot and it is kept the same way.reject Nkeeps a tentative generation from ever becoming permanent. When the image runs it, the image is marked diverged, and a restart returns the node to its boot default. Rejecting a rollback trial before its boot cancels it.rollback Nrecords a return to generation N's module set as the next-boot trial, then asks the host agent to restart the node. Nothing in the running image changes first.repairrejects the running generation when it is tentative, clears any pending trial, and asks the host agent to restart the node into its boot default.
Each verb prints the node's reply as it came, a token on the first line and key value lines after it:
| Token | Lines | What it means |
|---|---|---|
blessed |
generation |
The generation is permanent and the boot default. |
rejected |
generation, diverged |
The generation is rejected; diverged yes when the image runs it. |
rollback-committed |
generation (the trial recorded), target, overrides, restart |
The trial is on disk. restart is what the host agent answered. |
repair-committed |
rejected (a generation, or none), restart |
Whatever needed writing is on disk, which may be nothing. restart is what the host agent answered. |
A rollback or a repair stays committed whatever the restart line says. When it
begins error, the restart may not have happened: if the node is still up, restart the
unit yourself; the next boot stands up what was committed.
Refusals. A refusal is an error beginning module generation refused, with the kind
in parentheses, and the verb prints it as valis: the node refused: and the node's
text. A refused request changed nothing and asked for no restart.
not-a-request: the lines were not one of the four acts. The command line refuses most of these before anything is sent.fenced: a newer instance of the node has taken over.absent: the node holds no generation by that number, because it never recorded one or because retention has already dropped it. A rollback to such a number is refused this way too.not-running: you can bless only the generation the image runs.not-tentative: the generation is already permanent, failed, or rejected.change-in-progress: another module change held the module change lock for more than 5 seconds.trial-in-progress: a rollback trial's one boot is under way; repair first.no-durable-store: the node keeps no lifetime record, as on the plain file backend, so a bless, a reject, or a rollback has nothing to act on. A repair on such a node is not refused: it records nothing, asks only for the restart, and answersrepair-committedwithrejected none.- For a rollback to a generation the node holds, every reason the node found, the first
giving the kind:
not-retained(the node still lists the generation, but it falls outside what retention keeps, and the pass that drops it has not run yet),never-ran,failed,rejected,bytes-missingorjailed(with the designation),not-admitted(with the designation: a signature no longer admits it now),withdrawn(with the designation, and whether its publisher or you withdrew it), anddowngrade-undeclared(with the system). Only the last two can be overridden, with--override-withdrawnand--override-downgrade.
Exit statuses. 0 for any of the four tokens; 1 for a fault on this machine or a reply
this client cannot read; 2 for a command line the verb does not take (printed with its
usage, before any key or pin is read), a refusal before your key was sent, a node with
no /modules/generation door, a node refusal, or any other error the node answers; 3
when the node could not be reached or the session with it failed. A reply lost after
the request was sent adds a line telling you to check valis module list and
valis module generations before you ask again.
The settings. Three node settings govern the lifetime, each read on every pass of the health loop:
valis --node NODE config set module.health-window-seconds 1800 # 60 to 86,400; 900 by default
valis --node NODE config set module.bless-required true # off by default
valis --node NODE config set module.restart-when-degraded true # off by default
module.health-window-seconds is how long a tentative generation must pass every
health check before the node makes it permanent. With module.bless-required on,
nothing becomes permanent until you bless it. With module.restart-when-degraded on,
the first degraded report of each episode repairs the node, as valis module repair
would.
A fourth, module.withdrawn, lists the module versions you treat as withdrawn, by
designation, separated by commas. A rollback to a generation holding one is refused as
withdrawn unless you pass --override-withdrawn. Withdrawal is not revocation: a
listed version that is running keeps running, and a restart admits it again.
A rollback, and how its boot ends. A rollback's trial gets one boot. The node records
the attempt before any module of the trial loads, and that boot jails nothing. When
the boot passes its health gate a minute after it reports ready, the trial becomes
permanent and the boot default: trial reads none and boot-default reads the
trial's number, and the journal logs module.trial-passed. Any other end returns the
next boot to the boot default, and the node's status line says why, as in
the rollback to generation 7 failed (its boot died standing up SYSTEM), and the node
returned to generation 5. last-failed-trial then reads the trial, the time, and the
reason, and the journal logs module.trial-failed at ERROR with the reason:
died-in-module (naming the system), stopped-before-verdict,
died-unattributed, gate-failed, passed-unrecorded, boot-floor,
boot-record-unreadable, or no-cause-recorded. After passed-unrecorded or
stopped-before-verdict, roll back again to retry the trial.
A degraded node. When two passes of the health loop in a row find something the
running generation bound no longer served, the journal logs module.degraded at
ERROR, naming each failing check, and degraded on the lifetime names them too. It
logs module.recovered when a pass finds every check passing. Run valis module
repair to return to the boot default, or set module.restart-when-degraded.
Retention. The node keeps five generations or sixty days of them, whichever keeps
more, and always keeps the boot default, the running generation, the pending trial,
and the most recent permanent generation. After the boot default moves, the next pass
of the health loop drops the rest from the lifetime record and deletes the identity
directory of each module only a dropped generation named. It never deletes bytes a
kept generation, the running image, or its booted generation names, nor anything in
.jail, a link, or a directory the running image loaded a system from. The journal
logs module.retention-pruned naming each designation deleted, and
module.retention-failed at ERROR for anything it could not do.
The binary stamp. A release that passes its trial records a module generation under
the new binary, so differs means a binary changed some other way, such as by a host
act. A boot whose
booted generation was recorded under a different binary stands every module up as it
would otherwise and logs module.binary-stamp-mismatch at WARN, naming both stamps;
stamp on the lifetime reads differs.
The journal. Besides those above, the node logs module.generation-recorded for
each generation, module.generation-blessed, module.generation-promoted, and
module.generation-rejected, module.rollback-committed with one
module.rollback-override at WARN for each override used, and module.repair-requested
at WARN, naming whether you or the health loop asked. None names a path.
journalctl -u valis.service | grep -E '"event":"module\.(generation|trial|rollback|repair|degraded|recovered|retention|binary)'
From a 9P client. In the owner's session, /modules/generations/N/ holds the files
cause, status, ran, parent, recorded-at, binary, overrides, and
modules, and /modules/lifetime the lines above, all read-only and rendered when
opened. To act, write to /modules/generation one item per line, bless and N,
reject and N, rollback and N followed by any of override-withdrawn and
override-downgrade, or repair, then read the reply, then clunk. N is 1 to 20
decimal digits with no sign. The first read runs the request and the fid keeps its
outcome. A session that is not the owner's finds no /modules.
⚠ Where this stands.
- ubik's modules panel does not yet read the lifetime or offer these acts.
- Retention deletes the bytes a dropped generation named even when they were placed again later under the same identity.
- Retention follows no link only while nothing else writes under the module source root during a deletion.
- Bytes of the generation the running image booted stay on disk once retention drops its row, since nothing names them for a later pass.
- The node holds the fact that retention is due in memory, so a restart between a boot-default move and the next pass leaves the pruning for the next move.
:53was read on one scratch guest only across a restart, a rollback, and the boots that blame a module, reach the floor, or end a failed trial; What a recovery costs gives the readings and what they leave out.- A restart you ask for while a module is still settling is not counted against it:
the next boot logs
module.boot-stopped-before-verdictat INFO and counts nothing. A stop that arrives while a module loads or starts still counts against that module, and so does a stop the node never sees in order, such asSIGKILLor a stop before the node reports ready.
15.4.1. What a recovery costs
A rollback and a repair both take effect by restarting the node, and :53 does not
answer while the node restarts. A module the node dies in as it boots costs more, since
each death is a restart of its own. I measured both so you can weigh a rollback before
you ask for one. The readings were taken on 2026-11-06 on a scratch guest
condensed from genesis, running commit ca53289e9af2 under Debian 13
and systemd 257, and serving a trivial test zone. It serves no real names. Read the
numbers as what that guest did, never as a guarantee.
How the gap was read. A probe on the guest's host sent dig queries for the zone's
SOA over UDP and over TCP, each waiting one second for its answer: about seven a
second on each transport while the node answered. A dark interval below is the span
from the first failed query to the last. The figure in parentheses runs from the last
answer before the outage to the first answer after it, an upper bound. A dark span
shorter than about a second cannot be told from none. To show that the probe registers
an outage at all, I stopped the unit, waited 10 seconds, and started it, with the probe running: it read UDP
dark for 11.358 seconds and TCP for 12.604.
A restart. Three plain restarts, systemctl restart valis on the guest, and three
owner restarts, valis --node NODE restart, which asks the host agent for the same
restart a rollback and a repair ask for. In seconds:
| Restart | UDP :53 dark |
TCP :53 dark |
Command to ready |
|---|---|---|---|
| plain | 1.13 (2.77) | 2.28 (2.90) | 3.37 |
| plain | 1.14 (2.63) | 2.29 (2.99) | 3.15 |
| plain | 1.14 (2.90) | 2.31 (2.99) | 3.46 |
| owner | 1.13 (2.70) | 2.29 (2.91) | 3.80 |
| owner | 1.13 (2.52) | 2.29 (2.86) | 3.50 |
| owner | 2.27 (3.00) | 2.28 (2.88) | 3.74 |
Command to ready runs from the command to the journal's Started valis.service,
when the node reported ready. The longest dark span was 2.27 seconds over UDP and 2.31
over TCP, and no upper bound in the table exceeded 3.00 seconds, for either kind of restart. The
fulcrum parent that binds :53 was replaced along with the valis resident on every
restart, so nothing holds the socket across one: :53 is dark from the unit's stop
to its start.
A rollback. Two rollbacks were timed, one with --override-downgrade and one with
both overrides. The first read UDP dark for 1.13 seconds (2.89) and TCP for 2.28
(2.93); the verb returned 0.9 seconds after it was issued, and the node answered
valis module list again 5.7 seconds after it was issued. The second read UDP 1.13
(2.79) and TCP 2.29 (3.07). Each trial passed its health gate.
A downgrade. A test module declaring data format 1 was replaced by one writing format
2: disabled, the node restarted, and the format 2 version installed fresh, since a
supersede that changes the module's grouping form, where its format is declared, is
refused as restart-required. A
rollback to the format 1 generation was refused as downgrade-undeclared, naming the
system, and nothing was recorded. With --override-downgrade it committed, the
journal logged module.rollback-override at WARN naming the system, the trial passed
its health gate, and the format 1 version answered. :53 read UDP dark for 1.134
seconds (2.907) and TCP for 2.283 (3.170).
A repair asks the host agent for the same restart. No repair was timed.
Promotion. With module.health-window-seconds at 120, the probe ran through a
supersede and the whole window after it: 1,251 queries on each transport, none
failed. The node made the generation permanent 130 seconds after recording it.
What systemd applied there. systemctl show valis on the guest read:
Type=notify,Restart=on-failure, andRestartUSec=1s, withRestartSteps=10andRestartMaxDelaySec=5min, so the delay grows when the unit keeps failing.TimeoutStartUSecandTimeoutStopUSecof1min 30s, andWatchdogUSec=30s.RestartPreventExitStatus=78.StartLimitIntervalUSec=0, so no start limit applies, andStartLimitBurst=5is inert beside it.
These set how soon a boot that dies is tried again, and so how long a module the node keeps dying in holds the node down before its third death leaves it out. The readings below were taken with test modules built to end the node's process while it booted or soon after, beside a good module that never failed.
A module the boot keeps dying in. A test module that ends the process while it loads
was installed and blessed, and the node restarted with it armed. The journal logged
module.boot-died-in on three boots running, then module.boot-left-out and
module.jailed, and the fourth boot stood up the set without it and recorded a
generation whose cause is jailed. valis module jail listed the module, and the good
module answered. Over UDP :53 was dark for 1.135, 6.940, and 4.665 seconds, and
answered between those spans, early in the second and third boots, before the load
ended the process. Over TCP it was dark for 9.812 and 5.060 seconds. From the first
failed query to the last was about 15.3 seconds, and :53 answered on the jail boot,
with no failed query after it.
Boots with no module to blame. A test module that ends the process about 20 seconds
after a boot, once loading is over, was installed and blessed, and the node restarted
with it armed. Three boots died, no module.boot-died-in line appeared, and the
journal logged module.boot-floor with a count of 3 and 2 modules. The fourth boot
stood up only the resident set: both installed modules read as not running in the
module list and on the status line. Over UDP :53 was dark for 1.133, 3.410, 3.535,
and 5.670 seconds, and over TCP for 2.286, 3.429, 3.889, and 5.167, answering while
each boot lived. On the floor boot it answered 487 UDP and 488 TCP queries over about
65 seconds, with 0 failures on either.
A failed trial. A rollback to the generation holding the module the boot dies in
is refused as jailed, with no override able to clear it, until you parole the
module's bytes. After a parole, a rollback to it committed (the --override-downgrade
passed with it was not used), and its trial boot died once. The journal logged module.trial-failed with the
reason died-in-module, naming the system, the next boot returned to the boot
default, and the status line read the rollback to generation 14 failed (its boot died
standing up valis-scratch-load-breaker), and the node returned to generation 12. Two
such rollbacks were run, and the probe ran through the second only. Over UDP :53 was
dark for 1.134 and then 3.526 seconds, and over TCP for one span of 5.769 seconds; it
answered on the return boot, with no failed query after it.
What was not measured. :53 during the first of the two failed trials, which ran
with no probe. The good module during the three boots that died loading the bad one,
since no query was sent to it then. A rollback refused as not-retained, which the
guest never produced. Nor was a larger zone's load time read, or a module that hangs
or crashes as it loads. The architecture document's module lifetime validation map names the
tests behind each behaviour.
15.5. Checks on the module path, each shown able to fail
I believe a check on the module path only after I have watched it fail. A check that cannot fail reports the same clean result whether the node is sound or the check is broken, so each row below names what was broken to turn it red, beside what it reads when nothing is broken. A row with an empty last column is unproven.
Run a suite with scripts/test-run.sh suites valis/tests/SUITE, where SUITE is the
test file's name without .lisp, or run them all with make test. The other readings
were taken once, on 2026-11-01, on two test guests running commit ad7e9af575dc: the
reached guest, which a second machine on the same network drove over the owner port,
and the genesis guest, condensed from genesis by make deploy-node. The readings from a
serving node were taken once, on 2026-11-05, on a node serving real names and running
commit 53b9feda003c. A second machine drove it as its owner over the owner port with
the valis command line, and the readings of its process, its descriptors, and its
network namespace were taken on its host, changing nothing. Where a row names the run's
grading checks (the scripts that graded each reading) as red, each was fed a reading with the failure planted and failed: that
shows the check can fail, not that the node was read correctly. The suite repeats none
of these readings; to repeat one, you take it again.
| What the node guarantees | Check | What it read | Broken to show it red |
|---|---|---|---|
| An install is admitted only on the module's genuine identity and both signatures, the publisher's and the owner's. The same module with one byte changed is refused before the loader runs, including when a second machine asks. | byte-changed-in-place-refused-as-not-its-identity in tests/module-install-test.lisp; admit-module-with-only-a-vouch-refused in tests/module-two-signature-test.lisp; from a second machine on the same network, valis module install against the reached guest over the owner port. |
Both suites pass. From the second machine: installed-serving, the module recorded and loaded and answering on its port; the one-byte copy refused as vouch-refused, with the module list and the module's load counter unchanged. |
With the check that a directory is named for its identity removed, the one-byte test failed. With either signature alone accepted, fifteen tests failed, the vouch-only test among them. From the second machine, the first install was refused for want of an owner signature while the node held no active standing grant; that red covers the two signatures only, and the one-byte refusal from the second machine has no red of its own. |
| Revoking the owner's signature alone, or the publisher's, refuses the next install of that module. | revoked-owner-signature-refuses-next-admission and publisher-list-naming-the-vouch-refuses-next-admission in tests/module-vouch-revocation-test.lisp. |
The suite passes. | With revocation made to do nothing, the owner test failed. With the publisher list check forced to pass, both publisher list tests failed. |
| The durable module manifest records each installed module's real identity, never the empty placeholder the resident pair carries. | admission-writes-record-with-identity in tests/module-install-test.lisp; size-is-in-the-framing in tests/module-identity-test.lisp. |
module-install-test: 306 executed, 306 passed, at ad7e9af575dc. module-identity-test passes. |
With an install made to record the placeholder in place of the identity, the assertion that the record carries the identity failed. With the file size left out of the identity's framing, the framing test failed. |
| A node built from the deployment artifacts with no hand edit comes up reachable over the owner path with no development listener, and its manifest's relative paths resolve under a module source root that exists. | On the genesis guest. template-carries-every-boot-requirement and listener-port-stays-out-of-template-and-site-facts in tests/deployment-template-test.lisp; restart-readmits-from-identity-directory in tests/module-source-root-test.lisp. |
On the genesis guest: the build's acceptance run passed 25 of 25; the source root existed, owned by the run account; no development listener in the environment file, the settings, the boot record, or the listening sockets; the owner client listed the modules over the owner port after the pin and the open; all three relative paths of an installed module were found under the root. Both suites pass. | With the module source root left empty in the site file, the deployment preflight refused and named it, and changed nothing. With the source root line removed from the template, or a development listener port added to it, a template test failed. With re-admission resolving against the bare root, the restart test failed. |
A signed self-update of a module the node carries takes effect with :53 never rebuilt and the inherited descriptors kept. An unsigned change is refused, and the running module is undisturbed. |
signed-change-takes-effect-at-the-same-port and unsigned-change-leaves-the-port-serving-old in tests/module-supersede-test.lisp; on the genesis guest, a supersede while queries ran against :53. |
The suite passes. On the genesis guest: superseded, and the new version answered. The process, its boot identity, its inherited descriptors and sockets, the listeners, the director's steering, and the zone serial were the same before and after. Forty-two queries each over UDP and TCP :53 had no failure. The unsigned copy was refused and the load counter stayed byte for byte the same. |
With the update done as retire, install, and attach, the signed test failed on the port, the announcement, and the adapter's identity. With the unsigned version signed, both unsigned tests failed. The guest readings have no red of their own. |
| After a restart the node admits its installed modules again from the manifest and the stored signatures with the owner absent. A manifest that mixes the resident pair with installed records reassembles, and a revoked module fails closed. | revoked-module-fails-restart-readmission-closed in tests/migration-end-to-end-test.lisp; coincident-boot-stands-the-set-up in tests/boot-readmission-test.lisp; condensation-skips-resident-records-in-mixed-manifest in tests/module-manifest-test.lisp; restarts of both guests with no owner session open. |
The suites pass. On both guests the journal recorded each installed module re-admitted after every restart, and the module list showed each one recorded and loaded. | With the revocation check bypassed, both revoked tests failed with the module loaded. With re-admission confined to an evacuating boot, the boot test failed. With the resident records not skipped, reassembly refused the mixed manifest, naming gopher. |
| A signed module installs into a running node that serves real names, from a second machine over the owner port, and serves at once with no restart. Its port answers inside the node's network namespace and never on the node's public address. | On the serving node: from the second machine, valis module install with --dist, then valis module list and show; a query on the module's port inside the node's network namespace; from the second machine, a probe of that port on the node's public address beside probes of :53 and of a closed port; the boot identity from the owner view before and after. a-module-lives-its-whole-life-on-a-serving-node in tests/module-lifecycle-test.lisp. |
installed-serving, with one adapter line naming the module's port. The listing showed the module recorded and loaded at that designation with divergence none, and show named the publisher and the standing grant as the owner credential. The port answered the first version inside the namespace. From the second machine it read no answer, while :53 completed and the closed port read no answer in the same step. The boot identity was the same before and after. The suite passes. |
With the edge domain the lifecycle test declares dropped, its check that the node serves real names failed. On a test guest, with the module's port admitted to the firewall, the public probe completed and the check failed. The run's grading checks failed for each of these readings. |
| The node lists each owner session under the transport key the session presented. | On the serving node, from the second machine after the install: valis session list. the-owner-lists-sessions-including-its-own in tests/owner-reads-test.lisp. |
A live session under the transport DID the owner view names as the owner, beside an older anonymous session. Unproven: that this is the session that installed the module, since the install's session had closed and the listing names the session that asked for it; and that this DID is the one the second machine's enrolled key carries, since the two were not compared. The suite passes. | With the verb made to drop the first session from its output, the test failed. The run's grading check failed when fed another DID, and still passed with the anonymous line removed, so it reads the DID's own line. |
A signed change supersedes a module a node serving real names runs, with :53 never rebuilt and the inherited descriptors kept, and the reply names each item of the old version still alive. |
On the serving node: from the second machine, valis module install of the second version, then valis module list; the module's port inside the namespace before and after; on the host, before and after, the main process id, the restart count, the resident's process id and command line, its inherited socket descriptors, the namespace's listening sockets by inode, and the zone serial; UDP and TCP queries to :53 from the second machine for the length of the change. a-supersede-reply-names-each-residue-item and a-long-residue-is-capped in tests/modules-over-the-wire-test.lisp; the lifecycle test. |
superseded, previous the first version, recorded yes, tentative until-restart, and residue 1 with one residue-code line naming the handler file, its protocol constructor, and a closure made in it. The listing showed the second version recorded and loaded, with divergence superseded, and the port answered the second version before and after. Every snapshot read the same before and after. Of 111 UDP and 106 TCP queries spanning the change, none was missed. The suites pass. |
With the supersede made a retire and attach, the lifecycle test failed on the adapter's identity. With the reply printing the old version's absolute path, the residue test failed; with its 32-line cap dropped, the cap test failed. On a test guest, a restart between snapshots made them differ, and the query loop aimed at nothing timed out seven times in seven. The run's grading checks failed for each of these readings. |
| An unsigned change to a module a node serving real names runs is refused, the module keeps serving, and the refused bytes go to the jail. | On the serving node, from the second machine: valis module install of a one-byte copy carrying no publisher vouch; the module's load counter before and after; valis module jail. unsigned-change-leaves-the-port-serving-old in tests/module-supersede-test.lisp; an-unsigned-change-is-jailed-and-loads-nothing in tests/module-change-test.lisp; a-key-without-the-owner-view-reads-no-jail in tests/owner-reads-test.lisp. |
module install refused (vouch-refused), exit 2. The load counter stayed byte for byte the same, the port kept answering the second version, and :53 missed no query. The jail listed the copy's designation as refused-landing, and the journal logged module.jailed for it. The suites pass. |
With the unsigned version signed, both unsigned tests in the supersede suite failed. With the jail reader answering an empty listing, rather than a refusal, to a key without the owner view, the owner-reads test failed. The run's grading checks failed for each of these readings. |
| The owner disables a module a node serving real names runs: it stops serving at once, its record goes, and the node reports it disabled and still resident. | On the serving node, from the second machine: valis module disable, then valis module show; the module's port inside the namespace; :53 queries throughout. The lifecycle test. |
disabled-still-resident, retired 1, from a build that counted only the running version's admission (a node at this commit reads retired 2), recorded no, threads-left 0. show read recorded none, disabled still-resident, and diverged yes. The port stopped answering, and :53 missed no query. The journal logged module.disabled with one admission retired and the record not kept. The suite passes. |
With the disable made to skip the retirement, the lifecycle test's checks that the port went silent failed, while retired still read 2. The run's grading checks failed for each of these readings. |
After one restart of a node serving real names, the module disabled before it is gone, the jail still holds the refused copy, and :53 answers. |
On the serving node, from the second machine: valis restart, then the boot identity, valis module list, valis module jail, and a UDP and a TCP query of :53. |
The boot identity changed. The listing no longer named the module. The jail still listed the unsigned copy, and :53 answered over UDP and TCP with the zone serial unchanged. |
On a test guest, the same readings across a restart showed the boot identity change. The lifecycle test restarts inside one process, which keeps the module's system loaded, so it shows the record gone but cannot show the module unlisted; only a restart of the unit shows that. The run's grading checks failed for each of these readings. |
| Each guarantee here is shown able to go red before it is believed green. | This table, by eye: every row above names what was broken to turn its check red. | No row above has an empty last column. The two readings with no red, both in the session row, are written there as unproven. | None of its own: an empty last column above would show it. |
What the node does not do yet. These readings also show where the node falls short of where it is going.
- No reading above covers a boot that leaves out a module it died in.
- The session listing cannot tie an install to the session that made it. Each verb opens a session of its own and closes it when it ends, so a listing taken after an install names its own session under the same key, never the install's. On the serving node that showed the key, not the install's session, and the DID it listed was not compared with the enrolled key on the second machine.
- A connect that does not complete in time exits 3, as a node not reached, in the suite and on a test guest. Nothing timed out on the serving node, so it has no such reading.
- The refusal of a module its publisher vouched for and the owner never granted is shown in the suite and on a test guest, not on the serving node: showing it there means withdrawing that node's standing grant, which cannot be reinstated, only replaced by a new one.
- The journal records a jailing and a disable as events,
module.jailedandmodule.disabled, but an install or a supersede only as a line of text. On the serving node the events agreed with the jailing and the disable; the install and the supersede rest on the node's replies and the readings inside its namespace. - No door or verb lists the revocations a node holds, so you cannot read back what a publisher list you filed now refuses.
- A supersede names what the old version left alive and does not clear it. On the serving node the reply named one item, the old version's protocol constructor and the closure it made, and only a restart clears it.
- No node has yet served an owner port other than 564.
- The seed zone the genesis guest was built with names a different guest's address in its A records. The serial readings above are unaffected.
16. Backup and restore
The backup-critical set is irreplaceable: the owner Ed25519 seed (0600), the
store head, the store blocks the head's root tree reaches, the vouch and revocation
stores, the grant audit trail beside them (see
Managing standing grants), the owner keyring of enrolled device keys (see
Enrolling and retiring an owner device key), the node's configuration (its own settings and every module's, under
<data-root>/config/), and the ACME account and leaf keys. Losing the owner seed is
losing the node's sovereign identity; there is no re-mint.
Run both verbs as the run account, never as root. The configuration directory is
private to the account that owns it, and a backup reads the grant audit trail only
when the account running it owns the file, so a root-run backup is refused at
whichever of the two it reaches first, before anything is written. A restore
run as any account other than the one owning --state-dir (or, for a directory not
made yet, the nearest directory above it) is refused before any write, with exit
code 2, because the node could not open the files it would land. A restore run as
root into a directory that does not exist yet is refused the same way, since
whatever it made would belong to root. The same holds for --acme-store-dir.
Name the binary by its full path, since sudo searches only its own secure_path. A
restore needs no XDG_DATA_HOME: --state-dir names its target.
The valis binary carries the backup and restore verbs. Each seals or opens a single
encrypted, self-contained artifact, not a copy of the whole StateDirectory. The
passphrase is read interactively (or piped on standard input for an unattended run);
it is never a command-line flag, so it never lands in shell history or /proc.
Back up. Seal the backup-critical set into one artifact:
sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis --backup --out /path/to/valis-backup.sealed # prompts: Backup passphrase:The artifact is small. It carries the node's configuration as written, so it carries what that configuration names: absolute paths such as the ACME store (
acme.store-path,/var/lib/valis/acme/on a standard install) and the module source root, and the node's own domains and addresses. It carries no database password: the configuration never holds one. A memory-hard key derivation (argon2id) turns the passphrase into the key, and authenticated encryption (ChaChaPoly) seals it: a tampered artifact is rejected rather than silently restored. Treat it as you would a private-key store, and store the artifact and the passphrase separately. An owner seed that is absent, or that is not exactly 32 octets, stops the backup with exit 1 before anything is sealed. A backup can also be driven remotely by the authenticated owner over the/backup/ctl9P door (backup <passphrase> [name];statusreports the last outcome). The passphrase rides one ctl line over the sealed owner session (never cleartext on the wire, never shell history) and the artifact always lands in the node's backup directory,<data-root>/backups/(/var/lib/valis/valis/backups/under the shipped unit), under NAME or, with no name, a timestamped bundle. NAME is a bare file name: a path, a name starting with a dot, or a name already present there is refused with anerror destinationreply, so no session can aim a sealed copy of the seed elsewhere. To write the artifact to a path of your choosing, runvalis --backup --out <path>on the host. A restore is deliberately not served over 9P: it requires the unit down and a pristine state directory, so it staysvalis --restoreon the host.Restore. Open the artifact into a pristine state directory. When the state directory does not exist yet, make it as the run account first; starting the unit once also has systemd make it:
sudo install -d -o valis -g valis -m 0700 /var/lib/valis sudo -u valis /opt/valis/bin/valis --restore --in /path/to/valis-backup.sealed --state-dir /var/lib/valis # prompts: Restore passphrase:The restore lands the configuration first, then places the ACME store where that configuration says the node keeps it. The ACME store must lie inside
--state-dir: one the configuration places elsewhere is refused before any write, with exit code 2, so a restore never overwrites certificates outside its target. To restore somewhere else, such as a scratch directory for a proof, name where the ACME store lands with--acme-store-dir:sudo -u valis mkdir -m 0700 /tmp/restore-proof sudo -u valis /opt/valis/bin/valis --restore --in /path/to/valis-backup.sealed \ --state-dir /tmp/restore-proof --acme-store-dir /tmp/restore-proof/acmeThe restored configuration still names the original ACME store, so a node you boot from a proof directory looks for its certificates there; the proof checks the keys under
--acme-store-dir. A non-pristine target (one that already holds valis state, configuration, or an ACME store where this restore lands one) is refused up front, before any write, with exit code 2; a wrong passphrase or corrupt artifact fails cleanly with a single message and exit code 1. The normal resident boot then condenses the durable namespace from the restored head: restore is the first-deploy path run from a saved head instead of from genesis, and a faithful restore reproduces the origin's durable identity byte-for-byte. Deploy, evacuate, and restore are one condense mechanism, see the contract's deploy/evacuate/restore section.The keyring on restore. A restore lands the owner keyring with its retirements, so a key retired before the backup was taken stays retired, and it prints every key it restores, live or retired, for you to review:
valis: restored owner key did:key:z6Mk... label=laptop live
A restore cannot know of a key retired after its backup was taken. Retire any such key again once the restored node is up. A backup taken before the node kept a keyring restores the seed alone, with a warning saying the node admits only its own key until you enrol your devices again. A binary older than the keyring refuses a backup that carries one, before writing anything. A keyring the backup cannot read whole, or one written for another owner, stops the backup before anything is written, and one a restore cannot land rolls the whole restore back.
The grant audit trail on restore. Backups carry the trail from container format 6. A restore lands it right after the revocation store and appends one
restoredentry, so the trail shows when the node came back. A backup in an older format restores with this warning, and the restored trail starts at the restore:valis: WARNING: this backup is in container format 5, which cannot carry the grant audit trail. The restored node's trail starts at the restore: when each grant was filed and reviewed before it is not known.
Each active standing grant then has no known filing and no review, so each is due a review at once: review them again (see Managing standing grants). A target that already holds a trail is refused as not pristine unless the backup carries one, which then replaces it whole. When the
restoredentry cannot be written, the restore stands and a warning beginningthe restore stands, but its restored entry was not written to the grant audit trailsays so. A backup that cannot read the trail is refused before anything is written.
The restore-proven backup is the go-public gate, and nothing else enforces it. Do not cut a node over to a public address until a restore has been proven: restore the artifact into a scratch state directory and confirm the three properties a good restore demonstrates (the owner axis resolves, the store generation matches the source, the TLS keys load). Those properties, and why a silently-wrong restore is treated as worse than an obvious failure, are set out in backup-and-observability.org.
17. Restart, stop, and teardown
Restart is just re-running bring-up: it is idempotent. The single-writer instance fence makes a re-claim of the current generation a no-op, so recovery is safe to repeat.
systemctl restart valis.service- A restart spends a pending release trial's boot. On a node that keeps releases, the next start launches a trial that apply or a rollback recorded, whatever caused the start (see Releasing a new binary to a running node).
Restart as the owner with
valis restart, when you hold the owner key, or when the node has told you a restart clears threads a retired module left running. The node asksfulcrumto have systemd restartvalis.service, so it is the same stop and start assystemctl restart, with the same few seconds without service. Run it from your own machine and name the node:valis --node NODE restart- It waits for the new boot. After it asks, the verb waits, 120 seconds by default
(
--wait SECONDSsets the bound), until the node answers again over the same pin from a boot identity other than the one it had when asked. It then printsvalis: the node is back, bootwith the new identity, and exits 0. The owner port keeps its address across the restart, so the wait needs no shell on the host. - Every owner session ends. The restart ends every session the node held, including ubik's. Each valis verb opens a session of its own, so the next one you run attaches to the new boot. ubik attaches again on its own once the node is back, checking the pin as it does, and every panel reads the node again. When you restart the node from ubik's restart panel, the panel shows the node's boot identity before and after; Managing a node from ubik covers the panels.
- Exit statuses. 0: the node came back from a new boot, or you passed
--wait 0and the restart was accepted. 2: the node refused, or the client refused before asking (for example, for a key the node has not enrolled), so nothing was asked. 3: the node could not be reached to ask, or did not come back within the wait; the message says which. - A refusal says why. A node running without a supervisor that can restart it, such
as a development image or
--run, says so and tells you to restart it the way it was started. Whenfulcrumis too old to serve the restart request, the refusal says it cannot restart the node: restart the service on the host withsystemctl restart valis.service.
- It waits for the new boot. After it asks, the verb waits, 120 seconds by default
(
Restart as the owner from the node's host, the break-glass route, when you cannot use the route from your machine (the owner port closed, the pin or the device key lost, or the node not answering on the owner port). Run the verb inside the resident's namespace as the run account (see the break-glass route), and confirm from the host:
systemctl show -p MainPID -p ExecMainStartTimestamp valis.service sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \ /opt/valis/bin/valis restart --wait 0 \ --keyfile /var/lib/valis/valis/keyfile \ --endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)" systemctl show -p MainPID -p ExecMainStartTimestamp valis.service- Pass
--wait 0from inside the namespace. A restart replaces the namespace and draws a new fabric port, so a verb entered into the old namespace, or pinned to the old port with--endpoint, can never see the new boot and would wait out the whole bound. With--wait 0the verb asks, prints the boot the node answered from, and exits. - Confirm on the host.
MainPIDisfulcrum, which every restart replaces, so a differentMainPIDand a laterExecMainStartTimestampafter the restart than before it say the node was restarted. Run any owner verb again through a freshnsenterto see it answering. Where the node's owner port is open and you name it with--endpoint, the address survives the restart and the verb's own wait works, with no root shell at all.
- Pass
Stop drives an ordered teardown. On
SIGTERMthe resident unwinds in reverse bring-up order (long-lived active modules drain first, then edge, then fabric, then listener);fulcrum(theMainPID) removes thesk_lookuplink and tears the netns down.KillMode=mixedlets fulcrum drive that ordered pair-teardown rather than systemd SIGTERMing both at once.systemctl stop valis.service- This stop does not depend on the stop notice.
systemctl stopsignalsfulcrumfirst, andfulcrumends the run on that request and reaps the resident. The resident gives its stop notice as it does on any orderly stop, but the outcome does not depend on it, so a warning about the notice during this stop needs no action. The same holds for an owner restart, which goes throughfulcrumand systemd.
- This stop does not depend on the stop notice.
- A signal sent to the resident's own process is a stop the resident begins itself:
SIGTERMorSIGINTdelivered to the resident directly, for examplekillwith its PID. The bullets below cover this stop only.- The resident tells fulcrum the stop is intended. Before it stops anything, the
resident sends
fulcruma stop notice over its control session, and it closes that session last. Once the session ends,fulcrumwaits up to 70 seconds for the resident to exit and exits with the resident's status; a resident still running at 70 seconds is killed andfulcrumexits 1. The journal showsthe host agent was told this stop is intendedat info level. - A warning about the notice means the stop can read as a failure. An older
fulcrumhas no file to take the notice, and the resident warnsthe host agent serves no stopping file; updatefulcrumto clear it. Any other warning about the notice (a request still held the session, the session had ended, orfulcrumrefused or did not acknowledge it) means the notice did not land this once. Either way the stop itself carries on, andfulcrumgives the resident only 2 seconds from the session's end: a resident that exits in that time has its status passed on, and one still running makesfulcrumexit 1.
- The resident tells fulcrum the stop is intended. Before it stops anything, the
resident sends
- A superseded writer fails closed. Two instances sharing one operator-state database
coordinate only through the single-row write-fence; a superseded writer is fenced out
(the
fenced-outcondition) and writes nothing. This is the split-brain stop that makes a cross-location handover safe. Expect the older instance to fail closed once a successor claims a higher generation.
18. Legacy from-$HOME operator mode
The default run account is a dedicated static valis:valis. A legacy operator running
the resident out of their own account is supported via a drop-in: install
legacy-operator.conf.example as
/etc/systemd/system/valis.service.d/legacy-operator.conf and set the <operator> /
<operator-home> placeholders:
install -d -m 0755 /etc/systemd/system/valis.service.d
install -m 0644 deploy/systemd/valis.service.d/legacy-operator.conf.example \
/etc/systemd/system/valis.service.d/legacy-operator.conf
# ... edit <operator>/<operator-home>, then ...
systemctl daemon-reload
Overriding User=/=Group and pointing XDG_DATA_HOME (and HOME) at the operator's
home keeps valis rooting durable state there; on the first boot after such a move, the
resident relocates pre-XDG ~/.valis state cleanly. To fall all the way back to
~/.local/share, clear the inherited StateDirectory (an empty StateDirectory=).
19. Troubleshooting
| Symptom | Cause / fix |
|---|---|
Error opening shared object … libfixposix.so |
iolib dlopen=s the *unversioned* SONAME. Install the runtime package *and* create the =libfixposix.so symlink (or install -dev). |
c3po's serving codec … not available at boot |
a plain :valis core was staged. Stage the serving-capable core from the make dist delivery tarball instead. |
:443 dark unexpectedly |
check edge.domain is set (v get edge.domain, with the v wrapper from changing a setting) and a cert is in the custody store; read the boot line for which of the two. A genesis dark :443 is healthy. With a cert in custody and :443 still dark, run valis cert reload; if it stays dark with a usable cert for edge.domain, the launcher config names no :edge-port. |
setpriv: … Operation not permitted / no privilege drop |
Under NoNewPrivileges the execed setpriv helper keeps only the ambient caps, so emptying AmbientCapabilities breaks the privilege drop outright. Do not narrow the ambient set. |
| DNS boot cannot build its zone index | Postgres must be reachable over its local UNIX-domain socket under peer auth: verify the socket path in operator-state.dsn (valis config get operator-state.dsn, in the running-node or stopped-node form changing a setting gives) exists and the run account's uid maps to the database role. |
Type=notify times out despite the resident appearing to run |
NotifyAccess=all is required (valis is fulcrum's fork-child, not MainPID); and the launch chain must not clear the environment, or $NOTIFY_SOCKET never reaches valis. |
unit restarts but never opens :443 after a cert renewal |
renewal hot-swaps in place through the shared cell: a restart is not required and not the renewal path. Check the renewal manager's custody-store writes. |
new connections are refused while the unit stays active (running) and healthy |
descriptor exhaustion, which no instrument on the node reports: an accept loop that cannot accept is still an alive thread, so the liveness gate holds and the status line still says serving. Count /proc/<pid>/fd against Max open files in /proc/<pid>/limits for the resident, per descriptor headroom above. |
For the architectural why behind each of these, the privilege split, the descriptor
handoff, the sandbox directives that ship vs. those deliberately omitted
(MemoryDenyWriteExecute, LockPersonality), read the annotated
valis.service unit and the host-deployment contract.