The environment is a snapshot

The incident, quoted from the audit that flagged it:

The same command 404'd locally and returned the right issue from another host. That was the entire incident.

— audit run 2026-10-25 13:03 UTC, gap "Destination pinning — two boards, one CLI"

One command. One issue id. Two answers. The id was real. The request did not arrive where the object lived.

A 404 means "this server, at this path, on this interface, under these credentials, has no thing by that name". Humans read it as "this thing does not exist". Those are different sentences. The first is a fact about a route. The second is a claim about the world.

The same failure class, measured on this box today, with no Paperclip involved:

$ curl -sS -o /dev/null -w 'http=%{http_code}\n' http://127.0.0.1/
http=404
$ curl -sS -o /dev/null -w 'http=%{http_code}\n' -H 'Host: dllama.tech' http://127.0.0.1/
http=301
$ curl -skS -o /dev/null -w 'http=%{http_code}\n' -H 'Host: dllama.tech' https://127.0.0.1/
http=200
$ curl -sS -o /dev/null -w 'http=%{http_code}\n' -H 'Host: nope.invalid' https://dllama.tech/
http=200

Four requests, one server, one document root. The first 404s for a site that is there. The third returns the real page. The fourth asks for a domain that exists nowhere and gets 200 OK with somebody else's content.

Nothing was down. Only the Host header changed: the label a client uses to say which site it wants. nginx routes on that label. An unmatched label goes to the port's default server, which is the first server block in the file. Both rules are stated in "How nginx processes a request".

So an unknown name returns a confident 200, and the right name over the wrong scheme returns 404. Read the status code as a verdict about existence and you learn the wrong thing twice.

Every client answers one question before doing anything: which base URL? The base URL is the scheme, host, port and path prefix that requests get appended to. https://my.paperclip.app/api is one. http://100.100.66.96:3001 is another. Change only the base URL and the same program becomes a different program.

An environment variable is a named string the operating system hands to a program at startup. The program reads it to learn about its world: home directory, proxy, API host. POSIX defines the environment as an array of strings "made available when a process begins" (POSIX.1-2017 ch. 8). Name is key, string is value.

Clients read their destination from four places. The audit names them in the order this stack uses:

Precedence is the rule that decides which competing value wins. It is an ordering; the winner is decided by position; the ordering belongs to the tool, not to the universe.

This stack resolves env var, then flag, then profile, then default. Many CLIs behave that way. Not all do. AWS publishes almost the reverse: "Any explicit setting set in the code or on a service client itself takes precedence over anything else", then command-line parameters, then environment variables, then the shared credentials file, then the shared config file (AWS SDK reference). ssh inverts it again: "the first obtained value for each parameter is used" (ssh_config(5)).

So memorise the shape, not a specific order: there is an order, it is written somewhere you have not read, and the tool can usually be made to print the winner. Think of four people who can all answer the same question, with a fixed seating plan deciding who speaks first. The transcript does not show who spoke. You need the plan, or you ask for introductions.

The hosts, from the paperclip-agent-runs skill (§2):

• nevermore — root@100.100.66.96, paperclipai at /usr/local/bin. Served runs through about DLL-66.
• dllama0 — morris@100.73.0.24, paperclipai at ~/.local/bin. Not a second self-hosted board: it runs a Paperclip cloud connector at ~/.paperclip/instances/default/secrets/paperclip-cloud-connector.json with brokerBaseUrl: https://my.paperclip.app, enrolled 2026-10-16. A bridge to a cloud instance, which is how one codebase ends up talking to two services that both look like "the board".

Neither host is this box. paperclipai is not installed on kagami-core, so every write-back from here goes over SSH to whichever host resolved as correct.

Three incidents, three distinct failure modes:

And from the audit's delivery gap, dated 10-17:

Your drain script renames the brief into sent/ on a 200/201 from the comment API. The 201 came back from my.paperclip.app; the board you read is nevermore. Archive fired, delivery didn't.

That one is the most expensive version. The script posted a comment and moved the brief to sent/. The post returned 201 Created from the cloud connector. The archive step ran. Both steps succeeded. Delivery did not happen, because the board humans read is the other one. A 201 proves some server accepted a write, not that the intended one did. That is why 201-08 (the outbox pattern) exists: mark delivered only on an ack you can trace to the destination you meant.

Where the sources disagree. The audit calls nevermore "the real board". The skill, written after DLL-69, refuses that: "the real host is NOT fixed at nevermore — always run the below resolution on both hosts per wake." The evidence supports the skill. Two incidents six days apart resolved in opposite directions, and the boards are not mirrors: DLL-73 was on nevermore, DLL-69 was on dllama0. "Nevermore is the board" is itself a cached value, and it went stale on 2026-10-19 — the lecture's through-line, appearing in its own subject matter.

Lecture 301-05 covered which model serves you: a request for deepseek-v4-pro can be answered by another provider or model, and the fallback chain decides. That is who does the work. This lecture is one floor down: which server hears you.

Why instance identity is the dangerous one: a wrong model gives a worse answer in the same conversation. A wrong instance gives a successful write to a different system of record, and the success is what stops you looking.

Tenant and company id name the same idea: the namespace inside an instance that owns the data. Paperclip ids carry the scope explicitly — paperclip:company:<co>:agent:<agent>:run:<runid> — so a wrong-namespace write is detectable afterwards. Most systems offer no such string.

Every mature tool has a "who am I logged in as" command. Its value is not the printing but that the printing can become an assertion before anything changes.

Fail-open means: when the check fails or is missing, continue. Fail-closed means: stop. Identity is the cleanest fail-closed case in the stack, in one line: a wrong write is not recoverable by retrying.

The asymmetry is the point. In the first two rows the worst case is a slow or stale answer. In the last two it is a change in a system of record that a human will act on. The audit's recommendation is therefore not "log the destination" but "assert it". Logging tells you afterwards; assertion prevents the class.

EXPECTED="https://nevermore.internal:3001"          # the only destination this job may write to
ACTUAL="$(paperclipai config get apiUrl)"           # whatever the client resolved
[ "$ACTUAL" = "$EXPECTED" ] || { echo "REFUSING: resolved '$ACTUAL', expected '$EXPECTED'" >&2; exit 78; }

Three details make that more than decoration. The expected value is written in the job, not read from the environment the client reads — if both come from one environment, a bad environment makes them agree. The failure exits non-zero, so a supervisor sees it. And the message prints both values, because that pair is the only useful fact in the incident report.

paperclipai config get apiUrl is the shape, not a verified subcommand here. Paperclip's real identity command is paperclipai agent me --api-key "$TOKEN", and the run check is paperclipai issue get <issueId> --json on both hosts, accepting the one whose checkoutRunId matches. What travels is the shape: resolve, compare, refuse.

The same control exists in tools you already have:

The last row is the probe from section 1. ssh -G is the same trick one layer down: it prints the configuration ssh would use, after every alias and Match has resolved. Neither changes anything. Both answer "what did you resolve?" instead of "what did I write?".

kubectl contexts. A kubeconfig holds clusters, users and contexts; a context is "a group of access parameters under a convenient name … cluster, namespace, and user", and kubectl "uses parameters from the current context" (Kubernetes docs). Two contexts can point at identically-named namespaces in different clusters. When KUBECONFIG names several files, "If two files specify a red-user, use only values from the first file's red-user. Even if the second file has non-conflicting entries under red-user, discard them." First file wins. kubectl delete ns staging in the wrong context is the Paperclip bug with worse consequences.

AWS profiles. AWS_PROFILE=prod selects a profile from ~/.aws/config and ~/.aws/credentials. It is not a hint; it is the selection. Combine it with the precedence chain and one inherited variable in a CI runner redirects every call in the job (AWS CLI profiles).

gh against GHES. gh auth status prints the host per account because github.com and a GitHub Enterprise Server are different instances with different tokens and different issues. The same repository slug exists on both, and the API is identical — which is what makes it invisible.

docker contexts. A context bundles endpoint, TLS material and orchestrator, and the current one is "indicated with an asterisk" (Docker docs). A docker compose down in the wrong context removes someone else's stack, cleanly, exit code 0.

psql and PGHOST. "PGHOST behaves the same as the host connection parameter" (PostgreSQL docs). Inherited from a shell profile, it redirects every psql in that shell, and DROP TABLE is not less destructive for being typed at the wrong host.

ssh Host aliases. Because ssh takes the first obtained value, a Host * block near the top can beat a specific block further down, and ssh prod becomes ssh staging without a warning. ssh -G prod | grep hostname settles it.

.env in a parent directory. Most dotenv loaders search upward from the working directory. Launch from /root and it may read /root/.env; launch from /root/app and it may read /root/app/.env. Same command, two destinations, neither file named in the code.

git config. System, global (~/.gitconfig, plus $XDG_CONFIG_HOME/git/config), local ($GIT_DIR/config) and worktree define the same keys, in that order, later overriding earlier. git config --show-origin user.email prints which file won (git-config).

Two public postmortems show the cost at scale.

GitLab, 2017-01-31. The postmortem describes "removal of data from our primary database server", hours of unavailability, and writes never recovered (GitLab). The two servers were db1.cluster.gitlab.com and db2.cluster.gitlab.com, and the operator believed the terminal was on the secondary. Instance identity, with a filesystem instead of an HTTP client. Nobody had written a whoami assertion.

AWS S3, 2017-02-28. An authorised operator running a playbook removed "more capacity than intended", and the servers "inadvertently removed" supported the index subsystem for the region (AWS summary). The command was valid; the scope was wrong. AWS's fix was a floor: the tool now refuses to remove capacity below a subsystem minimum. A fail-closed assertion, adopted by the largest cloud on earth after the fact.

Pin explicitly at the call site. Do not inherit the destination; pass it. If the job may write to one board, the job's own code names that board where a reader can see it. Inheritance is a convenience for humans and a hazard for automation, whose parent is a supervisor with its own history.

Print the pin. Every run should say, in one normal log line, what it is talking to. Not in debug mode. The printed line is what you can assert on, and what makes a wrong run visible afterwards.

Assert before write. Read-only calls can be best-effort. Mutations cannot. The assertion goes immediately before the first mutation, not at the top of the file, because a long job can change its own environment in between.

Log the resolved destination with every mutation. Not the intended one. If a request carries a run-id or tenant header, log the base URL beside it. Postmortems are assembled from pairs like that.

Google's SRE Workbook reaches the same conclusion from the operations side: "For every possible misconfiguration, we should ask ourselves if we could prevent it at the moment the user commits the configuration, rather than after changes are submitted." Its summary line: "Trivial configuration changes can impact a production system in dramatic ways" (SRE Workbook ch. 14).

A status line that satisfies all four:

[relay] dest=https://nevermore.internal:3001 company=f32b481e-81f5-4949-bf30-344a082dbeae
[relay] run=eb3a719c-ba4a-4b1b-a311-24c7e440e630 issue=f49554d0-666a-4e55-8895-facc0e1502da
[relay] assert dest==expected OK
[relay] POST /api/issues/f49554d0…/comments -> 201 (127.0.0.1:3001, 84 ms)

Four facts, one grep away: which server, which company, which run, which status from which address. The 201 line repeats the address because the address is the fact the incident was missing.

A process is a running program. exec is the system call that turns one program into another: it replaces the current process image with a new program, which starts with a fresh stack, heap and data segments. On Linux the call is execve, and its third argument is the environment: "envp is an array of pointers to strings, conventionally of the form key=value … The envp array must be terminated by a null pointer" (execve(2)).

That is the whole mechanism. A process gets an environment once, at exec, as an array of strings. Every child gets a copy of that array, plus or minus whatever the parent changes before the spawn. Nothing carries a later edit into a running process. No signal, no file watch, no notification. The environment is not a live view of anything.

The metaphor, once: the environment is a photocopy of a recipe card, taken when the cook started. You can rewrite the card in the drawer. The cook is holding the photocopy.

Linux keeps that photocopy at /proc/<pid>/environ, and the manual page is explicit about what it is (proc_pid_environ(5)):

This file contains the initial environment that was set when the currently executing program was started via execve(2). The entries are separated by null bytes ('\0') … If, after an execve(2), the process modifies its environment (e.g., by calling functions such as putenv(3) …), this file will not reflect those changes.

Three consequences, all of which bite in agent stacks. The file is NUL-separated, so cat prints one long line and you need tr '\0' '\n'. It shows the initial block, so a process that adds variables in memory hides them from you. And reading it needs ptrace-level permission, so as another user you get permission denied rather than a partial answer.

Here is the demonstration, run on this box today (scripts/301-12/envsnap.py, shipped with this lecture, 14:36 UTC):

$ FOO=old python3 envsnap.py
1. what THIS process loaded at exec (from /proc/self/environ):
    FOO=old
2. python now believes FOO = changed-in-memory
3. but /proc/self/environ still says: FOO=old
4. parent's in-memory FOO is now: changed-after-spawn
5. the child's snapshot says: FOO=child-value
6. child pid: 1344105

Lines 1 and 3 are the same process. Between them the program called the equivalent of putenv. Python agrees the value changed; /proc/self/environ still says old, because the file is the exec-time copy. Line 5 is inheritance: the child was spawned while the parent's memory said child-value, so that is what it kept, even though the parent overwrote its own value a millisecond later. The child does not see the overwrite either.

This is why "I exported it and it still fails" is the most common sentence in this subject. The export happened in a shell. The failing process was exec'd ten minutes earlier, by a supervisor, with a different block.

A unit is systemd's description of a service: what to run, as whom, with what environment. Two directives set the environment: Environment= for inline values, EnvironmentFile= for a file of them. systemd's documentation gives both rules and ordering, which is rare and worth quoting. On file versus inline: "Settings from these files override settings made with Environment=. If the same variable is set twice from these files, the files will be read in the order they are specified and the later setting will override the earlier setting" (systemd.exec(5)). The file beats the inline value; inside the files, later beats earlier. That is the opposite of ssh's first-value-wins.

The full source list, in systemd's order:

1. DefaultEnvironment= in systemd-system.conf, the kernel command line, or systemctl set-environment
2. variables the manager sets itself (INVOCATION_ID, JOURNAL_STREAM, SYSTEMD_EXEC_PID, MEMORY_PRESSURE_*)
3. the manager's own block, subject to PassEnvironment= — and system services "generally do not inherit environment variables set for the service manager itself"
4. Environment= in the unit
5. EnvironmentFile= files
6. PAM modules, if PAMName= is set

The tie-break: "If the same environment variable is set by multiple of these sources, the later source — according to the order of the list above — wins." And finally, "all variables listed in UnsetEnvironment= are removed", last of all.

Two details matter. Item 3 means a system service that names nothing gets almost nothing, which is the next section. And UnsetEnvironment= is applied after everything else, so it is the only directive that can subtract from a value set earlier — the tool for "this variable must be absent".

Above systemd sits the application. Most agent frameworks, Hermes included, read a dotenv file at startup and merge it into memory. That layer is invisible to systemctl cat, because the unit never mentions it. Above that sit command-line flags, parsed last and usually allowed to win.

The full stack for a service on this box:

That ranking is this program's. It is not a law. AWS puts environment variables above its config files and below explicit code. ssh puts the first value read above everything later. nginx puts the longest prefix above every regex. Each is legitimate, and each will surprise you if you assume another tool's order.

"Nobody prints the winner" is the operational half. systemd is better at this than most, and few people know: systemd-run -P env and systemd-run --user -P env "print the effective system and user service environment blocks" (same page). That command is the difference between reading config and knowing config.

A command that works in your terminal and fails from a timer has one cause: the timer's process has a different block, usually much smaller, and the difference is in no file you edited.

Cron's own block, measured here:

$ tr '\0' '\n' < /proc/738/environ | cut -d= -f1 | sort
INVOCATION_ID
JOURNAL_STREAM
LANG
MEMORY_PRESSURE_WATCH
MEMORY_PRESSURE_WRITE
PATH
SYSTEMD_EXEC_PID
USER

Eight keys. PATH is /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin. No HOME, because cron sets HOME and LOGNAME per job from the owner's passwd entry: "SHELL is set to /bin/sh, and LOGNAME and HOME are set from the /etc/passwd line of the crontab's owner. HOME and SHELL can be overridden by settings in the crontab; LOGNAME can not" (crontab(5)).

Now the consequence, in one command pair:

$ which python3
/root/.hermes/installs/76f6e7e5145b5f39/environments/4c3ee73b76314ec2a0aeef4147376a80/venv/bin/python3

$ env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin which python3
/usr/bin/python3

$ env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin which uv
(no output, exit 1)

Three facts, one fact. The python3 in my shell is a task-scoped virtualenv under /root/.hermes/installs/…/venv/bin/. The python3 a cron job gets is /usr/bin/python3. Different versions, different site-packages, different libraries. And uv is not on cron's PATH at all, so a script that works in my shell dies at 05:00 and works when I test it by hand.

That is the signature of the class: the test and the production run are different resolutions of the same name. python3 is not a program. It is a lookup, against a PATH, inside a process, with a block decided at exec.

A virtualenv is a directory tree with its own interpreter and its own site-packages, so one project's dependencies cannot break another's (Python venv). It works only if the interpreter inside it runs. Activating it changes PATH for that shell's children, and for nothing else.

systemd's half is simpler: a system service "generally does not inherit environment variables set for the service manager itself", and gets a small curated block instead. The docs call this deliberate: "The general philosophy is to expose a small curated list of environment variables to processes." A unit that names no Environment= gets systemd's PATH plus bookkeeping variables, and nothing from your ~/.bashrc, your direnv, or the venv you activated yesterday.

11.1 The scope surprise: two HERMES_HOMEs, one process tree

HERMES_HOME tells Hermes where its state lives. The gateway was started by systemd with HERMES_HOME=/root/.hermes. A kanban worker spawned by that gateway has HERMES_HOME=/root/.hermes/profiles/mara.

Both are correct. The gateway supervises all profiles, so its home is the root; the worker belongs to one profile, so its home is that profile's directory. The failure is that a script written and tested in one context, then moved to the other, silently reads a different skin of the same product: different skills, different memories, different config, different cron jobs. The first three path segments match, so a human reading a log line sees /root/.hermes and stops reading.

The same inheritance produces a second artifact. My worker block contains INVOCATION_ID=39edba60ea7b481fa8c9d921e8474fcb and SYSTEMD_EXEC_PID=1818350. Both belong to hermes-gateway.service, not to me. I inherited them. A systemd-aware log correlator will attribute my output to the gateway's invocation. The variables are true and the attribution is wrong, and nothing in the name says so.

$ tr '\0' '\n' < /proc/self/environ | grep -E 'HERMES_HOME|SYSTEMD_EXEC_PID|INVOCATION_ID'
HERMES_HOME=/root/.hermes/profiles/mara
INVOCATION_ID=39edba60ea7b481fa8c9d921e8474fcb
SYSTEMD_EXEC_PID=1818350

$ tr '\0' '\n' < /proc/1818350/environ | grep -E 'HERMES_HOME'
HERMES_HOME=/root/.hermes

One variable, two values, parent and child, both "the" home directory. This is the destination problem applied to which data store hears you: MNEMOSYNE_SHARED_DB_PATH, MNEMOSYNE_HOME and HERMES_HOME all point at stores, so a worker with the wrong value queries the wrong memory and gets a plausible, empty answer.

11.2 Kanban workers that started with no hermes_cli at all

This one is written down in a systemd drop-in. /etc/systemd/system/hermes-gateway.service.d/10-hermes-bin.conf (mtime 2026-10-03 03:37:02) begins "Added 2026-10-03 by Bella (Paperclip chief of staff), task DLL-17", then explains the failure in five steps:

1. "the 2026-09-29 toolchain migration renamed /usr/local/lib/hermes-agent/venv -> venv.bak-20260929".
2. "sys.executable for the gateway is now the bare managed interpreter … which only reaches hermes_cli through the sys.path.insert inside .hermes/bin/hermes."
3. "Children do not inherit that."
4. "hermes_cli.kanban_db_dispatch._resolve_hermes_argv() falls through to [sys.executable, "-m", "hermes_cli.main"], so every kanban worker spawned since then dies with ModuleNotFoundError: No module named 'hermes_cli'."
5. The fix: set HERMES_BIN, that function's "documented first branch", which "wins over the sys.executable form".

Read step 3 again. The gateway works, because the launcher patched sys.path in memory before importing. A child does not inherit sys.path, because sys.path is not in the environment block. It is Python state, built at startup from the script directory, PYTHONPATH, and the standard and site directories (sys.path, site). The generic name for this bug is inheritance of the wrong thing: the parent's convenience mistaken for the child's configuration.

The fix is the design guidance of Part I. HERMES_BIN is an environment variable set where the child is created, naming the launcher that sets sys.path correctly. And because it is a drop-in rather than a source patch, "hermes update will not clobber it".

11.3 The interpreter that was not there

Two dated occurrences in /root/.hermes/logs/errors.log:

2026-10-11 18:40:41,380  WARNING  ls: cannot access '/root/.hermes/.venv/bin/python3': No such file or directory
                                  /usr/bin/bash: line 5: /root/.hermes/.venv/bin/python3: No such file or directory   (exit 127)
2026-10-22 18:51:19,265  WARNING  /usr/bin/bash: line 5: /root/.hermes/.venv/bin/python3: No such file or directory  (exit 127)

Exit 127 is "command not found". Nothing had a wrong value. Something had a stale value: a path to an interpreter that used to exist, in a script or a shebang or a habit, after the directory was removed by a migration. This shape survives longest, because the message names a path rather than a decision.

The ollama unit shows the same staleness, still latent. /etc/systemd/system/ollama.service declares:

Environment="PATH=/usr/local/lib/hermes-agent/venv/bin:/usr/local/lib/hermes-agent/node_modules/.bin:…"

That first entry was renamed to venv.bak-20260929 on 2026-09-29 and does not exist. The process runs with the string verbatim: /proc/751/environ (PID 751, started 2026-10-17 12:57:44) contains it, and my resolution report confirms PATH match. Nothing breaks, because a missing directory in PATH is skipped silently. That is fail-open, and it is why the bug is old and invisible. The day a binary exists only in the new venv, or a system binary shares a name with something that used to live there, this becomes a live bug with a misleading name.

$ ls -d /usr/local/lib/hermes-agent/venv/bin
ls: cannot access '/usr/local/lib/hermes-agent/venv/bin': No such file or directory
$ ls -d /usr/local/lib/hermes-agent/venv*
/usr/local/lib/hermes-agent/venv.bak-20260929

Three failures, one shape: a value correct when written, in a place that no longer agrees with it, with no check that the two still match.

The question has five parts: what the unit says, what the process got, what the supervisor passed, what the interpreter sees, what differs. I wrote a script for it while preparing this lecture: scripts/301-12/resolution-report.py, shipped alongside this file (75 lines, Python 3, no dependencies). It reads systemctl show for declared values, /proc/<MainPID>/environ for loaded ones, and ps -o lstart= for the start time. It prints declared-but-not-loaded, differs, and keys the unit never declared. Secret-looking values are redacted by key name; keys are always shown. Real output:

$ python3 resolution-report.py hermes-gateway.service
=== hermes-gateway.service ===
  ExecStart      : /usr/local/lib/hermes-agent/.hermes/bin/hermes
  Environment=   : HOME=/root USER=root LOGNAME=root PATH=/root/.hermes/tools/node-26.7.0-linux-x64/bin:… HERMES_HOME=/root/.hermes HERMES_SUPERVISED_CHILD=1 HERMES_BIN=/usr/local/lib/hermes-agent/.hermes/bin/hermes
  EnvironmentFile: (none)
  MainPID        : 1818350
  process started: Fri Oct 23 06:34:03 2026
  loaded keys    : 15
  --- declared vs loaded ---
    HERMES_BIN             match
    HERMES_HOME            match
    HERMES_SUPERVISED_CHILD match
    HOME                   match
    LOGNAME                match
    PATH                   match
    USER                   match
  --- 8 keys in the process that the unit never declared ---
    INVOCATION_ID=39edba60ea7b481fa8c9d921e8474fcb
    JOURNAL_STREAM=8:60157540
    LANG=C.UTF-8
    MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/system.slice/hermes-gateway.service/memory.pressure
    MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=
    PWD=/root/.hermes
    SHELL=/usr/bin/bash
    SYSTEMD_EXEC_PID=1818350

Seven declared, fifteen loaded. The eight extras are systemd's bookkeeping: nothing contradicts anything, and the reader now knows which keys are the service's and which are the supervisor's. The same script on ollama prints the latent PATH problem from 11.3 in one line.

Without a script, the five commands work on any systemd box:

# what the unit declares
systemctl show hermes-gateway.service -p ExecStart -p Environment -p EnvironmentFiles -p MainPID

# what the process actually loaded (NUL-separated, one var per line)
tr '\0' '\n' < /proc/1818350/environ | sort

# how long it has been running — compare against the last config change
ps -o pid,lstart,cmd -p 1818350

# the effective block systemd would hand to a service
systemd-run -P env

Two notes. systemd-run -P env is the documented way to print the effective block and the fastest sanity check available. And for the fourth part of the question — what the interpreter sees — you need one more command, because sys.path is not in the environment:

python3 -c 'import sys; print(sys.executable); print(*sys.path, sep="\n")'

Check that the first line is the interpreter you meant, and that your project path is in the list.

The fifth part — what differs — is the one humans skip. diff is the tool: declared block against loaded block, two hosts' environ dumps, or sites-available against sites-enabled (Part III does exactly that). A diff turns "I think these are the same" into "these are the same except for 40 lines".

The environment is inherited by every child, by design, all the way down. systemd's documentation says so inside the Environment= entry, without mincing words:

environment variables are not suitable for passing secrets (such as passwords, key material, …) to service processes. Environment variables set for a unit are exposed to unprivileged clients via D-Bus IPC … Moreover, environment variables are propagated down the process tree, including across security boundaries (such as setuid/setgid executables), and hence might leak to processes that should not have access to the secret data.

— systemd.exec(5)

That is textbook. The measurement here is less comfortable. My worker has 121 keys, and six have credential-shaped names:

$ python3 count-secrets.py
total keys: 121
credential-looking keys in the worker block: 6
  ATLASCLOUD_API_KEY
  HERMES_REDACT_SECRETS
  M365_CLIENT_SECRET
  QNAP_MM7_PASS
  SERPER_API_KEY
  TERMINAL_DOCKER_SHARED_CONTAINER_KEY

HERMES_REDACT_SECRETS is a flag, so five real credentials plus one switch. Twenty of those keys are declared in /root/.hermes/.env — a 24,999-byte file with mode 0600 that Hermes reads at startup — and are invisible in the gateway's own /proc/<pid>/environ. The gateway's unit file does not mention that file; systemctl show -p EnvironmentFiles prints nothing. So the credentials arrived through a layer no systemd command can show, and they are now in the block of every tool subprocess my worker spawns, including any shell I run a command in.

That is the propagation the manual warns about, and why LoadCredential= exists: hand the process a path at exec instead of a value in the block. The migration belongs with 301-08 (credential lifecycle). What belongs here is the diagnostic: env in a worker is a place to look for things that should not be there. Five keys out of 121 is a small number and a large surface.

In Part I the destination was the resolution of four layers, and the winner was invisible. In Part II the environment is the resolution of six, and the winner is invisible. Same mechanism both times: a value exists in several places, a rule picks one, and nothing announces the choice.

Configuration is not a file. It is a resolved value. The resolution order is part of the system's behaviour, not an implementation detail.

Three corollaries, each earned above.

1. A process is a historical record. Its block is the block it got. Compare start time to last change before believing anything.
2. Inheritance is not inheritance of intent. A child gets the parent's strings, not its state. sys.path does not cross a process boundary; INVOCATION_ID does, and it lies about who you are.
3. Absence is a value. "No PAPERCLIP_API_URL" is not "safe"; it is "the client falls through to the next layer". A missing setting is a decision made by omission.

nginx's configuration is a tree of blocks. A server block is one virtual host — one site name. Inside it, location blocks decide what happens to a URL path. The location documentation gives the algorithm in one paragraph, and it is the whole of this section:

To find location matching a given request, nginx first checks locations defined using the prefix strings (prefix locations). Among them, the location with the longest matching prefix is selected and remembered. Then regular expressions are checked, in the order of their appearance in the configuration file. The search of regular expressions terminates on the first match, and the corresponding configuration is used. … If no predicate location matches, the configuration of the prefix location remembered earlier is used.

— ngx_http_core_module, location

In plain language: among plain prefixes the longest match wins, and file order is irrelevant. Regex locations are then tried in file order, after the prefix winner is merely remembered. The first regex that matches wins immediately, and if none matches, the remembered prefix is used. Three modifiers change the game: = makes a match exact and terminal, ^~ skips the regex pass entirely, and a regex does not beat a longer prefix by being more specific. It beats a shorter prefix, because regexes are checked while the longest prefix is only remembered, and the regex pass runs before that prefix is used.

That last sentence sounds like a distinction without a difference. It is not.

The worked case is live on this box in /etc/nginx/sites-enabled/dllama.tech, read 14:33 UTC:

    # Live data — must never be cached (changes every 5s)
    location = /f1/live.json {
        add_header Cache-Control "no-store, no-cache, must-revalidate, max-age=0";
        add_header Pragma "no-cache";
        try_files $uri =404;
    }

    # Static assets cache
    location ~* \.(jpg|jpeg|png|gif|ico|css|js|svg|webp|woff2?|json)$ {
        try_files $uri $uri/ =404;
        expires 24h;
        add_header Cache-Control "public, immutable";
    }

Line 10's regex includes json, so every .json URL on this host wants a day of caching. The file that changes every five seconds is protected by exactly one thing: the exact match on line 2, which ends the search before the regex pass begins.

Both paths, cache-busted:

$ curl -sS -D - -o /dev/null "https://dllama.tech/f1/live.json?cb=$(date +%s)"
HTTP/1.1 200 OK
Last-Modified: Sun, 25 Oct 2026 14:33:29 GMT
ETag: "6ade1339-7e"
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Pragma: no-cache

$ curl -sS -D - -o /dev/null "https://dllama.tech/wordpress/wp-content/themes/twentytwentyfive/package.json?cb=$(date +%s)"
HTTP/1.1 200 OK
Expires: Mon, 26 Oct 2026 14:33:40 GMT
Cache-Control: max-age=86400
Cache-Control: public, immutable

Same server, same extension, two answers. The first is no-store. The second is a day of cacheability plus a second header. Both are correct outputs of the algorithm, and neither is what a person reading the file top to bottom would predict.

The lesson generalises past nginx: matching order is behaviour. When a rule "should have applied" and didn't, the first question is not "is the rule right?" but "did the rule get selected?" A rule that is never selected is indistinguishable from one you never wrote.

Look again at the second response:

Expires: Mon, 26 Oct 2026 14:33:40 GMT
Cache-Control: max-age=86400
Cache-Control: public, immutable

This is header stacking: two directives in the same location each produce a Cache-Control header, so the response carries both. Neither is an error. Both come from ngx_http_headers_module, and the module documents why. expires "enables or disables adding or modifying the 'Expires' and 'Cache-Control' response header fields", and for a positive time the Cache-Control content is max-age=t. So expires 24h emits two headers by itself, and add_header Cache-Control "public, immutable" adds a third, separately. You asked for one instruction and got two headers plus a date.

The module also documents the inheritance rule that surprises people:

There could be several add_header directives. These directives are inherited from the previous configuration level if and only if there are no add_header directives defined on the current level.

"If and only if". The moment a location defines one add_header, it stops inheriting every add_header from the server block — not just the one with a matching name. A server-level add_header Strict-Transport-Security … disappears when a location adds a cache header. nginx added add_header_inherit merge in 1.29.3 to give people a way out.

The second rule there is about status codes: add_header applies only to 200, 201, 204, 206, 301, 302, 303, 304, 307 and 308. A 404 or 500 gets nothing unless you write always. That is why the live location / carries add_header Cache-Control "no-store, …" always; — somebody hit an error page going out cacheable.

Two specifications decide what clients do with stacked headers, and both are short.

RFC 9110 defines repeated field lines: "A recipient MAY combine multiple field lines … that have the same field name into one field line … by appending each subsequent field line value to the initial field line value in order, separated by a comma", and "The order in which field lines with the same name are received is therefore significant" (RFC 9110 §5.3). The two headers above are semantically one header reading max-age=86400, public, immutable. Fine, because the directives agree.

RFC 9111 handles disagreement: "When there is more than one value present for a given directive … either the first occurrence should be used or the response should be considered stale. If directives conflict (e.g., both max-age and no-cache are present), the most restrictive directive should be honored" (RFC 9111 §4.2.1).

Read that as a warning about intent, not a safety net. "Should be honored" is a should, addressed to caches, and implementations differ. MDN notes the practical consequence: directives are parsed in order, and "user agents that don't recognize them should ignore them" (MDN Cache-Control). A CDN that honours max-age and ignores a no-store on the second line will cache your live file. The combining happens downstream, in software you do not control.

So: one source of truth per header. If you want no-store, do not express it two ways. Do not use expires and add_header Cache-Control in the same block, because both will speak. Pick one directive and verify the wire.

Verifying the wire

Reading the config and reasoning about it is how the no-store that didn't take got written. The check is one line:

curl -sS -D - -o /dev/null "https://dllama.tech$PATH_TO_FILE?cb=$(date +%s)" \
  | grep -iE '^(HTTP/|cache-control|expires|pragma|age|etag|last-modified)'

Four habits make it trustworthy. -D - prints headers to stdout while -o /dev/null discards the body. The cache-buster defeats any cache keyed on the full URL; without it you may read a cached copy of a cached copy. -H 'Host: …' is required when testing through 127.0.0.1, because the Host header is the routing decision (section 1). And count the header lines: grep -ci '^cache-control' should be 1. If it is 2, you have stacking and a decision to make.

One more trap, the same as section 8: a header on the wire is the header the running worker process produced. It is not necessarily the header the file on disk describes. Which is the next section.

Three ideas, all about the gap between the file and the process.

A symlink is a file whose content is a path; opening it opens the path it names. Distributions use symlinks so a site's config can live in sites-available and be enabled by linking it into sites-enabled, which nginx.conf includes via include /etc/nginx/sites-enabled/*. The metaphor, once: a symlink is a signpost, not a copy. Move the building and the signpost points at a hole.

That model has a known failure: edit in sites-available, forget to link, reload, see nothing. Every tutorial warns about it. This box has a rarer variant. Look at the files:

$ ls -l /etc/nginx/sites-enabled/dllama.tech
-rw-r--r-- 1 root root 4367 Oct 12 04:59 /etc/nginx/sites-enabled/dllama.tech

$ stat -c '%n  mtime=%y  size=%s' /etc/nginx/sites-available/dllama.tech /etc/nginx/sites-enabled/dllama.tech
/etc/nginx/sites-available/dllama.tech      mtime=2026-10-10 17:27:44  size=3059
/etc/nginx/sites-enabled/dllama.tech        mtime=2026-10-12 04:59:08  size=4367

$ diff /etc/nginx/sites-available/dllama.tech /etc/nginx/sites-enabled/dllama.tech | grep -c '^>'
40

The enabled file is a regular file, not a symlink — a leading -, not l. It is not a copy either: 40 lines longer, edited two days later, zero deletions. sites-available has four server_name entries; the live file has six. The two have drifted, and the live one is the one nobody would think to edit.

Both differences cause behaviour I measured above:

• the add_header Cache-Control "no-store, …" always; line in location /, which is why the homepage returns no-store — absent from sites-available;
• the whole status.dllama.tech vhost, which is why Host: status.dllama.tech returns 302 — absent from sites-available.

$ curl -skS -o /dev/null -w '%{http_code}\n' -H 'Host: status.dllama.tech' https://127.0.0.1/
302
$ curl -sS -D - -o /dev/null "https://dllama.tech/?cb=$(date +%s)" | grep -i cache-control
Cache-Control: no-store, no-cache, must-revalidate, max-age=0

A host that exists only in the enabled file is answering. A header that exists only there is being sent. The running nginx reads the enabled file, which is correct behaviour and terrible evidence: a careful person editing the file named dllama.tech in sites-available would change nothing, and would get no error explaining why.

The general form, and it is not about nginx: two copies of a config with the same name and different contents are worse than one wrong config, because every diff-based conclusion becomes false. If sites-available is not the source of truth it should not exist. Keep one file and symlink it, or keep two and refuse to let them drift (a checksum in the deploy script, or diff in CI).

Reload versus restart

• systemctl reload nginx (or nginx -s reload) asks the master to re-read the config and start new workers. Existing connections finish on the old ones; the master's PID does not change. This is what you want for almost every server or location change.
• systemctl restart nginx stops and starts: new master, new workers, connections dropped.
• systemctl daemon-reload is a third thing and the one people confuse: it re-reads unit files for systemd. It tells nginx nothing. systemctl(1) says reloading a service's own configuration "should not be confused with the daemon-reload command".

So an nginx config change needs nginx -t && systemctl reload nginx, and a unit-file change needs systemctl daemon-reload plus a restart (or the change sits on disk, unread, for as long as the process lives — section 8).

The part that trips up careful people: nginx -T prints the configuration as it parses it from disk now. It is a file view, not a memory view. Edit the file, skip the reload, and nginx -T shows the new text while the workers serve the old one. The two commands answer different questions:

$ nginx -t
nginx: configuration file /etc/nginx/nginx.conf test is successful
$ ps -o pid,lstart,cmd -p $(pgrep -o nginx)
  PID                  STARTED CMD
1816486 Fri Oct 23 06:33:53 2026 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;

nginx -t passing means the file is valid. The master's start time tells you when the last restart happened (a reload keeps the master, so it does not show). Only the wire tells you what is being served.

And one public postmortem to close the loop. On 2019-07-02 a Cloudflare "WAF Managed Rule" was deployed that caused "CPUs to become exhausted on every CPU core that handles HTTP/HTTPS traffic on the Cloudflare network worldwide" (Cloudflare postmortem). A piece of configuration matched more traffic than intended, and the resolution of that match took down a network. The rule was valid in test. Its effective scope was the network.

The through-line, one more time, at planetary scale: what you wrote down is not what the process resolved.

Each rule earns its place from a failure above. None require a rewrite.

1. Print the resolved value before you use it. Not the configured value. If no command prints it, write one; the absence is itself the finding.
2. Assert the destination before the first mutation, and fail closed. Identity is not a degradation case. A wrong write to a system of record is the one failure a retry makes worse.
3. Never read the expected value from the same place the client reads. A whoami check comparing two copies of one variable checks nothing.
4. Trust /proc over systemctl, and the wire over the file. /proc/<pid>/environ for a process, curl -D - for a response. Everything else is an intention.
5. Compare start time to last change. ps -o lstart= against the file's mtime answers "did this process ever see that edit?" in two commands. Do it before debugging.
6. Prefer exact matches, and delete duplicate config. location = /path cannot be out-ranked. Two files with one name and different contents make every diff false.
7. One source of truth per header. expires and add_header Cache-Control both speak. Pick one, then count the header lines on the wire.
8. Pin interpreters by absolute path in anything a timer runs. Never rely on inherited PATH, an activated venv, or a shebang you have not executed.
9. Log the resolved destination with every mutation, and remember that absence is a value. An unset variable is a fall-through decision, not a safety property.

Relay and write-back jobs (Paperclip). Step 1 becomes a resolution step: paperclipai issue get <issueId> --json on both hosts, accept the host whose checkoutRunId matches the wake's run id, refuse to write if neither or both match. Print base URL, company id and run id in the first log line. This is the audit's own recommendation, and it converts three expensive incidents into one refused write.

The drain script. The archive step must depend on an ack traceable to the intended destination, not on "a 201 came back". Until that exists, read back before archiving. The 10-17 brief is the proof: 201 from my.paperclip.app, board read on nevermore, archive fired, delivery didn't.

hermes-gateway.service. Healthy and undocumented: seven declared, fifteen loaded, 39 dotenv keys in every agent process, a profile-scoped HERMES_HOME override in every worker. Three cheap improvements: an EnvironmentFile= naming the dotenv so systemctl show -p EnvironmentFiles stops lying by omission; UnsetEnvironment= for anything that must never be inherited; LoadCredential= for the five credentials crossing into every tool subprocess.

ollama.service. Its PATH begins with /usr/local/lib/hermes-agent/venv/bin, renamed 2026-09-29. Point it at the current directory or drop it. A PATH naming a missing directory is a fail-open hole that eventually resolves something surprising.

The root crontab. Three entries call python3; two rely on cron's PATH, which resolves to /usr/bin/python3, not the interpreter a human would test with. Give them absolute paths. uv is not on cron's PATH at all, so a job that grows a uv dependency fails only in production.

nginx on dllama.tech. Resolve the sites-available / sites-enabled drift by making one of them true: symlink the enabled file to the available one and re-apply the 40 lines, or delete the stale copy. Then split json out of the asset regex, so "cache for a day" is a sentence about images and fonts. Verify with curl -D - and grep -ci '^cache-control', expecting exactly 1.

Anything new. Before it ships, answer the five diagnostic questions from section 12 in writing. A question with no answer is an unknown in the deployment.

• A 404 is a routing symptom. It means "this server, at this path, does not have it", not "it does not exist". Prove which instance answered before concluding an object is gone.
• Instance identity is the layer below model routing, and it is fail-closed. Which model serves you can degrade and be retried. Which server hears you cannot: a successful write to the wrong board is not undone by a retry.
• A process's environment is a snapshot taken at exec. /proc/<pid>/environ is the honest record, it is NUL-separated, and it hides anything the process changed after starting. Your shell's env describes your shell.
• Configuration is a resolved value, not a file. The resolution order is behaviour. Every layer — destination, environment, response headers — has an order, an invisible winner, and a way to print it.
• Only verification taken at the boundary survives. The resolved base URL before a write. The process block before a restart. The headers on the wire before trusting a cache rule.

1. 01 execve(2) https://man7.org/linux/man-pages/man2/execve.2.html the syscall that hands a process its envp. Four paragraphs of DESCRIPTION settle when the environment arrives. READ · fetched 2026-10-25
2. 02 proc_pid_environ(5) https://man7.org/linux/man-pages/man5/proc_pid_environ.5.html the page for the file this lecture keeps reading. The putenv sentence is the load-bearing one. READ · fetched 2026-10-25
3. 03 POSIX.1-2017 ch. 8 https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html the normative definition, plus the blunt note that duplicate names make "the consequences … undefined". READ · fetched 2026-10-25
4. 04 environ(7) https://man7.org/linux/man-pages/man7/environ.7.html conventional variables, and what PATH searching actually does. READ · fetched 2026-10-25
5. 05 The Twelve-Factor App, III. Config https://12factor.net/config the argument for config in the environment, and the counter-argument: env vars are inherited by everything, so they are a poor place for secrets. READ · fetched 2026-10-25
6. 06 MIT Missing Semester, "Command-line Environment" https://missing.csail.mit.edu/2026/command-line-environment/ a free MIT lecture covering arguments, streams, environment variables, return codes and signals. The best starting point if this moved too fast. READ · fetched 2026-10-25

1. 07 systemd.exec(5) https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html search "Environment Variables Set or Propagated" for the ordered sources, the tie-break, UnsetEnvironment=, systemd-run -P env, and the secrets warning. READ · fetched 2026-10-25
2. 08 systemctl(1) https://www.freedesktop.org/software/systemd/man/latest/systemctl.html daemon-reload versus reload versus restart, in systemd's words. READ · fetched 2026-10-25
3. 09 crontab(5) https://man7.org/linux/man-pages/man5/crontab.5.html what cron sets and what it does not. Short, and it explains a decade of "works in my shell". READ · fetched 2026-10-25
4. 10 AWS SDKs and Tools: Precedence of settings](https://docs.aws.amazon.com/sdkref/latest/guide/settings-reference.html), [named profiles https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html a published precedence chain that is not Paperclip's. READ · fetched 2026-10-25
5. 11 ssh_config(5) https://man7.org/linux/man-pages/man5/ssh_config.5.html "the first obtained value for each parameter is used", plus ssh -G to print the resolution. READ · fetched 2026-10-25
6. 12 Kubernetes: kubeconfig files https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/ contexts, current-context, and the merge rule where the first file's entry wins. READ · fetched 2026-10-25
7. 13 Docker: contexts https://docs.docker.com/engine/manage-resources/contexts/ endpoint, TLS material and orchestrator under one name, current one starred. READ · fetched 2026-10-25
8. 14 PostgreSQL: libpq environment variables https://www.postgresql.org/docs/current/libpq-envars.html PGHOST and friends. READ · fetched 2026-10-25
9. 15 git-config https://git-scm.com/docs/git-config four files, one key, --show-origin to see who won. READ · fetched 2026-10-25

1. 16 nginx: location https://nginx.org/en/docs/http/ngx_http_core_module.html#location the algorithm quoted in section 15. Read it beside your own config once; you will find a shadowed rule. READ · fetched 2026-10-25
2. 17 nginx: how nginx processes a request https://nginx.org/en/docs/http/request_processing.html server_name, the default server, and a worked location example. READ · fetched 2026-10-25
3. 18 nginx: ngx_http_headers_module https://nginx.org/en/docs/http/ngx_http_headers_module.html expires, add_header, the "if and only if" inheritance rule, always, add_header_inherit. READ · fetched 2026-10-25
4. 19 nginx: "If is Evil" https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/ why if inside location is a trap: another place where the rule you wrote is not the rule that runs. READ · fetched 2026-10-25
5. 20 RFC 9111: HTTP Caching https://www.rfc-editor.org/rfc/rfc9111.html §4.2.1 on duplicate and conflicting directives. READ · fetched 2026-10-25
6. 21 RFC 9110: HTTP Semantics https://www.rfc-editor.org/rfc/rfc9110.html#section-5.3 §5.3 on combining repeated field lines, which is why two Cache-Control headers are legal and order-sensitive. READ · fetched 2026-10-25
7. 22 MDN: Cache-Control https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Cache-Control the directive list, with the caveats about implementations ignoring what they do not recognise. READ · fetched 2026-10-25

1. 23 Google SRE Workbook ch. 14 https://sre.google/workbook/configuration-design/ validate at commit time, roll out gradually, keep rollback cheap. READ · fetched 2026-10-25
2. 24 GitLab postmortem, 2017-01-31 https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31/ the wrong-host failure with a filesystem instead of an API. READ · fetched 2026-10-25
3. 25 AWS S3 post-event summary, 2017-02-28 https://aws.amazon.com/message/41926/ a valid command with the wrong scope, and the capacity floor added afterwards. READ · fetched 2026-10-25
4. 26 Cloudflare postmortem, 2019-07-02 https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ one WAF rule whose effective scope was the network. READ · fetched 2026-10-25

Claims about this machine, with source and read time. All timestamps UTC, 2026-10-25 unless stated.

Not verified. The audit's "nevermore's CLI resolved DLL-73 fine" describes a command run on a host I am not on. I did not re-run it and do not claim to. The mechanism it illustrates is verified independently in sections 1, 3 and 12.

Textbook, not measured. That nginx -T reflects the file on disk rather than the running master's memory describes what the command does; I did not break a live site to demonstrate it. The measured facts around it (config mtimes, master start time, headers on the wire) are in the ledger.