> ## Documentation Index
> Fetch the complete documentation index at: https://docs.r5d.chat/llms.txt
> Use this file to discover all available pages before exploring further.

# Private automation operations

> Default automation availability, bounded private execution, recovery and disable controls.

## Scope and release status

Private automations are available to every account by default. The chart enables durable runs, schedules, webhooks, bounded inference, and the executor. Each automation still requires its owner's approval. Private services remain a separate opt-in facility.

The platform accepts a bounded, authenticated webhook and returns a fixed asynchronous acknowledgement. It never returns tenant code, proxies a workspace port, serves tenant HTML, or offers a public result/inference API. Workspaces retain only the private authenticated daemon transport; do not add tenant Ingress, NodePort, LoadBalancer, port-forwarding or tunnel product features. Ingress policy does not prevent arbitrary same-user shell code from making outbound relays.

## Packaging and enablement

The control-plane image also includes `.output/automations/executor.mjs` and `migrate.mjs`. The executor is an independent Deployment, not a timer started by an HTTP import. It has no Service or public health listener. It reuses the control plane's existing Secret and namespace-scoped workspace permissions. Do not inject provider or infrastructure credentials into automation command environments.

The following values default to `true`: `automations.durableRuns`, `enabled`, `schedules`, `webhooks`, `inference`, and `allOwners`. Every current and future account has access without maintaining an allowlist. This does not bypass authentication, enabled-account checks, grants, quotas, ownership, or version-specific approval. `automations.ownerIds` defaults to `[]`; to deliberately restrict access, set `allOwners: false` and list explicit owner IDs. `privateServices.enabled` defaults to `false` and additionally requires `privateServices.capacityApproved` (runtime `R5DCHAT_SERVICE_CAPACITY=1`).

Outside Helm, the control plane uses the same enabled defaults when the corresponding `R5DCHAT_*` variables are absent. Explicit values must be exactly `true` to enable a capability; `false` disables it. A nonempty `R5DCHAT_AUTOMATION_OWNER_IDS` restricts access unless `R5DCHAT_AUTOMATION_ALL_OWNERS=true` is also set. Existing explicit overrides survive upgrades: remove old opt-out settings or set the values above to `true` when enabling an older installation.

`executor.enabled` defaults to `true` and creates the dedicated process; `executor.roles` defaults to `executor,triggers,retention`; add `services` explicitly for private-service authorization renewal. There is no fixed shared executor run cap or per-owner/per-automation active or queued run cap. Ordinary chat is a conversation-owned agent loop, not an automation run per message. Each conversation has one fenced loop owner; independent conversations run concurrently without numeric caps. Inputs arriving while its loop is active remain in the agent inbox or steer it at a tool/model boundary. Tools have no read/exclusive or owner-wide queue; only same-owner writes/edits to the same normalized lexical path are ordered within a server process. This is not a cross-replica filesystem lock. Daily quotas, per-run bounds, authorization and resource limits still apply. Default poll interval is 1000ms, drain deadline 30000ms, termination grace 45s. These are process knobs, not throughput promises. Use resource requests/limits independent of the web deployment. SIGTERM stops claims and drains; exceeding the deadline must surface unknown dispatched effects rather than replay them. Deployment readiness is only process health, not evidence that an external action succeeded.

`automations.policy` is a **fixed ceiling declaration checked against the protocol**, not editable working quota configuration. Changed values fail chart rendering. A reviewed code/protocol change and tests are required to change these ceilings:

| Scope | Conservative ceiling |
| - | - |
| Run | 900s, 64KiB input, 256KiB output, 8192 tokens |
| Owner/day | 100 runs; 100000 tokens reserved atomically |
| Child depth / attempts | 2 / at most 3; unknown effects are not retryable |
| Webhook | 64KiB body, 32 backlog, 30/min endpoint baseline |
| Private services | 1 per owner; authorization expires after 60s |
| Run events / results / dedupe | 7 / 30 / 90 days |

Provider cost may be unavailable; token ceilings are not exact monetary billing. Verify owner/IP/endpoint throttling and fair multi-owner progress under your workload, not just a single-owner benchmark. No per-service cgroup sandbox or exact CPU isolation is promised; same-user root can inspect other same-user files and processes.

## Daily checks and retention

Track oldest queued run/trigger age, queue depth per owner, webhook accepted/rejected/429/503 counts, attempts, stale leases, unknown outcomes, cancellation latency, executor RSS/CPU, workspace wake latency, PG connections/locks, PVC free bytes, and retention sweep errors. Never record webhook bodies, secrets, prompts or browser profiles as metric labels. Alert on sustained growth, capacity approaching a ceiling, or a missed cleanup window; numeric SLO thresholds require measured staging evidence.

Events expire before results. An expired event cursor returns `cursor_expired`; fetch retained status rather than restarting the action. Removing visible history must retain request digests, idempotency keys and provider occurrence/event tombstones through their retry horizon. A replay after that horizon requires a separately reviewed decision. Do not use `TRUNCATE`, cascading history deletion, or manual tombstone removal to fix disk pressure.

Keep logs and payloads bounded, private, redacted and outside artifact publication or broad search/indexing. Secret values are encrypted with the settings encryption key; rotate/revoke through owner controls, not log export. Browser state and mutable SQLite belong in the private workspace home, separate from immutable release files. Audit access/export/deletion; access to a project's shared chat does not confer ownership of private credentials or automation state. Legal erasure must explicitly reconcile retained dedupe metadata and backups instead of silently allowing immediate replay.

## Recovery decision table

| Observation | Safe first response |
| - | - |
| Unknown booking/send outcome or missing operation result | Pause; reconcile against the external system. Never infer failure from a lost response. |
| Expired claim before dispatch | Allow only policy-approved bounded retry after fencing; validate no intent/effect exists. |
| Executor restart / lease expiry after dispatch | Reconnect to daemon journal; verify generation and process quiescence. Otherwise needs-attention. |
| Missing/changed release manifest or incompatible old daemon | Stop and repair the approved release/image. Do not execute from a mutable checkout. |
| Secret expiry, CAPTCHA/MFA, revoked membership | Pause and request owner repair/reapproval; do not bypass checks. |
| PostgreSQL unavailable | Reject new durable admission with retryable failure; never acknowledge dropped work. |
| Disk full / log cap / OOM | Stop admission, preserve ledger and journals, add safe capacity, reconcile effects before resuming. |
| Rate/quota exhaustion | Back off, inspect owner attribution/fairness and retry timing. Do not raise limits without a new capacity review. |

Killing a process cannot undo external effects. Duplicate notifications after a provider accepted a send but lost its response are possible independently of booking deduplication. A successful process exit is not proof of a booking: require remote confirmation evidence.

## Backup and paused restore

Back up PostgreSQL, workspace homes (including SQLite WAL/checkpoints and daemon journals), artifact storage, and the encryption key as a coordinated recovery set. Quiesce admission/dispatch and application writers or use an application-consistent SQLite backup; a raw copy of a live DB file alone is insufficient. Encrypt backups, restrict access, test integrity and document timestamps, image versions and RPO/RTO measurements. Never publish secret-bearing diagnostics as artifacts.

Restore into an isolated namespace with all feature flags off, empty owner allowlist and no executor or private services. Before starting any restored workspace, persist automations paused, schedules not enabled, webhook endpoints non-executing, services desired stopped with expired authorization, and queued/running work quarantined for reconciliation. Invalidate restored delegated tokens, preserve tombstones, and verify no old service authorization is accepted. An environment kill switch alone is not a durable paused restore.

Compare external reservation/action state with restored intent/results; a backup predating remote success cannot determine that success did not occur. Never automatically replay queued events, missed schedules or unknown commands. Have the owner review fresh triggers and new approvals after reconciliation. Do not restore old dependency releases into an already-running workspace.

## Emergency disable and rollback

1. Persist owner pause/disable and stop desired services; use the operator kill switch for new admission/dispatch. It is not an instant remote process kill.
2. Persist cancellation for active runs, observe process-group shutdown, and quarantine unknown outcomes. Observe service authorization expiry; inspect actual processes.
3. Drain and scale down the executor; disable triggers/inference/services, set `automations.allOwners: false`, clear the owner allowlist, then disable automation/durable flags. Render the exact rollback manifests and validate them before applying.
4. Roll back to a recorded compatible image/chart only after checking schema/protocol compatibility. Keep additive migrations, run/operation history and dedupe state; never apply destructive down-migrations.
5. Verify no new dispatch, no public workspace route, retained owner read/export/disable access, and no restored service restart. Re-enable only with an audited reconciliation and separate capacity decision.

See [backup and restore](/operate/backup-restore) and [upgrades](/operate/upgrades). Run `bun scripts/test-automation-chart.ts` and the explicit `bun --no-env-file scripts/test-automations.ts` integration harness before approving an upgrade. The latter creates its own disposable PostgreSQL and rejects missing or skipped required suites; it is distinct from legacy empty integration commands.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.