Skip to main content
Each user gets one workspace: a Deployment that scales between zero and one, plus permanent supporting objects. This page covers how it behaves in production and how to operate it.

​
Resource names

Names derive from a SHA-256 hash of the user ID, so they are stable and reveal nothing about the user:
All carry the labels app.kubernetes.io/managed-by: r5d-chat, app.kubernetes.io/name: r5d-chat-workspace, r5d.chat/workspace, and r5d.chat/user-id.

​
Starting a workspace

1

A request needs the workspace

A chat run, file browse, upload, preview, or command triggers it.
2

Objects are reconciled

The PVC, identity Secret, and Service are created if missing, then the Deployment is set to one replica. Concurrent requests for the same user share one in-flight reconciliation.
3

Readiness is awaited

The control plane polls until the daemon’s readiness probe passes, up to 90 seconds, then returns a 503 with workspace_start_timeout.
4

Work proceeds

Requests are proxied to the daemon over the Service with the workspace’s bearer token.
A cold start typically takes a few seconds, dominated by image pull on first use and volume attach. Kata adds VM boot time.

​
Chat responses do not wait for a cold start

A chat response starts its user’s workspace in the background, alongside the model request, instead of before it:
  • The model starts answering immediately. A response that uses no workspace tool never waits for the workspace.
  • A workspace tool (files, search, or shell) waits for readiness only when the model calls it, before the call is recorded or sent. Stopping the response while a tool waits cancels that call; the workspace keeps starting, so the next turn finds it warm.
  • If the start fails, the waiting tool call fails with the reason, such as Workspace did not become ready in time, and the model can explain the failure. The next workspace tool call tries to start the workspace again. The control plane, or the executor for durable runs, also logs a workspace.start.failed event.
See Troubleshooting for the response timing log that separates each phase of a slow first response.

​
Activity and leases

Two mechanisms keep a busy workspace alive:
  • Activity timestamps update on every workspace operation.
  • Active leases are held by long-running work. Starting a shell command increments the lease count; polling to completion or cancelling decrements it.
A reconciler runs every 60 seconds and scales to zero only when the desired state is running, the lease count is zero, and the last activity is older than WORKSPACE_IDLE_SECONDS.
Because a running shell command holds a lease, a long build is never reaped mid-task even if it exceeds the idle timeout.

​
Scaling to zero

Scale-down sets the Deployment to zero replicas. Everything else stays: the Service, identity Secret, database records, and the PVC with all files. The next request scales the same Deployment back to one and remounts the same home directory. Set workspace.idleSeconds according to your tradeoff between cost and cold starts. 1800 seconds is a reasonable default; interactive-heavy installations often prefer 3600.

​
Inspecting workspaces

Exec into a running workspace to verify its toolchain:

​
Installing dependencies

Kubernetes workspaces run as UID 0 inside the container. sudo is not installed or needed: use apt-get update && apt-get install -y <package> directly. APT still uses its _apt download user; the security policy grants only the user-switching and file-management capabilities needed for this. It does not provide host root, privileged mode, host devices, or service managers. Local daemon deployments use the operator’s own permissions instead. Only /home/r5d is persistent. Keep Python virtualenvs and Node project dependencies there; system packages installed into /usr, /etc, or /var must be checked/reinstalled after container replacement (including an idle stop/wake). For frequently needed system dependencies, bake them into the workspace image. Installing packages also requires permitted network egress and sufficient disk. A control-plane upgrade does not restart a running workspace or replace its security policy. An existing workspace receives the new template only on a wake after it is fully stopped. Let active work finish and use the normal idle lifecycle, or arrange an owner-approved stop/wake; do not force a rollout that kills work. No home-volume recreation is required. To verify package support without touching a live workspace, use a locally built workspace image and the disposable Docker smoke test (it contacts Debian mirrors, never Kubernetes, and mounts no host files or credentials):
The test uses the same security policy as the workspace controller, verifies apt-get update, install/remove, ownership and maintainer scripts, and records its exact container and cleanup status in an ignored per-run manifest.

​
Storage

Home volumes are ReadWriteOnce and mounted whole at /home/r5d. They are private working directories and are not the source of truth for shared artifacts. Project/chat artifacts are remote control-plane resources stored on the separate chart-managed artifact PVC and accessed with r5dchat artifacts. The chart never uses subPath, which keeps both home and artifact mounts compatible with Kata. workspace.storageSize applies when a PVC is created. Changing it affects only new workspaces. To grow an existing volume, your storage class must allow expansion:
Some drivers require the workload to be scaled to zero first, and some require a restart to grow the filesystem.
Workspace PVCs are annotated with helm.sh/resource-policy: keep. They survive helm uninstall and must be deleted explicitly. Deleting one destroys that user’s files permanently.

​
Removing a user’s workspace

Deleting workspace data should be a deliberate, audited action:
The control plane recreates a clean workspace on that user’s next request.

​
Troubleshooting

The pod failed to become ready within 90 seconds. Check pod events for image pull failures, volume attach problems, or insufficient quota:
workspace.imagePullSecret is unset while the image is private. The control plane’s own imagePullSecrets does not apply to workspace pods.
Usually no node can satisfy the volume or the resource requests. Check that the storage class exists and provisions ReadWriteOnce volumes, and confirm namespace quota headroom.
Leases may be stuck above zero from processes that never completed, or the control plane may be failing its reconcile loop. Check control-plane logs and confirm WORKSPACE_IDLE_SECONDS is what you expect.
The container exceeded its memory limit. Raise workspace.resources.limits.memory and confirm with:
Private files outside /home/r5d are not persisted. Only the home directory is on this workspace’s volume; everything else resets when the pod restarts. Published artifacts persist independently in control-plane artifact storage.