Cubicle

Troubleshooting

Fixes for Cubicle errors listed by the exact message you see, covering CLI sign-in and API key errors, permission and rate-limit refusals, capacity, Box states such as failed and restartRequired, SSH readiness and CLI installation, with an index of the SSH, private network, console, command, snapshot, hibernation and member errors that their guides fix.

Find the message you see, then follow its fix. Headings quote it as the CLI, the installer, an SDK error or a Box's fields show it. Errors that belong to one feature are fixed in its guide, and Errors fixed on other pages lists them by message. CLI errors, API errors and SDK errors show how each one reports an error.

Signing in

not logged in; run `cubicle login`

The CLI has no stored login on this computer. You have not signed in here, you ran cubicle logout, CB_CLI_CONFIG_DIR points at a different directory than the one you signed in with, or the CLI cannot read the token from the credential store it saved it in (the macOS Keychain or secret-tool).

cubicle login

For unattended use, such as CI, use an API key instead of a login:

CB_CLI_API_URL=https://api.cubicle.sh CB_CLI_API_KEY=cbk_… cubicle boxes list --org acme

login expired; run `cubicle login`

Your login session has ended. Sessions last 30 days from sign-in and are not extended by use. They end earlier when you revoke them (cubicle sessions revoke, or Account → Sessions in the Dashboard), when you set a new password, or when your account is disabled. The private network disconnects when its login ends. The HTTP API returns 401 unauthenticated with the same message.

cubicle login
cubicle connect    # only if you use the private network

CB_CLI_API_KEY requires CB_CLI_API_URL

You set an API key for the CLI without the API it belongs to. Set both:

export CB_CLI_API_URL=https://api.cubicle.sh
export CB_CLI_API_KEY=cbk_…

API key is invalid, expired or revoked

The API returns 401 unauthenticated for a key that was revoked, has passed its expiry, or belongs to a service account that was deleted or removed from its organization. Keys cannot be renewed. An owner or admin, signed in interactively, creates a new one:

cubicle api-keys create --org acme ci github-actions --scope write

Key names are unique among an account's keys that are not revoked, including expired ones. If the old key had the same name, revoke it first with cubicle api-keys revoke --org acme ci KEY_ID; cubicle service-accounts list --org acme shows key IDs. A service account that was deleted or removed cannot get new keys (api-keys create fails with 404 not_found: service account not found): create a new one with cubicle service-accounts create --org acme ci, give it a key, and grant it its Boxes again, since a new account has none of the old grants. See Run CI jobs in Boxes.

cannot reach https://api.cubicle.sh: …

The CLI got no response within 30 seconds, or could not connect. Check your network connection and any proxy, and check the URL in the message. It is the API you signed in to with cubicle login, or CB_CLI_API_URL when you use an API key.

API keys

api_key_not_allowed: API keys cannot use this route

HTTP 403. You used an API key on a route that needs an interactive sign-in. API keys work only with /v1/me and the Box, base image, snapshot and operation routes; see Routes that accept API keys. In the CLI, the commands that need a login are connect, ssh, ssh-config, grants, members, keys, sessions, service-accounts, api-keys and devices. A malformed cbk_ token, such as one cut short, can get this error on any route, so check that the key is complete.

unset CB_CLI_API_KEY
cubicle login

Service accounts cannot use SSH or the private network. Run commands with exec instead, and have an owner or admin grant Boxes from an interactive login.

insufficient_scope: this API key is read-only

HTTP 403. A read key can only make GET requests, so it cannot create, start, stop, hibernate, delete, exec, snapshot or restore. A key's scope cannot change. Create a write key, revoking the read key first if you reuse its name:

cubicle service-accounts list --org acme    # find the read key's ID
cubicle api-keys revoke --org acme ci KEY_ID
cubicle api-keys create --org acme ci github-actions --scope write

Permissions

forbidden: an explicit box grant is required

HTTP 403. Exec and the desktop console need an explicit grant on the Box, even for the organization's owner and admin. An owner or admin adds the grant:

cubicle members list --org acme             # find the user's ID
cubicle grants add acme/job-1 USER_ID

For a service account, take its ID from cubicle service-accounts list --org acme.

Two related refusals:

  • forbidden: box grant or organization admin role required: a member without a grant tried to start, stop, hibernate, delete, snapshot or restore the Box. Ask for a grant.
  • forbidden: organization admin role required: only an owner or admin can add grants and manage members and service accounts.

See Box grants for the rules and Grant access to a Box for the steps.

Rate limits

rate_limited: Too many requests; retry later

HTTP 429. The user or service account that sent the request has spent its request budget, which all API keys of one service account share. Retry-After and error.retryAfterSeconds give the seconds to wait.

  • Wait Retry-After seconds before the next request.
  • Poll less often, and share one Cubicle client between waits so maxPollsPerSecond paces them together.
  • The SDK already retries a 429. When it gives up, it throws a CubicleApiError with code rate_limited and retryAfterSeconds set.

Every refused request also counts against your IP address. Keep retrying without waiting and the address is blocked.

blocked: this address is blocked

HTTP 403. Too many refused requests came from your IP address, so it is blocked for everyone behind it, such as a shared office network or CI runners. Retry-After and error.retryAfterSeconds give the seconds left.

  • Stop sending requests from that address until Retry-After has passed. The SDK does not retry a 403.
  • Fix the client that kept retrying, so that it waits for Retry-After after a 429.
  • A block without Retry-After does not expire by itself. Ask a Cubicle operator to lift it.

Creating a Box

capacity: no healthy node with the default base image has capacity for this box

HTTP 503. No healthy Node that offers the Box's base image has the free vCPUs, memory and disk the Box asks for, so nothing was created. The message names the base image you chose. Try again later, or create a smaller Box. The SDK retries it like any 503, then throws a CubicleApiError with code capacity; see Retries.

The same code comes with two other messages:

  • the node holding this snapshot is unavailable or the node holding this snapshot has no capacity for this box: a Box created from a snapshot can only run on the Node that holds the snapshot. See Snapshot and restore a Box.
  • free boxes are at capacity right now; try again shortly: a start of a free Omarchy Box found no room on its Node for the vCPUs and memory it gave back while stopped or hibernated. Try again shortly.

Box state

conflict: box is not running

HTTP 409. Exec needs a Box whose requested state is running; this one is stopped or hibernated. Start it, wait until ssh.ready is true, then run the command again. See Wake a stopped or hibernated Box.

// wake starts a stopped or hibernated Box and waits until it is ready
const result = await box.exec(["uname", "-a"], { wake: true })
console.log(result.exitCode)
// => 0

State stays pending, provisioning or unknown

The Box is waiting on its Node, which has not reported it yet, is creating its disk, or has not reported recently; see What each state means. Requests you make meanwhile are recorded and applied when the Node reports again, and waits give up with timed out waiting for the box in the CLI or a CubicleTimeoutError (code wait_timeout) in the SDK.

If a Box stays in one of these states, ask a Cubicle operator and give them its ID.

box failed: <error>

The Box's state is failed, and observed.error says why. The CLI's --wait prints that error. A failed start is not retried until you start the Box again. See Failures are reported, not retried.

observed.errorCauseFix
box exited unexpectedly; start it againThe guest shut down, rebooted or crashed. A sudo reboot over SSH ends here as well.Start it.
guest did not become reachable: …The guest did not finish booting within the boot timeout, 180 seconds by default.Start it, and watch the boot in the console.
Saved VM configuration is incompatible; Stop then Start for a cold bootSaved memory from hibernation cannot be resumed on the Node as it is now.Stop it, discarding the memory, then start it. See Stop, hibernate and wake.
disk is smaller than its source imageThe requested diskGiB is smaller than the base image.Delete it and create it again with a larger disk.

Until its Node takes up a new start or stop, the Box still reports the earlier failure. The CLI's --wait stops at the first failed report it reads, even that leftover one, so after a failure send the request without --wait and check cubicle boxes list instead:

cubicle boxes stop acme/job-1      # only for a saved-memory error
cubicle boxes list --org acme      # repeat until it shows stopped
cubicle boxes start acme/job-1
cubicle boxes list --org acme      # repeat until it shows running and ssh=ready

The SDK's waits look past earlier reports: cubicle.boxes.start() rejects with a CubicleBoxError (code box_failed) only if this start fails too.

restartRequired: true

The Box's host restarted, so the Box is held stopped with its memory lost, and nothing starts it again automatically. Until you start or stop it, ssh.reason and console.reason are restart_required (ssh=restart_required in cubicle boxes list), and SDK readiness waits reject with a CubicleBoxError whose code is restart_required.

cubicle boxes start acme/job-1 --wait

See A host restart needs an explicit start.

ssh=<reason> or (not ready: <reason>)

SSH is not ready. cubicle boxes list shows the Box's own reason as ssh=<reason>. cubicle ssh-config also checks your own access, so its (not ready: <reason>) can name a reason that concerns you rather than the Box, such as grant_inactive or no_active_key. When a Box is ready gives every reason with its Dashboard wording and what to do, and SSH into a Box covers the SSH errors themselves.

Installing the CLI

Checksum mismatch; the download may have changed during an update. Try again.

The archive did not match its published SHA-256 checksum, for example because a new release was published while you downloaded it. Run the installer again:

curl -fsSL https://api.cubicle.sh/downloads/install | sh

If it keeps failing, check whether a proxy on your network alters downloads.

Cubicle supports macOS and Linux, not …

The CLI runs only on macOS and Linux, on x64 and arm64 processors. Windows is not supported. Another processor gets Unsupported processor …. Use a macOS or Linux computer for the CLI, SSH and the private network. The Dashboard, the TypeScript SDK and the HTTP API do not need the CLI.

Run as your normal user, not with sudo; only the optional tunnel needs administrator rights.

You ran the installer with sudo. Run it as yourself; the tunnel step asks for sudo on its own:

curl -fsSL https://api.cubicle.sh/downloads/install | sh

Errors fixed on other pages

These errors belong to one feature or one surface. The page on the right explains the cause and the fix.

MessageWhere to fix it
box has no trusted host key or guest account yet (<reason>); retry when it is runningSSH into a Box
Permission denied (publickey).SSH into a Box
ssh: Could not resolve hostname …SSH into a Box
the host key for jevbox-<box ID> changed. Verify the box was restored or replaced, then rerun with --accept-new-host-keySSH into a Box
WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!SSH into a Box
the Cubicle tunnel is not installed; run `cubicle tunnel install`Private network
the Cubicle tunnel helper is not responding; reinstall it with `cubicle tunnel install`Private network
VPN startup timed out; inspect vpn.logPrivate network
confirmation required; pass --yesPrivate network
systemd-resolved is required or systemd-resolved must be activePrivate network
cubicle.internal resolver already existsPrivate network
key_unavailable: device key is already registered or revokedPrivate network
vpn_unavailable: …Private network
Connection refused from a service in a BoxPrivate network
Console requires an explicit grant on this BoxDesktop console
Console unavailable: <reason> or Console is not ready: <reason>Desktop console
Close an existing VNC viewer before opening anotherDesktop console
Disconnected. Open Console from the Dashboard to reconnect.Desktop console
box is not running at the requested generationRun commands
An exec error ending in unknown guest accountRun commands
requester is not authorized on this boxRun commands
exec control interrupted by a Box lifecycle transition; command is not replayed, or exec unknown: … in the CLIRun commands
metal service restarted during exec; outcome unknown and not replayedRun commands
unsupported Unicode escape sequenceRun commands
cubicle: output truncatedSet a timeout and an output limit
payload_too_large: request body too largeMove files
stop the box before taking a snapshot or stop the box before restoring a snapshotSnapshot and restore
box must be stoppedSnapshot and restore
a snapshot or restore is in progress for this boxSnapshot and restore
snapshot not foundSnapshot and restore
snapshot is not completeSnapshot and restore
disk must be at least N GiB (the snapshot's size)Snapshot and restore
choose a base image or a snapshot, not both, or choose --base or --snapshot, not both in the CLISnapshot and restore
box job-2 is still being created from a snapshot of this boxSnapshot and restore
hibernation_failed: box job-1 could not hibernate: …; the Controller retries at …Stop, hibernate and wake
only a running box can hibernateStop, hibernate and wake
the box requires an explicit Start after a host rebootStop, hibernate and wake
box_superseded: box job-1 is now requested …, not …Wait for a state
not_found: organization not foundOrganization members
not_found: user not found or not_found: member not foundOrganization members
box_create_disabled: this organization cannot create boxes; ask an operator to enable itError codes
conflict: a box named job-1 already existsConflicts
idempotency_conflictIdempotency keys
missing_api_url or missing_api_keyRun CI jobs in Boxes

On this page