Cubicle

Waiting

How @cubicle/sdk waits for Boxes and operations: what each call waits for and its default timeout, wait: false, timeoutMs, AbortSignal, waitUntil and waitUntilReady, polling through outages and rate limits with maxPollsPerSecond, and recovering a failed create with idempotencyKey.

The HTTP API answers as soon as it accepts a request, and the SDK polls it until the change has taken effect. To poll the API yourself, see Requests and waiting.

What each call waits for

CallResolves whenResolves withDefault timeout
boxes.createssh.ready is true. With start: false, it returns without waiting.Box5 min
boxes.startssh.ready is true.BoxInfo5 min
boxes.stopstate is stopped at the current generation.BoxInfo5 min
boxes.hibernatestate is hibernated at the current generation.BoxInfo5 min
boxes.deleteThe Box is gone from its organization's listing: its Node removed it and released its CPU, memory and disk.Nothing5 min
boxes.waitUntil, waitUntilReadyThe Box reaches the target.BoxInfo5 min
boxes.execThe command's operation is done.ExecResultThe command's timeoutMs (60 s by default, at most 10 min) plus 2 min
boxes.snapshotThe snapshot operation is done.Snapshot15 min
boxes.restoreThe restore operation is done.Operation15 min
operations.waitThe operation is done.Operation15 min

Handle methods such as box.start() wait in the same way. Ready is ssh.ready, as When a Box is ready defines it, and at the current generation means after the Node applied your request, as Generations explains, so a stopped report from before a start does not count. Waits also ignore a failed report left over from before the wait began.

Return without waiting

Pass { wait: false } to create, start, stop, hibernate or delete to return as soon as Cubicle accepts the request. start, stop and hibernate then resolve with the record the API returned, which shows the new desired.state but not yet the new state. create resolves with a Box handle, and delete with nothing.

const accepted = await cubicle.boxes.stop("acme/job-1", { wait: false })
console.log(accepted.desired.state) // => "stopped"

// … do other work, then:
await cubicle.boxes.waitUntil(accepted.id, "stopped")

exec, snapshot and restore always wait. To run several commands at once, start them all and then await them together:

const [test, lint] = await Promise.all([box.exec(["make", "test"]), box.exec(["make", "lint"])])
console.log(test.exitCode, lint.exitCode)

Change the timeout

timeoutMs, one of the WaitOptions, sets how long a call waits, in ms:

await box.start({ timeoutMs: 10 * 60_000 })
  • It is capped at 2^31 − 1 ms, about 24.8 days.
  • A negative value, or one that is not a number, rejects with invalid_option when the wait begins. The request that starts the change has been sent by then, so a create still creates the Box.
  • For exec, timeoutMs is the command's own timeout. The wait lasts that long plus 2 minutes, and has no separate option.
  • exec with wake: true waits up to 5 minutes for the Box to become ready before it sends the command.

When a wait times out

A wait that runs out of time rejects with a CubicleTimeoutError. Its last field is the most recent BoxInfo or Operation the wait read, or undefined if no read succeeded. The request is not undone: the Box may still get there, and a command may still finish.

import { Cubicle, CubicleTimeoutError } from "@cubicle/sdk"

const cubicle = new Cubicle()

try {
  await cubicle.boxes.start("acme/job-1", { timeoutMs: 60_000 })
} catch (error) {
  if (!(error instanceof CubicleTimeoutError)) throw error
  console.log("not ready yet:", error.last?.state)
  // The start is still in progress: keep waiting.
  await cubicle.boxes.waitUntilReady("acme/job-1")
}

For exec, snapshot and restore, last is the Operation, or undefined if no poll succeeded. Wait for it again with cubicle.operations.wait(error.last.id). When exec with wake: true times out while waiting for the Box, last is the BoxInfo instead.

Stop waiting with a signal

Every call that sends a request takes an AbortSignal as signal. Aborting it stops the wait and any request in flight. The call then rejects with the signal's reason, as fetch does, not with a CubicleError. A request Cubicle already accepted still takes effect: the Box still starts, and a command keeps running.

const signal = AbortSignal.timeout(30_000)

try {
  await box.start({ signal })
} catch (error) {
  if (!signal.aborted) throw error
  // Stopped waiting. The Box is still starting; check it later with box.info().
}

Use timeoutMs when you want a CubicleTimeoutError that carries the last record. Use signal to cancel from outside, for example when your own job is cancelled.

If a signal aborts while create is sending its request, you cannot tell whether the Box was created. Pass your own idempotencyKey so you can repeat the call safely; see Recover a failed create.

Wait for a state

waitUntil(ref, target) polls GET /v1/boxes/{box} until a Box is "ready", "stopped" or "hibernated". waitUntilReady(ref) is short for waitUntil(ref, "ready"). They never change the Box: use them after a call with wait: false, or for a Box that another process is changing.

The Box's desired state must already match the target, running for "ready". Otherwise the wait rejects with box_superseded at once, because the Box will not get there on its own.

A wait that cannot succeed rejects with a CubicleBoxError. Its box field holds the latest record.

codeMeaningWhat to do
box_failedThe Box reported failed after the wait began. box.observed?.error says why.Start it again, or delete it.
restart_requiredThe wait was for ready, such as waitUntilReady()'s, and the Box's host restarted.Start it again. exec with wake: true starts it for you.
hibernation_failedThe wait was for hibernated, such as hibernate()'s, and saving the memory failed. The Box keeps running.Nothing is required: Cubicle retries at box.hibernationRetryAt. Or stop the Box.
box_supersededAnother request changed the Box's desired state, or the Box was deleted.Read the Box again and decide.

Outages and refused polls

A wait polls until its timeout. These failures only delay its next poll:

  • The API cannot be reached, or does not answer within requestTimeoutMs.
  • The API answers 429 rate_limited, 502, 503 or 504.

The delay is the Retry-After the API sent, if any. Otherwise it doubles from 1 s, or from pollIntervalMs if that is longer, up to 30 s. maxRetries does not apply to a wait's polls.

Any other error ends the wait at once, for example not_found for a Box that does not exist, or unauthenticated for a revoked key. Once a wait for a Box state has seen the Box, not_found means the Box was deleted, and the wait rejects with box_superseded.

The request that starts a change, such as the POST for a start, is retried like any other request; see Retries.

Poll pacing and the rate-limit budget

  • Each wait polls every pollIntervalMs, 1 s by default.
  • All waits of one client share one pacer. Together they send at most maxPollsPerSecond polls a second, 5 by default. With 50 concurrent waits, each one polls about every 10 s.
  • Only wait polls are paced. Other requests are sent at once.

All API keys of one service account, in every process, share one rate limit of 600 requests a minute. When several processes use the same service account, lower maxPollsPerSecond in each so that their combined polling stays well under 10 a second:

import { Cubicle } from "@cubicle/sdk"

// One of three workers that share a service account.
const cubicle = new Cubicle({ maxPollsPerSecond: 2 })

Recover a failed create

When the wait in create fails, the Box still exists:

  • CubicleBoxError.box is its record. CubicleTimeoutError.last usually is too; it is undefined if no poll succeeded.
  • Repeating create with the same idempotencyKey and parameters returns the same Box, and waits for it again.

The SDK generates a new key for every call, so pass your own to make a create repeatable, for example across restarts of your job:

import { Cubicle, CubicleBoxError, CubicleTimeoutError } from "@cubicle/sdk"

const cubicle = new Cubicle()
const params = { name: "job-1" }
const options = { idempotencyKey: "nightly-2026-11-05" }

async function createJobBox() {
  try {
    return await cubicle.boxes.create(params, options)
  } catch (error) {
    if (error instanceof CubicleTimeoutError) {
      // Created, but not ready in time: the same key returns the Box and waits again.
      return await cubicle.boxes.create(params, options)
    }
    if (error instanceof CubicleBoxError && error.code === "box_failed") {
      // The Box failed to start. Remove it, and use a new key for the next attempt.
      await cubicle.boxes.delete(error.box.id)
    }
    throw error
  }
}

const box = await createJobBox()

After you delete the Box, use a new key: the old key keeps returning the deleted Box. The rules for keys are in Idempotency keys.

On this page