Move files into and out of a Box

Cubicle has no file upload or download API yet. Move files with the tools you already use:

API keys cannot use SSH, so programs use exec or a URL.

Before you beginCopy Anchor Link

• Over SSH: the cubicle CLI with the tunnel installed, an SSH key on your account, a grant on the Box and a running Box. The SSH quick start sets these up.
• With exec: the SDK or the CLI, set up as in the Quickstart, with a write key or a CLI login, a grant on the Box (its creator gets one) and a running Box. The examples use acme/job-1.
• Files belong to the guest account that writes them, and each person and service account has its own. Relative paths start in that account's home directory.

Copy files over SSHCopy Anchor Link

Write a file with execCopy Anchor Link

Exec has no standard input, so a file's content travels inside the command's argv, within exec's limits:

• The whole request body is at most 64 KiB (65,536 bytes) after JSON encoding. Escaped characters, such as quotes, backslashes and newlines, take 2 bytes or more, and non-ASCII characters 2 to 4.
• argv is at most 1024 strings of up to 65,536 characters each.
• argv strings are text. They cannot carry NUL bytes or bytes that are not valid UTF-8, so encode anything other than plain text with base64. Base64 is a third larger than its data: 48 KiB of base64 carries 36 KiB.

Pass the content and the path as separate arguments after an sh -c script, which reads them as $1 and $2, so neither needs quoting:

const config = JSON.stringify({ retries: 3 }, null, 2)

await box.exec(["sh", "-c", 'printf %s "$1" > "$2"', "sh", config, "config.json"])

Read a file with execCopy Anchor Link

The file comes back in the command's output:

• stdout is decoded as UTF-8, and invalid bytes become U+FFFD (�). Read text with cat, and anything else with base64 -w 0; Read the result says why.
• stdout and stderr share one maxOutputBytes budget: 64 KiB by default, at most 1 MiB. Output past it is discarded and truncated is true, so always check it.
• Base64 is a third larger than its data, so one exec with a 1 MiB budget returns at most 768 KiB of a file.

const summary = await box.exec(["cat", "results/summary.txt"])
if (summary.exitCode !== 0 || summary.truncated) throw new Error(summary.stderr || "too long")
console.log(summary.stdout)

const plot = await box.exec(["base64", "-w", "0", "results/plot.png"], {
  maxOutputBytes: 1_048_576, // 1 MiB of base64, so up to 768 KiB of data
})
if (plot.exitCode !== 0 || plot.truncated) throw new Error(plot.stderr || "over 768 KiB")
const png = Buffer.from(plot.stdout, "base64")

Move larger files in partsCopy Anchor Link

This helper writes and reads files of any size through exec. It sends 36 KiB parts, each written at its own offset so that sending one again is harmless, and reads 768 KiB parts. It compares SHA-256 digests at the end, so a file that arrives incomplete or changes while it is read raises an error. It uses sh, base64, dd, truncate, stat and sha256sum in the Box, which both base images include.

import { createHash } from "node:crypto"
import { readFile, writeFile } from "node:fs/promises"
import { Cubicle, type BoxRef } from "@cubicle/sdk"

// 36 KiB of data is 48 KiB of base64, which leaves room for the path in the 64 KiB request body.
const WRITE_PART = 36 * 1024
// 768 KiB of data is exactly 1 MiB of base64, the most output that one exec returns.
const READ_PART = 768 * 1024

/** Runs a script with `sh -c`, and returns its stdout unless it failed or was cut. */
async function sh(box: BoxRef, script: string, args: string[], maxOutputBytes?: number) {
  const result = await box.exec(["sh", "-c", script, "sh", ...args], { maxOutputBytes })

  if (result.exitCode !== 0 || result.timedOut || result.truncated) {
    throw new Error(`exec ${result.operationId} exited ${result.exitCode}: ${result.stderr}`)
  }

  return result.stdout
}

const sha256 = (data: Uint8Array) => createHash("sha256").update(data).digest("hex")

/** Writes `data` to `path` in the Box, and checks the Box's copy. */
async function writeBoxFile(box: BoxRef, path: string, data: Uint8Array) {
  for (let offset = 0; offset < data.length; offset += WRITE_PART) {
    const part = Buffer.from(data.subarray(offset, offset + WRITE_PART)).toString("base64")
    await sh(
      box,
      'printf %s "$1" | base64 -d | dd of="$2" seek="$3" oflag=seek_bytes conv=notrunc status=none',
      [part, path, String(offset)],
    )
  }

  // Set the exact length, which also creates an empty file, then compare digests.
  const stdout = await sh(box, 'truncate -s "$2" "$1" && sha256sum < "$1"', [
    path,
    String(data.length),
  ])

  if (stdout.split(" ")[0] !== sha256(data)) throw new Error(`${path} in the Box does not match`)
}

/** Reads `path` from the Box. */
async function readBoxFile(box: BoxRef, path: string): Promise<Buffer> {
  const [size, digest] = (await sh(box, 'stat -c %s "$1" && sha256sum < "$1"', [path])).split(/\s+/)
  const parts: Buffer[] = []

  for (let offset = 0; offset < Number(size); offset += READ_PART) {
    const part = await sh(
      box,
      'dd if="$1" skip="$2" count="$3" iflag=skip_bytes,count_bytes status=none | base64 -w 0',
      [path, String(offset), String(READ_PART)],
      1_048_576,
    )
    parts.push(Buffer.from(part, "base64"))
  }

  const data = Buffer.concat(parts)

  if (sha256(data) !== digest) throw new Error(`${path} changed while it was read`)

  return data
}

const cubicle = new Cubicle()
const box = cubicle.boxes.ref("acme/job-1")

await writeBoxFile(box, "input.tar.gz", await readFile("input.tar.gz"))
await writeFile("report.pdf", await readBoxFile(box, "results/report.pdf"))

• The Box must be running. Pass wake: true in sh if it may have hibernated.
• Writing a 1 MiB file takes 30 execs, and reading it takes 3. Each exec is one non-GET request, and a service account may send 120 of those a minute (Rate limits): a 4 MiB write takes 115 execs. Move larger files through a URL.
• The directory must already exist. Create it first with mkdir -p.

Move large files through a URLCopy Anchor Link

Boxes can open connections to the public IPv4 internet (outbound traffic), so a command in the Box can download its input from a URL and upload its results to one. With object storage, create short-lived presigned URLs in your program and pass them to the Box. Your storage credentials and your API key stay out of the Box.

The examples use Python 3's standard library: both base images include /usr/bin/python3. One exec runs for at most 10 minutes, so each transfer must finish within its timeoutMs.

// Presigned URLs from your storage service: a GET URL for the input, a PUT URL for the output.
const { INPUT_URL, OUTPUT_URL } = process.env
if (!INPUT_URL || !OUTPUT_URL) throw new Error("set INPUT_URL and OUTPUT_URL")

const download =
  'import shutil, sys, urllib.request; shutil.copyfileobj(urllib.request.urlopen(sys.argv[1], timeout=60), open(sys.argv[2], "wb"))'
const upload =
  'import os, sys, urllib.request; f = open(sys.argv[2], "rb"); urllib.request.urlopen(urllib.request.Request(sys.argv[1], data=f, method="PUT", headers={"Content-Length": str(os.path.getsize(sys.argv[2])), "Content-Type": "application/octet-stream"}), timeout=60)'

const fetched = await box.exec(["python3", "-c", download, INPUT_URL, "input.tar.gz"], {
  timeoutMs: 600_000,
})
if (fetched.exitCode !== 0) throw new Error(fetched.stderr)

// … run the job, which writes results.tar.gz …

const sent = await box.exec(["python3", "-c", upload, OUTPUT_URL, "results.tar.gz"], {
  timeoutMs: 600_000,
})
if (sent.exitCode !== 0) throw new Error(sent.stderr)

• When the server refuses a request, the command exits 1 with Python's error in stderr, such as urllib.error.HTTPError: HTTP Error 403: Forbidden, and a download creates no file. A server that refuses an upload before reading it can instead end it with urllib.error.URLError, such as [Errno 32] Broken pipe.
• The upload sets Content-Length from the file's size. If you signed the URL for a particular content type, send that type instead of application/octet-stream.
• Anyone who has a presigned URL can use it until it expires, so give it a short expiry time.
• Any other HTTP client in the Box works the same way.

VerifyCopy Anchor Link

Compare SHA-256 digests on both sides. The helper above does this itself. Over SSH:

ssh cubicle-acme-job-1 sha256sum data.csv
# 3b5d…  data.csv
shasum -a 256 data.csv   # on Linux: sha256sum data.csv
# 3b5d…  data.csv

With exec, run ["sha256sum", "data.csv"] and compare its stdout with the digest of your copy.

TroubleshootingCopy Anchor Link

• 413 payload_too_large "request body too large": the exec request is over 64 KiB. Send the content in smaller parts, as the helper does.
• truncated: true, or cubicle: output truncated: the output went past maxOutputBytes. Read the file in parts, or raise maxOutputBytes to at most 1 MiB from the SDK or the API.
• Replacement characters (�) in stdout: the file is not UTF-8 text. Read it with base64 -w 0 and decode it.
• dd: failed to open '…': No such file or directory: the directory does not exist in the Box. Create it with mkdir -p first.
• Permission denied: the guest account cannot write there. Write under its home directory or /tmp. Commands sent with an API key cannot use sudo; see Root and sudo.
• HTTP Error 403 from a URL, or Broken pipe on an upload: the presigned URL has expired, or was signed for another method or content type. Create a new one.
• A download or upload times out or cannot connect: the URL's host must have a public IPv4 address. Boxes cannot reach IPv6 or private address ranges.
• rsync fails because the remote command is not found: rsync is not installed in the Box. Install it over SSH, or use scp -r.
• ssh: Could not resolve hostname …: run cubicle ssh-config for the Box and cubicle connect. SSH troubleshooting lists the other causes.

RelatedCopy Anchor Link

• SSH into a Box: the alias, keys, grants and host keys.
• Run commands: exec's options, results and errors.
• Limits and defaults: every exec limit in one table.
• Networking: what a Box can reach.
• Give an AI agent a Box: run an agent's commands with exec.