Storage Mounts

A storage mount shows a bucket, or one prefix inside it, as a folder in a sandbox. Programs in the sandbox read and write the objects as ordinary files. Tenki runs the mount on the sandbox's host, so bucket credentials never enter the sandbox. Tenki gives a mount's credentials only to the host that runs the sandbox, and stops once the sandbox is paused or terminated. The mount also keeps working when the sandbox has no outbound network access. With access keys, the host uses the key itself, so a mount's prefix and read-only setting limit what the sandbox can do, not what the key allows. Give each connection a key limited to the buckets and actions its mounts need.

Storage mounts work with Amazon S3, Cloudflare R2, Google Cloud Storage, and S3-compatible services such as MinIO, Wasabi, Backblaze B2, and Tigris.

Storage mounts are in preview and are enabled per workspace. Contact Tenki support to turn them on for yours. The Storage tab under Settings appears once they are on.

Use a storage mount for data that already lives in a bucket or that other tools read, such as datasets, model weights, build artifacts, and logs. For caches and other data that only sandboxes use, a persistent volume is faster and behaves like a local disk.

Setting up a bucket takes three steps:

1. Add a storage connection for the bucket, once per workspace.
2. Test the connection.
3. Mount it as a default mount for every new sandbox, when you create a sandbox, or on a running sandbox.

The CLI commands are under tenki sandbox storage and need Tenki CLI 2.10.0 or later. The SDK methods are in 1.4.3 or later of @tenkicloud/sandbox for TypeScript and tenki for Python, and 1.4.4 or later of github.com/LuxorLabs/tenki-sdk-go/sandbox for Go.

Storage connections

A storage connection records the provider, bucket, endpoint, region, default access, and how Tenki signs in to the bucket. Default access is read-only or read-write. A mount uses it unless the mount sets its own access.

Tenki stores credential values as Tenki Secrets in the s3_storage scope, and the connection keeps only the secret names. Tenki never passes s3_storage secrets to CI jobs or into sandboxes. They always belong to the whole workspace and cannot be moved to a repository.

Because connections write and delete secrets, adding, updating, or deleting one needs the same access as the Secrets API. Your account needs workspace-edit permission, and an API key or CLI login needs Allow Secrets access. Listing and testing connections, editing default mounts, and mounting buckets do not need Secrets access.

Add a connection

In the dashboard:

1. Open Settings > Storage and click Add connection.
2. Enter a name, pick the default access and the provider, and enter the bucket and the provider's fields.
3. Pick the authentication method and fill in its fields. For each credential, choose Enter value to store a new secret, or Use secret to pick an existing s3_storage secret.
4. Click Add connection.

With the CLI, credential values come from environment variables or files, never from command arguments. Each credential has three flags, and you pass one of them:

• --<field>-env NAME stores the value of the environment variable NAME as a new secret.
• --<field>-file PATH stores the contents of a file as a new secret.
• --<field>-secret NAME uses an existing s3_storage secret.

The fields are access-key-id, secret-access-key, session-token, and r2-parent-token. This example adds a read-only connection to an R2 bucket with access keys:

import { TenkiSandbox } from "@tenkicloud/sandbox";

const sandbox = new TenkiSandbox();

const connection = await sandbox.createStorageConnection({
  name: "media",
  provider: "R2",
  endpoint: "https://<account-id>.r2.cloudflarestorage.com",
  bucket: "media",
  authMethod: "ACCESS_KEY",
  defaultAccess: "READ_ONLY",
  credentials: {
    accessKeyId: { value: process.env.R2_ACCESS_KEY_ID! },
    secretAccessKey: { value: process.env.R2_SECRET_ACCESS_KEY! },
  },
});

Tenki names the new secrets after the connection, for example STORAGE_MEDIA_ACCESS_KEY_ID and STORAGE_MEDIA_SECRET_ACCESS_KEY, and records which secrets it created for the connection. Entering a value never overwrites a secret Tenki did not create for this connection. List connections with tenki sandbox storage connections list, or with listStorageConnections() in TypeScript, client.storage_connections.list() in Python, and ListStorageConnections in Go.

Keyless AWS

With a keyless IAM role, Tenki stores no AWS keys. The role is an IAM role in your own AWS account, and its policies decide what Tenki can reach. For each mount, Tenki's OIDC issuer signs a short-lived token, AWS STS exchanges it for short-lived credentials, and Tenki renews them while the mount runs. Tenki also attaches a session policy to those credentials, so each mount reaches only its own bucket, prefix, and access, even when the role allows more.

1. Pick a name for the role, such as tenki-storage. Its ARN will be arn:aws:iam::<your-account-id>:role/tenki-storage.
2. Add the connection in Tenki with provider Amazon S3, authentication Keyless IAM role (recommended), the bucket, its region, and that role ARN. The dashboard then opens Trust Tenki from your AWS account, which shows Tenki's provider URL, the audience, and the trust policy for this connection. To open it again later, choose Show trust policy from the connection's menu.
3. In the AWS IAM console, add an OpenID Connect identity provider with Tenki's provider URL and the audience sts.amazonaws.com. One identity provider per AWS account serves every connection.
4. Create the role with the name you picked, choose Custom trust policy, and paste the trust policy Tenki showed.
5. Give the role a permissions policy for the bucket, like the one below.
6. Back in Tenki, test the connection.

With the CLI, step 2 looks like this:

tenki sandbox storage connections create datasets \
  --provider aws \
  --region us-east-1 \
  --bucket my-datasets \
  --auth aws-role \
  --role-arn arn:aws:iam::123456789012:role/tenki-storage \
  --default-access ro
tenki sandbox storage connections list --json

The connection's oidc_subject_pattern and the list's oidc_issuer and oidc_audience fill in the trust policy. Use the issuer without https:// as <issuer-host>:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Federated": "arn:aws:iam::123456789012:oidc-provider/<issuer-host>" },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": { "<issuer-host>:aud": "sts.amazonaws.com" },
        "StringLike": { "<issuer-host>:sub": "workspace:<workspace-id>:storage:<connection-id>:session:*" }
      }
    }
  ]
}

The subject names your workspace and this connection, so another workspace that learns the role ARN cannot assume the role. Keep the condition exactly as Tenki shows it. A broader sub pattern lets other connections, or other workspaces, use the role.

This permissions policy allows read-write mounts of my-datasets:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::my-datasets"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::my-datasets/*"
    }
  ]
}

Read-only mounts use only s3:ListBucket, s3:GetBucketLocation, and s3:GetObject, so drop the other actions if you never mount the bucket read-write.

In the SDKs, set the authentication method to AWS_OIDC_ROLE and pass only the role ARN: credentials: { roleArn } in TypeScript, StorageConnectionCredentials(role_arn=...) in Python, and StorageConnectionCredentials{RoleARN: ...} in Go.

AWS access keys

Use access keys when you cannot add an identity provider to the AWS account. Create an IAM user with the permissions policy above, create an access key for it, and add the connection with authentication Access keys.

tenki sandbox storage connections create datasets \
  --provider aws \
  --region us-east-1 \
  --bucket my-datasets \
  --default-access ro \
  --access-key-id-env AWS_ACCESS_KEY_ID \
  --secret-access-key-env AWS_SECRET_ACCESS_KEY

Tenki derives the endpoint from the region, so leave the endpoint empty. Tenki still limits each mount to its prefix and access on the host, but the key itself can do whatever its IAM policy allows, so limit that policy to the bucket.

The optional session token is for temporary AWS credentials. Tenki cannot renew those, and mounts fail once they expire, so prefer a long-lived key or a keyless role.

Cloudflare R2

An R2 endpoint has the form https://<account-id>.r2.cloudflarestorage.com, where the account ID is the 32-character ID in the Cloudflare dashboard. For a bucket in a jurisdiction, add it to the host, as in https://<account-id>.eu.r2.cloudflarestorage.com. The Tenki dashboard asks for the account ID and jurisdiction and builds the endpoint for you. R2 uses the region auto, so leave the region empty.

R2 connections authenticate in one of two ways.

With Access keys, Tenki uses the access key ID and secret access key of an R2 API token. Create the token in the Cloudflare dashboard under R2 > Manage API tokens, with Object Read & Write, or Object Read only for a read-only connection. The token can be limited to this bucket.

With Scoped temporary keys from an API token, Tenki asks Cloudflare for a short-lived R2 key for each mount, limited to the mount's bucket, prefix, and access. This method needs two values:

1. The access key ID of an account-level R2 API token with Object Read & Write. Temporary keys are derived from it. A token limited to specific buckets does not work, because Cloudflare's temporary credentials API rejects it.
2. A Cloudflare API token with the Workers R2 Storage Write permission, which Tenki uses to request the temporary keys.

Tenki does not need the R2 token's secret access key for this method.

tenki sandbox storage connections create media \
  --provider r2 \
  --bucket media \
  --s3-endpoint https://<account-id>.r2.cloudflarestorage.com \
  --default-access rw \
  --auth r2-temporary \
  --access-key-id-env R2_PARENT_ACCESS_KEY_ID \
  --r2-parent-token-file ~/cloudflare-api-token

In the SDKs, the method is R2_TEMPORARY and the API token goes in the r2ParentToken credential (r2_parent_token in Python, R2ParentToken in Go).

Google Cloud Storage

Tenki reaches Google Cloud Storage through its S3-compatible XML API with an HMAC key.

1. Create a service account and grant it access to the bucket: read access for a read-only connection, or read and write access otherwise.
2. In the Google Cloud console, open Cloud Storage > Settings > Interoperability and create an HMAC key for that service account. Google shows the secret only once, so copy it then.
3. Add the connection with provider Google Cloud Storage and authentication HMAC keys. The key's access ID goes in Access key ID, and its secret in Secret access key.

tenki sandbox storage connections create training \
  --provider gcs \
  --bucket my-training-data \
  --default-access ro \
  --access-key-id-env GCS_HMAC_ACCESS_ID \
  --secret-access-key-file ~/gcs-hmac-secret

Tenki sets the endpoint to https://storage.googleapis.com, and the region can stay empty. Use a service account key rather than one tied to a person, so the connection keeps working when people leave and you can limit the account to the buckets it needs.

S3-compatible

Use provider S3-compatible for MinIO, Wasabi, Backblaze B2, Tigris, and other services that speak the S3 API. Enter the service's endpoint and an access key pair. Set the region when the service signs requests with one, such as us-west-004 for a Backblaze B2 bucket in that region. Without one, the region is auto.

tenki sandbox storage connections create backups \
  --provider s3 \
  --bucket my-backups \
  --s3-endpoint https://s3.us-west-004.backblazeb2.com \
  --region us-west-004 \
  --default-access rw \
  --access-key-id-env B2_KEY_ID \
  --secret-access-key-env B2_APPLICATION_KEY

The endpoint must be a public https:// URL with no path. Saving a connection fails for IP addresses, localhost, and hosts ending in .localhost, .local, or .internal. Tenki also refuses to connect to a host that resolves to a private, link-local, or other non-public address, and a connection test then reports ENDPOINT_BLOCKED. A MinIO server on a private network cannot be mounted. Mounts also cannot reach the addresses of the Tenki host they run on.

Test a connection

Testing checks that Tenki can reach the bucket with the connection's credentials. In the dashboard, choose Test connection from the connection's menu, and the result appears next to its name. From the CLI:

tenki sandbox storage connections test media

The command prints Bucket reachable, or fails with the reason. With --json, it prints reachable, reason, and message, and exits with status 1 when the bucket is unreachable. The SDKs return the same fields from testStorageConnection(id) in TypeScript, client.storage_connections.test(id) in Python, and TestStorageConnection(ctx, id) in Go.

Update or delete a connection

tenki sandbox storage connections update media --default-access rw
tenki sandbox storage connections update media --secret-access-key-file ~/new-r2-secret
tenki sandbox storage connections delete media

Credentials you leave out of an update keep their current secrets. Updates and deletes check the connection's revision, so a change someone else saved first makes yours fail instead of overwriting it; read the connection again and retry.

Tenki rejects an endpoint or region change while running sandboxes mount the connection, because those mounts would keep using the old endpoint. The error names how many mounts use it. Detach them first, then update the connection.

A connection cannot be deleted while a running sandbox or a default mount uses it. Deleting a connection also deletes the secrets Tenki created for it, unless another connection uses them. That includes secrets created under an earlier name of a renamed connection. Secrets you picked with Use secret or --<field>-secret stay in Tenki Secrets, even when their names look like generated ones.

Rotate or revoke credentials

To rotate a key, rotate the secret or update the connection with the new value. Running mounts pick it up at their next credential refresh, within about 15 minutes.

Revoking a secret stops new mounts right away, and connection tests report SECRET_UNAVAILABLE. Running mounts lose access at their next credential refresh, even when programs in the sandbox have files open. They then show failed with credentials_revoked, and writes that were not uploaded yet do not reach the bucket.

Disabled by Tenki

Tenki support can disable a connection, for example after an abuse report, or detach a single mount. A disabled connection shows Disabled by Tenki in Settings > Storage and disabled by Tenki, contact support in tenki sandbox storage connections list. It cannot be edited, tested or deleted, and only Tenki support can enable it again. You also cannot create another connection to the same bucket in that workspace while it is disabled.

While it is disabled:

• Creating a sandbox or attaching a mount that names the connection fails with storage connection "<name>" was disabled by Tenki; contact support.
• Default mounts on the connection are skipped, so new sandboxes still start without them.
• Tenki can detach running mounts right away, and they then show detached_by_tenki. Otherwise they lose access at their next credential refresh and show failed with connection_disabled.

A mount Tenki detached shows detached_by_tenki; while Tenki finishes cutting it, it can show unmounting and its path stays taken. In both cases, writes that were not uploaded yet do not reach the bucket. Contact Tenki support with the workspace and connection name to find out why and to restore access.

Mount a bucket

Every mount has a connection, a mount path in the sandbox, an optional prefix, and an access mode. With a prefix, the folder shows only the objects under that prefix, and the mount cannot reach the rest of the bucket. A mount without an access mode uses the connection's default access.

Default mounts

Default mounts apply to every new sandbox in the workspace, so they suit buckets most sandboxes need, such as a shared dataset. A workspace can have up to 16 default mounts. Changing the defaults affects sandboxes created afterwards; running sandboxes keep their mounts.

In the dashboard, open Settings > Storage, click Edit defaults under Default mounts, click Add mount for each mount, and click Save defaults.

const defaults = await sandbox.getStorageDefaults();
const { connections } = await sandbox.listStorageConnections();
const datasets = connections.find((connection) => connection.name === "datasets")!;

await sandbox.saveStorageDefaults(
  {
    workspaceId: defaults.workspaceId,
    mounts: [
      ...defaults.mounts,
      { connectionId: datasets.id, prefix: "imagenet", mountPath: "/mnt/imagenet", access: "READ_ONLY" },
    ],
  },
  defaults.revision,
);

Saving replaces the whole list, so include the defaults you want to keep, with their IDs. The CLI keeps the ID of a default whose path did not change. tenki sandbox storage defaults get shows each default's ID, which you need to change or skip a default when you create a sandbox.

Mount at create

Mounts you pass when you create a sandbox are added to the workspace defaults. You can also change a default's path, prefix, or access for this sandbox, leave one default out, or skip all of them.

const session = await sandbox.createAndWait({
  storageMounts: [
    { connectionName: "media", prefix: "raw", mountPath: "/mnt/raw", access: "READ_ONLY" },
    { defaultMountId: imagenetDefaultId, mountPath: "/workspace/imagenet" },
  ],
  excludedDefaultMountIds: [otherDefaultId],
});

When you change a default, the fields you leave out keep the default's values. In the SDKs, an empty prefix mounts the whole bucket, and an empty access uses the connection's default access (StorageAccessDefault in Go).

In --storage-default, a bare / mounts the whole bucket, and default uses the connection's default access, as in --storage-default <id>/:default. --storage-mount accepts a connection name or ID.

In the dashboard, the Storage mounts section of the launch dialog for a published template lists the workspace defaults. Edit or remove a row, or add one, before you launch.

If a mount fails while the sandbox starts, the sandbox still starts. That mount shows failed with the reason in last_error, such as bucket_unreachable or credentials_unavailable. Check with tenki sandbox storage mounts list <session-id>, fix the cause, and attach it again at the same path.

A forked sandbox gets the source's mounts instead of the current defaults. Snapshots do not contain bucket data. A sandbox restored from a snapshot is a new sandbox, so it gets the workspace defaults and any mounts you pass. While Tenki takes a snapshot of a running sandbox, it unmounts the sandbox's storage mounts and mounts them again afterwards. The snapshot fails if a program has files open on a mount or an upload cannot finish, so close files on the mounts first. Pausing a sandbox also unmounts its storage mounts, and resuming it mounts them again.

Attach and detach

Attach and detach mounts on a running sandbox.

const mount = await session.attachStorageMount({ connectionName: "media", prefix: "raw", mountPath: "/mnt/media" });

const mounts = await session.listStorageMounts();
await session.detachStorageMount(mount.id);

In the dashboard, open the sandbox, click Attach under Storage mounts, fill in the connection, mount path, bucket prefix, and access, and click Attach storage. To detach, click the detach button next to the mount.

Attach returns once the folder is mounted or the mount has failed. A failed attach returns an error with the reason, and the mount stays listed as failed. Attaching again at the same path replaces the failed mount. If someone detaches the mount before the attach finishes, attach raises StorageMountAttachCancelledError (Go: ErrStorageMountAttachCancelled) and the mount ends detached. If the mount fails before the attach finishes, attach returns that failure like any failed attach.

While a detach runs, the mount shows unmounting, which the dashboard labels Detaching. Detach fails with guest_unmount_failed while a program in the sandbox has files open on the mount. It then waits up to a minute for pending uploads. If an upload cannot finish, detach fails with writeback_failed, and the folder stays mounted so no data is lost. After either failure the mount shows failed, but the folder is still mounted. Fix the cause and detach again. If Tenki cannot reach the sandbox's host, detach returns an error and the mount stays unmounting; detach again to retry.

Terminating a sandbox also waits for pending uploads, up to a minute for each mount. Writes that still cannot be uploaded stay on the host for 7 days, then Tenki discards them. Contact Tenki support with the session ID and the mount ID within that time if you need them. Detach first when you need to know that every write reached the bucket.

Limits

A storage mount is not a local disk, and some file operations behave differently:

• Renaming a file copies the object and deletes the old one, so renaming large files or folders is slow.
• There are no file locks, hard links, or ownership changes.
• Changes that other clients make to the bucket can take up to 5 minutes to appear, because the mount caches directory listings.
• When two writers change the same object, the last upload wins.
• Writes go to a cache on the host and upload about a second after the file is closed. Each mount's cache holds 2 GiB, so writing more than that before closing files fails with "No space left on device".
• Each mount's host process gets at most one CPU core and 512 MiB of memory.

Tenki checks these rules when you save a connection or a mount:

• A workspace can have up to 100 storage connections and up to 16 default mounts.
• A sandbox can have up to 16 storage mounts.
• Mount paths are absolute, up to 255 characters, and cannot be / or sit under /bin, /boot, /dev, /etc, /lib, /lib64, /proc, /root, /run, /sbin, /sys, /usr, or /var. Two mounts in one sandbox cannot share a path or nest inside each other, and a storage mount cannot overlap a persistent volume's mount path.
• Prefixes are up to 1024 characters, and leading and trailing slashes are ignored. Tenki rejects *, ?, and $, because AWS policies read them as wildcards and variables. It also rejects backslashes and empty, ., or .. path segments.
• Connection names start with a letter and use letters, digits, spaces, _, ., and -, up to 128 characters.
• Bucket names are 3 to 63 lowercase letters, digits, dots, and hyphens, and start and end with a letter or digit.
• Endpoints are public https:// URLs with no path, as described under S3-compatible.
• Endpoint and region changes are rejected while running sandboxes mount the connection, so detach those mounts first.
• A mount that fails while a sandbox starts leaves the sandbox running, with that mount failed.

Troubleshooting

A mount shows failed. Run tenki sandbox storage mounts list <session-id> and read the error column, or last_error with --json:

For any other reason, contact Tenki support with the session ID, the mount ID, and the reason.

A storage command fails with storage mounts are not enabled for this workspace. Storage mounts are in preview. Contact Tenki support to turn them on for the workspace.

Creating a keyless connection fails with keyless AWS role connections are not available here; use access keys instead. This Tenki deployment has no OIDC issuer for AWS to trust. Use AWS access keys instead.

Creating, updating, or deleting a connection fails with API key requires Secrets access. Create an API key with Allow Secrets access, or log in to the CLI again and select it. Your account also needs workspace-edit permission.

Saving a connection fails with secret ... already exists; select it instead of entering a value. A secret with the name Tenki would create already exists, and Tenki did not create it for this connection. Pick that secret with Use secret or --<field>-secret, or enter the value under a different connection name.

Create or attach fails with a storage connection not found error. A mount names a connection that does not exist in the workspace. Connection names match without regard to case. The SDKs raise StorageConnectionNotFoundError, or return ErrStorageConnectionNotFound in Go; see the SDK reference.

Create or attach fails with storage connection "<name>" was disabled by Tenki; contact support. Tenki disabled the connection; see Disabled by Tenki. The SDKs raise StorageConnectionDisabledError, or return ErrStorageConnectionDisabled in Go.

Saving fails with storage prefix must not contain *, ? or $. Remove the wildcard and mount the nearest parent prefix instead.

An update fails with endpoint and region cannot change while 2 mounts use this connection; detach them first. Detach the listed mounts, update the connection, and attach them again.

Writes fail with "No space left on device". The mount's 2 GiB upload cache is full of files that are still open or not uploaded yet. Close files sooner, or write large outputs in smaller parts.