gipper documentation

Serverless queue jobs

Gipper can create a Runpod GPU queue endpoint, submit JSON jobs, retrieve results, and verify deletion. This is separate from the existing Pod/SSH runner. The client uses Python's standard library; a worker may use its host's SDK.

The initial adapter is Runpod. The interface separates endpoint lifecycle from application inputs so another queue host can implement the same operations. Vast and Shadeform serverless execution are not implemented. There is no cross-provider placement or automatic job failover yet.

One temporary job

A first development run can use a verified upstream CUDA image and install the worker SDK at startup. This requires no local Docker access. After checking live 4090 Serverless stock and price, generate the spec with the observed rate:

python examples/serverless/bootstrap.py --rate-usd-h 1.34 > /tmp/gipper-smoke.toml
gipper serverless plan /tmp/gipper-smoke.toml
gipper serverless run /tmp/gipper-smoke.toml examples/serverless/input.json

Startup includes package installation and is not a baked-image latency measurement. For a worker with dependencies preinstalled, build and push the worker image from examples/serverless, then edit its TOML with your image reference and the current Serverless catalog rate.

docker build --platform linux/amd64 -t YOUR_REGISTRY/gipper-cuda-smoke:0.6.0 examples/serverless
docker push YOUR_REGISTRY/gipper-cuda-smoke:0.6.0
gipper serverless plan examples/serverless/runpod.toml
gipper serverless run examples/serverless/runpod.toml examples/serverless/input.json

The example runs matrix multiplication on CUDA, refuses CPU fallback, and returns the GPU, worker identity, and answer. A COMPLETED job must have answer == expected. Use an immutable image digest for repeatable production deployments. The Dockerfile installs worker dependencies at build time; application images belong to the caller.

run deletes the temporary endpoint in finally, including failed jobs and normal interruptions. SIGKILL or a lost machine cannot run finally; recover using the same ledger with gipper serverless reap --dry-run then gipper serverless reap. The reaper only deletes temporary endpoints whose recorded session deadline has expired. The existing Pod reaper does not manage serverless endpoints. Schedule the serverless command separately if you need unattended recovery.

Reuse an endpoint across revisions

gipper serverless deploy examples/serverless/runpod.toml
# Use the returned endpoint ID below.
gipper serverless submit ENDPOINT examples/serverless/input.json --execution-seconds 120
gipper serverless wait ENDPOINT JOB --timeout 600
gipper serverless status ENDPOINT
gipper serverless ls
gipper serverless delete ENDPOINT

deploy deliberately keeps the endpoint. Minimum workers are always zero in this first version. Idle workers can remain allocated and billed during idle_seconds; FlashBoot can help subsequent starts but does not establish a latency guarantee. Increasing the idle timeout trades money for readiness between revisions. Deliberately kept endpoints are not expired by the temporary-session reaper.

A result reports the provider's delayTime and executionTime in milliseconds, plus client waiting time. Queue delay can include startup and scheduling. It does not isolate image pull or model initialization. Health rollups do not prove a worker released its GPU or stopped billing. Retain the worker identity and instrument the application if distinguishing warm reuse from a new worker matters.

Python interface

from gipper.serverless import Client, load

spec = load("examples/serverless/runpod.toml")
client = Client(spec.provider)
result = client.run(spec, {"size": 256})  # temporary endpoint, verified cleanup
assert result["status"] == "COMPLETED", result

For reuse, call create(spec), submit(endpoint, input, execution_seconds), wait(endpoint, job_id, timeout), and delete(endpoint). Keep deletion in your own finally. wait cancels on timeout or interruption and verifies a terminal job state. A FAILED job returns its provider result so the application can interpret it; the CLI exits nonzero. A CleanupPending exception takes priority over continuing with another rental. Inspect the resource before making a replacement.

Costs and custody

rate_usd_h is caller supplied from the current catalog. plan computes an occupancy allowance using maximum workers times session duration plus the idle tail. It refuses a configuration exceeding max_usd. This is an admission check, not a provider-enforced cap. API delays, teardown delays, storage, and provider billing rules can add cost. session_seconds bounds waiting in run; it does not expire an endpoint created with deploy, or enforce a timeout on separately submitted jobs. Execution timeout is explicitly sent to both the endpoint and each job. Serverless receipts are separate from Pod cost/history and reconciliation; this version does not claim to reconcile serverless bills or enforce the Pod daily cap.

Create, verified settings, submission intent, job IDs, terminal timing, cancellation, and verified deletion are append-only serverless.* receipts. Input custody is recorded as a SHA-256 and byte count; raw inputs and outputs are not written to the ledger. Applications must retain their actual evidence separately. These receipts are not included in gipper custody report, which currently covers Pod uploads. Keys use Gipper's existing credential resolver and travel only in request headers. Non-idempotent creates and submissions are never retried automatically. If a response is lost, SubmissionUnknown and the receipt identify the ambiguity. Inspect the exact resource name before retrying creation; an unrecorded creation cannot be deleted through the owned-endpoint CLI and needs operator recovery.

Storage and GPU placement

runpod.pools contains catalog pool IDs, such as ADA_24, rather than GPU names. A pool can include multiple cards. Use excluded_types for explicit exclusions, and check every remaining card fits your image and memory requirements. min_cuda_version accepts newer compatible drivers. data_centers can narrow the allowed placement. Actual availability is checked by the operator before deployment.

An existing global_volume can be attached without binding model files to a data center. Alternatively, network_volumes attaches regional storage. They are mutually exclusive. Gipper neither creates nor deletes these user-owned volumes. Keep Python dependencies in the image and temporary decoding work on local disk. Global volume writes and updates have object-storage semantics; roll workers to pick up new versioned model assets. Global-volume latency is not qualified by the CUDA smoke test.

Runpod references are the Serverless configuration and global storage documentation.