Skip to content
ForgeForgeForge

Operations

Forge runs inside your process. It doesn’t start threads or daemons on its own, so the background work, the scheduler, and the maintenance sweep are loops you run. This page covers what to run and what to watch once Forge is behind real traffic.

Deploy migrations before the application

Section titled “Deploy migrations before the application”

Forge owns the objects listed in contract/schema-ownership.json. Point [postgres] at a database or schema reserved for Forge and do not write its objects directly.

Development and test environments migrate automatically by default. Production defaults to auto_migrate = false, so application startup validates the schema and fails before serving traffic when migrations are pending, locked, incompatible, or drifted. Run the migration API as a deployment job first:

const reports = await ForgeClient.migrate();
for (const report of reports) console.error(report.target, report.state, report.message);
if (reports.some((report) => report.state !== "applied")) process.exit(1);

migrationStatus/migration_status/MigrationStatus is read-only and also reports the safe PostgreSQL session identity when another migrator owns the advisory lock. validateSchema/validate_schema/ValidateSchema checks migration history without taking the lock. Every distinct measured database target receives one report. migration_lock_timeout_secs bounds lock acquisition and defaults to 30 seconds.

Forge starts from one v001_schema baseline. Unknown migration history and checksum drift are incompatible. Future schema changes append a migration; existing migration files remain immutable after release.

Workers

Section titled “Workers”

A queue holds jobs until a worker leases them. Run a worker per queue. Each binding has a managed worker that handles leasing, heartbeats, acks, retries, and graceful shutdown.

type SendEmail = { to: string };


await forge.worker<SendEmail>("emails", async (job) => {
  await send(job.payload);
}, { concurrency: 8, visibilitySeconds: 60, heartbeatSeconds: 15,
     retryBackoffSeconds: 0.25, drainDeadlineSeconds: 30,
     identity: "email-worker", signal: shutdown.signal, onError: report });

The shared worker state machine is polling → handling/heartbeating → settling → draining → stopped. Cancellation stops polling first. If cancellation wins the same boundary as dequeue, the fresh lease is released without starting user code. Active handlers drain until the configured deadline, then receive the language-native cancellation mechanism and their leases are released. Lease loss cancels the stale handler and prevents it from acknowledging or extending the newer owner’s lease. Backend retry waits use bounded 80–120% jitter. Worker identity appears only in diagnostics and never as a default metric label.

Run as many worker processes as you need; jobs are leased, so two workers never get the same job at once. A handler that crashes lets the lease expire and the job comes back, which is why handlers need to be safe to run twice. Jobs that exhaust their attempts move to "<queue>.dlq". Queue depth reports visible, in-flight, delayed, and oldest-visible age. Default metrics cover dead-letter transitions, redrives, lease loss, handler outcomes, queue depth/age, and outbox backlog without job IDs, payloads, worker identities, or queue names as dimensions.

Operator controls are typed library calls, not a second admin server. Batch enqueue returns per-item results; batch dequeue leases at most ten jobs with independent receipts. Pause/resume retains queued work and affects only new claims. Inspection, status filtering, counter-based throughput estimates, dead-letter redrive, and confirmed purge can be exposed through the application’s existing admin authorization boundary.

Transactional outbox

Section titled “Transactional outbox”

Use the versioned sql/forge_outbox_v1.sql contract when a domain change and queue enqueue must be atomic. Copy that application-owned table into the application’s migration, then insert its row with the application’s own PostgreSQL driver in the same transaction as the domain write. Forge does not migrate or own this table; grant the Forge runtime role SELECT and UPDATE on it.

BEGIN;
UPDATE orders SET state = 'paid' WHERE id = $1;
INSERT INTO app_forge_outbox_v1 (event_id, namespace, destination, payload)
VALUES ($2, 'billing', 'receipts', convert_to($3, 'UTF8'));
COMMIT;

Run one relay per replica with runOutboxRelay / run_outbox_relay / run_outbox_relay_until / RunOutboxRelay, passing the host language’s cancellation input. Claims use a bounded lease and FOR UPDATE SKIP LOCKED. The event UUID becomes the queue job UUID. A crash before enqueue leaves a reclaimable outbox row; a crash after enqueue but before marking the row dispatched repeats the deterministic enqueue and returns the same job ID; a crash after marking is complete. This is at-least-once relay processing with one effective queue row, not exactly-once handler execution. Invalid rows are retried with a bounded safe summary; payloads and trace baggage are never logged.

The scheduler

Section titled “The scheduler”

Cron and one-off schedules only fire when something ticks the scheduler. The example apps run one loop that ticks the scheduler and then runs the maintenance sweep, on every replica. Each due job is claimed once no matter how many replicas tick at the same time.

setInterval(async () => {
  try {
    await forge.runSchedulerOnce();
    await forge.maintain();
  } catch (e) {
    console.warn("scheduler tick failed:", e);
  }
}, 30_000);

Use schedulerDiagnostics / scheduler_diagnostics / Schedule::diagnostics / SchedulerDiagnostics for alerts. lag is the age of the oldest unpaused due row, dueCount is the current backlog, lastSuccessfulTick advances only after a complete pass, and enqueueFailures is cumulative. Alert on sustained lag or a non-advancing successful tick rather than on one transient failure. Pause and resume operate on one named schedule and preserve its payload and policy for inspection.

maintain() is the housekeeping sweep: it purges expired key/value rows and old finished jobs, reclaims leases left behind by crashed workers, drops stale rate-limit and dedup rows, and expires dead sessions and one-time tokens. It’s idempotent, so running it on every replica is fine. Rust also has run_scheduler(), a managed loop that ticks every 30 seconds on its own if you’d rather run the sweep separately.

Durability

Section titled “Durability”

The Postgres backend is durable and shared. The memory backend is neither, which is the point for tests but a trap in production. Pub/sub and rate limiting in particular depend on every replica seeing the same state, so on memory they only work within one process. Forge logs a warning when either resolves to memory, since that’s almost always a misconfiguration outside of tests. Keep production on postgres unless you have injected your own backend.

Health checks

Section titled “Health checks”

Liveness and readiness are separate. isLive() / is_live() / is_live / IsLive() only reports whether the local handle is open. probe() / Probe() runs one real operation against every enabled backend under one total deadline and returns redacted status, latency, error category, and last-success time per backend. By default every backend decides readiness; pass an explicit readiness set when a dependency is intentionally optional.

diagnostics() / Diagnostics() adds configuration, PostgreSQL version, migration state, runtime permissions, backend reachability, application/database clock skew, and unsafe-production-setting checks. It returns stable pass/warn/fail records with redacted messages. Applications expose or record this report through their existing deployment job, test suite, observability system, or authenticated admin surface; Forge does not create an unauthenticated diagnostics endpoint.

backendCapabilities() lists which provider backs each primitive, whether it is durable, and any caveats. It is synchronous configuration inspection, not a health check.

const report = forge.backendCapabilities();
// [{ primitive, provider, durable, caveats }]


if (!forge.isLive()) return new Response("stopping", { status: 503 });
const health = await forge.probe(2);

Metrics and tracing

Section titled “Metrics and tracing”

Metrics belong to each Forge instance. metricsSnapshot() / metrics_snapshot() / metrics_snapshot / MetricsSnapshot() returns bounded samples, while renderPrometheus() / render_prometheus() / render_prometheus / RenderPrometheus() emits Prometheus text without installing a collector or starting an exporter. Operations, pool use, retries, queue and worker state, scheduler and maintenance work, config sources, blob transfers, authentication, rate-limit decisions, and outbox backlog use fixed low-cardinality labels. Raw keys, user IDs, paths, queue names, job IDs, topics, model names, error messages, bodies, prompts, model output, content, tokens, and signed URLs are never labels or span attributes.

Queue and outbox propagation stores validated W3C traceparent and tracestate beside the payload. Baggage is empty by default and only explicit allow-listed keys cross the boundary. Forge creates producer, receive/process, settle, scheduler, relay, and blob spans with fixed semantic names when tracing is enabled. Applications remain responsible for model-call spans and may attach GenAI attributes through their OpenTelemetry instrumentation; Forge never inspects model payloads.

Errors and retries

Section titled “Errors and retries”

Forge failures carry the same code, retryable flag, safe message, operation, and optional backend identifier in Rust, JavaScript, Python, and Go. Only Unavailable and a Backend error explicitly marked retryable may be retried; worker and reconnect loops use that same rule. Config, NotConfigured, NotFound, Invalid, Limit, and Precondition require a changed request or refreshed state. The displayed message is safe and excludes raw database/provider errors, while the local cause chain remains available for debugging. The full table is in the Reference.

Scaling a primitive out

Section titled “Scaling a primitive out”

When one primitive outgrows the shared database, give it its own. Point it at a separate Postgres in forge.toml with a [databases.<feature>] table and it runs on an isolated pool, on its own server if you want, with no change to your code. See Configuration.

Known limitations

Section titled “Known limitations”

These limits matter in production.

Queue receipts are process-local in Node and Python. A receipt is only meaningful inside the process that dequeued the job: the binding holds the lease in an in-process map, so queueAck, queueNack, and queueHeartbeat must run in that same process. You can’t hand a receipt to another worker or persist it and settle the job later. A job left unsettled is reclaimed the usual way, by lease expiry on the server, and redelivered. Rust and Go pass the backend lease receipt directly and do not have this binding-map restriction.

Rate limiting on memory counts per process. Each process keeps its own buckets, so across replicas the limit is effectively multiplied by the replica count and every check under-counts. Use postgres for any limit that has to hold across processes.

Pub/sub is at-most-once and non-durable on every backend. A message reaches only the subscribers connected at publish time; there’s no persistence, replay, or delivery guarantee. On memory it’s in-process only, so nothing crosses replicas. Treat it as a notification, not a log, and reconcile durable state through the database.

Filesystem blob splits bytes from metadata. Objects live as files under the configured root while their metadata rows live in Postgres, so a put is two writes and is not atomic with your application’s SQL. A crash between them leaves a metadata row pointing at a missing file until the maintenance sweep reclaims it. Multiple replicas need a shared mount, since one replica can’t read files another wrote to local disk.

S3 policy remains provider-owned. Forge sends object bytes, web metadata, conditional request headers, provider-managed encryption headers, and multipart commands. Configure lifecycle expiration, retention, replication, versioning, object lock and legal hold, CDN caching and invalidation, CORS, public-access blocks, bucket policy, IAM, and KMS grants in the provider. Roll out those policies independently, verify them with provider tooling, and do not infer them from Forge head results.

Node kvIncr returns a JavaScript number. The counter is an i64 in the core, but JS numbers are f64, so a value past 2^53 loses precision on the way back. Real counters never reach that range; if yours might, read it back losslessly with kvGetBytes. The Python binding returns the exact integer.