Global Deploy

Configure multi-region nodes with read replicas to reduce read latency.

The Code

[database]
url = "${DATABASE_PRIMARY_URL}"
replica_urls = [
    "${DATABASE_REPLICA_US_EAST}",
    "${DATABASE_REPLICA_EU_WEST}",
    "${DATABASE_REPLICA_APAC}"
]
read_from_replica = true
pool_size = 50

What Happens

Forge maintains separate connection pools for the primary database and each replica. Queries route to healthy replicas via round-robin. Mutations always go to the primary. A background health monitor pings each replica every 15 seconds and removes unhealthy replicas from rotation. If all replicas are down, reads fall back to the primary automatically.

Queries that need read-after-write consistency can opt out of replica routing with the consistent attribute (see below).

All cluster coordination happens through PostgreSQL. Nodes in different regions find each other via the shared forge_nodes table without a service mesh or separate discovery service.

Configuration

Setting	Type	Default	Description
url	string	required	Primary database URL
replica_urls	string[]	[]	Read replica URLs
read_from_replica	bool	false	Route queries to replicas
pool_size	u32	50	Primary pool size (replicas get half)
pool_timeout_secs	u64	30	Connection acquire timeout
pools.default	config	none	Override primary pool size/timeout
pools.jobs	config	none	Isolated pool for background work
pools.observability	config	none	Isolated pool for metrics collection
pools.analytics	config	none	Isolated pool for long-running reports

Multi-Region Pattern

1. Set Up Geo-Replicated PostgreSQL

Use a managed database with automatic geo-replication:

PlanetScale

Primary: us-east-1
Replicas: eu-west-1, ap-northeast-1

Neon

Primary: aws-us-east-1
Read Replicas: aws-eu-west-1, aws-ap-southeast-1

Supabase

Primary: us-east-1
Read Replicas: Configure via Supabase dashboard

Any PostgreSQL-compatible service with read replicas works. Forge connects via standard Postgres wire protocol.

2. Deploy Nodes Per Region

Deploy Forge to each region, pointing to the local replica:

US East (Virginia)

[database]
url = "${PRIMARY_URL}"
replica_urls = ["${REPLICA_US_EAST}"]
read_from_replica = true

[cluster]
discovery = "postgres"

EU West (Frankfurt)

[database]
url = "${PRIMARY_URL}"
replica_urls = ["${REPLICA_EU_WEST}"]
read_from_replica = true

[cluster]
discovery = "postgres"

APAC (Tokyo)

[database]
url = "${PRIMARY_URL}"
replica_urls = ["${REPLICA_APAC}"]
read_from_replica = true

[cluster]
discovery = "postgres"

3. Route Traffic to Nearest Region

Use your CDN or load balancer to route users to the closest region:

User in London  → EU West node  → EU replica (low latency read)
User in Tokyo   → APAC node     → APAC replica (low latency read)
User in NYC     → US East node  → US East replica (low latency read)

All writes → Primary (single leader)

Patterns

Region-Local Reads with Global Writes

The default behavior. Queries read from the local replica, mutations write to the primary.

[database]
url = "postgres://primary.db.example.com/app"
replica_urls = ["postgres://replica-local.db.example.com/app"]
read_from_replica = true

Replication lag is typically under 100ms for managed databases. For most applications, eventual consistency on reads is acceptable.

Multiple Replicas Per Region

For high-read workloads, configure multiple replicas per node:

[database]
url = "${PRIMARY_URL}"
replica_urls = [
    "postgres://replica-1.local.example.com/app",
    "postgres://replica-2.local.example.com/app",
    "postgres://replica-3.local.example.com/app"
]
read_from_replica = true
pool_size = 100

The atomic counter distributes reads evenly across all three replicas.

Consistent Reads

Most queries tolerate replication lag (typically under 100ms). For queries that must see the latest data right after a mutation, use the consistent attribute to force reads from the primary:

#[forge::query(consistent)]
pub async fn get_order_receipt(ctx: &QueryContext, order_id: Uuid) -> Result<Order> {
    sqlx::query_as("SELECT * FROM orders WHERE id = $1")
        .bind(order_id)
        .fetch_one(ctx.db())
        .await
        .map_err(Into::into)
}

Without consistent, this query would read from a replica which might not have the order yet if it was just created. With consistent, it reads from the primary.

Isolated Pools for Workloads

Separate connection pools prevent one workload from starving another:

[database]
url = "${PRIMARY_URL}"
replica_urls = ["${REPLICA_LOCAL}"]
read_from_replica = true

[database.pools.default]
size = 30
timeout_secs = 10

[database.pools.jobs]
size = 20
timeout_secs = 60
statement_timeout_secs = 300

[database.pools.analytics]
size = 10
timeout_secs = 120
statement_timeout_secs = 600

When configured, background jobs, cron runners, daemon processes, and workflow executors all use the jobs pool. The observability pool handles internal metrics collection. The analytics pool is available via db.analytics_pool() for user code. A runaway analytics query cannot exhaust connections needed for user requests.

Cross-Region Cluster Discovery

Nodes discover each other through the primary database:

[cluster]
discovery = "postgres"
heartbeat_interval_secs = 5
dead_threshold_secs = 15

Each node registers in forge_nodes on startup:

INSERT INTO forge_nodes (id, hostname, ip_address, http_port, roles, status, ...)
ON CONFLICT (id) DO UPDATE SET last_heartbeat = NOW()

Nodes query this table to find peers. Leader election uses PostgreSQL advisory locks, so an external coordination service is not required.

Under the Hood

Health-Aware Round-Robin

Read queries distribute across healthy replicas via atomic counter:

let start = replica_counter.fetch_add(1, Ordering::Relaxed) % replicas.len();
for offset in 0..replicas.len() {
    let idx = (start + offset) % replicas.len();
    if replicas[idx].healthy {
        return replicas[idx].pool;
    }
}
// All unhealthy, fall back to primary
primary

Lock-free selection. A background task pings each replica every 15 seconds with SELECT 1 and marks it healthy or unhealthy. Unhealthy replicas are skipped in rotation. When all replicas are down, reads fall back to the primary with no configuration changes needed.

Separate Connection Pools

Primary and replicas maintain independent pools:

let primary = PgPoolOptions::new()
    .max_connections(pool_size)
    .connect(primary_url).await?;

for replica_url in &config.replica_urls {
    let pool = PgPoolOptions::new()
        .max_connections(pool_size / 2)  // Replicas get half
        .connect(replica_url).await?;
    replicas.push(pool);
}

Primary pool handles all writes. Replica pools handle reads when read_from_replica = true. Pool exhaustion on replicas does not affect write capacity.

Graceful Degradation

Replica health is monitored proactively:

1. Background task pings each replica every 15 seconds
2. Failed ping marks the replica unhealthy (logged as warning)
3. read_pool() skips unhealthy replicas in round-robin
4. If all replicas are down, reads fall back to the primary
5. When a replica recovers, the next health check marks it healthy again (logged as info)

No configuration changes needed. Failover and recovery are both automatic. Reads continue with higher latency until replicas recover.

Connection Routing

The context determines which pool handles your query:

// Mutation context - always primary
ctx.db()  // → primary pool

// Query context (default)
ctx.db()  // → round-robin across healthy replicas

// Query context (consistent = true)
ctx.db()  // → primary pool (bypass replicas)

Forge does not manage replication, only connection routing. The primary receives all writes and replicates to read replicas via PostgreSQL streaming replication.

Cluster Coordination via PostgreSQL

All coordination flows through the database:

Concern	Mechanism
Node discovery	forge_nodes table
Leader election	pg_try_advisory_lock()
Job distribution	FOR UPDATE SKIP LOCKED
Real-time sync	LISTEN/NOTIFY
Cron scheduling	UNIQUE(cron_name, scheduled_time)

Nodes in Tokyo, Frankfurt, and Virginia coordinate through the same primary database. Cross-region latency affects coordination but not local reads.

Testing

Local Multi-Region Simulation

Run multiple nodes pointing to different replicas:

# Terminal 1 - Simulating US East
DATABASE_PRIMARY_URL=postgres://localhost/app \
DATABASE_REPLICA_US_EAST=postgres://localhost:5433/app \
forge run

# Terminal 2 - Simulating EU West
DATABASE_PRIMARY_URL=postgres://localhost/app \
DATABASE_REPLICA_EU_WEST=postgres://localhost:5434/app \
forge run

Verify Replica Routing

Check which pool handles queries:

#[forge::query]
pub async fn debug_replica(ctx: &QueryContext) -> Result<String> {
    let row = sqlx::query_scalar::<_, String>("SELECT inet_server_addr()::text")
        .fetch_one(ctx.db())
        .await?;
    Ok(row)
}

With read_from_replica = true, consecutive calls return different replica addresses.

Failure Simulation

Stop a replica and verify degradation:

# Stop replica
docker stop postgres-replica-1

# Queries continue on remaining replicas
curl http://localhost:3000/api/query

# Restart replica
docker start postgres-replica-1

# Replica rejoins rotation automatically