Route cluster metadata to writable primary

README.md

@@ -22,6 +22,7 @@ Source: [GitHub](https://github.com/LunarSkyOSS/ObjectStore) · [Gitea mirror](h
 - [Java client](#java-client)
 - [Local cluster prototype](#local-cluster-prototype)
 - [Node transport TLS](#node-transport-tls)
+- [Metadata primary routing](#metadata-primary-routing)
 - [Migrating a local cluster](#migrating-a-local-cluster)
 - [Adding a cluster node](#adding-a-cluster-node)
 - [Cluster maintenance and recovery](#cluster-maintenance-and-recovery)
@@ -119,7 +120,7 @@ The [JDK-only Java client](client/README.md) works with ObjectStore and other S3
 
 ## Local cluster prototype
 
-The local cluster prototype starts three segment containers and one PostgreSQL container on the same Docker host. Copy `.env.cluster.example` to a private environment file, replace all four credentials, and run:
+The local cluster prototype starts three segment containers and one PostgreSQL container on the same Docker host. Copy `.env.cluster.example` to a private environment file, replace all five credential values, and run:
 
 ```sh
 docker compose --env-file /path/to/cluster.env -f compose.cluster.yaml up -d --build
@@ -129,6 +130,10 @@ docker compose --env-file /path/to/cluster.env -f compose.cluster.yaml run --rm
 
 The cluster S3 endpoint binds to `127.0.0.1:9001`; storage nodes and PostgreSQL have no published ports. The separate repair container holds the repair credential and restores missing or corrupt replicas.
 
+Multipart parts are stored on cluster nodes and indexed in PostgreSQL. Incomplete uploads count toward the logical capacity limit; abort them to release that capacity. Repair includes staged parts. The gateway upgrades the metadata schema when it starts, so back up the database before upgrading an existing cluster.
+
+Node UUIDs persist on their volumes, and replica manifests use those UUIDs so reordering configured URLs cannot move an existing replica. Each node also has an operator-assigned physical host UUID. New writes require acknowledgements from two different host UUIDs. The optional `CLUSTER_TEST_NODE_DOMAINS=true` override counts containers instead, solely for local process tests; all containers in this Compose file share one physical host.
+
 ## Node transport TLS
 
 The default local Compose cluster uses HTTP inside its private Docker network. For an HTTPS test, give each node a PKCS#12 keystore containing its private key and a certificate whose DNS subject alternative name matches its `CLUSTER_NODES` hostname. Give the gateway, repair, garbage collection, and maintenance processes a PKCS#12 truststore containing the issuing CA or each node certificate. Mount the files read-only and keep the keystores and password files outside Git.
@@ -146,9 +151,15 @@ The files must be readable by container UID 10001 without making private keys or
 
 This secures node traffic only. The local cluster still lacks automatic PostgreSQL failover, database TLS configuration, encryption at rest, and production multi-server validation. Its HTTP S3 gateway remains bound to localhost; use a separate trusted proxy for external TLS. Do not treat the TLS overlay as a production deployment.
 
-Multipart parts are stored on cluster nodes and indexed in PostgreSQL. Incomplete uploads count toward the logical capacity limit; abort them to release that capacity. Repair includes staged parts. The gateway upgrades the metadata schema when it starts, so back up the database before upgrading an existing cluster.
+## Metadata primary routing
 
-Node UUIDs persist on their volumes, and replica manifests use those UUIDs so reordering configured URLs cannot move an existing replica. Each node also has an operator-assigned physical host UUID. New writes require acknowledgements from two different host UUIDs. The optional `CLUSTER_TEST_NODE_DOMAINS=true` override counts containers instead, solely for local process tests; all containers in this Compose file share one physical host.
+`POSTGRES_JDBC_URL` can override the database URL for the gateway, repair, garbage collection, and maintenance processes. Its local Compose default now uses `targetServerType=primary`, and `/ready` returns unavailable when the database is read-only or in recovery. The base Compose file still starts and waits for its own single PostgreSQL container; it is not an HA deployment. In an independently managed deployment, list the PostgreSQL hosts in the JDBC URL and keep `targetServerType=primary`:
+
+```text
+jdbc:postgresql://db-a:5432,db-b:5432/objectstore?targetServerType=primary&connectTimeout=3&socketTimeout=10
+```
+
+ObjectStore opens a new database connection for each operation, so the JDBC driver can select a promoted primary after the old one is stopped. This does **not** promote a standby, fence the old primary, configure synchronous replication, or guarantee that a recently acknowledged write reached the standby. Those jobs belong to a separately operated PostgreSQL HA system. Never allow two writable metadata databases: they can diverge while serving different ObjectStore requests. A request interrupted during failover has an uncertain outcome; verify it before retrying a non-idempotent operation. For remote database connections, configure PostgreSQL TLS and use JDBC `sslmode=verify-full` with a mounted CA certificate. The provided local Compose database does not enable TLS.
 
 ## Migrating a local cluster
 

TESTS.md

@@ -27,9 +27,13 @@ The script compiles the source and test programs into `out/classes`, then runs:
 
 The script exits nonzero on failure. The test programs use temporary local directories and loopback HTTP or HTTPS ports; they do not use an existing ObjectStore volume. `ClusterTlsTest` uses the JDK's `keytool` to create disposable test certificates.
 
+## Disposable metadata routing test
+
+Run `sh scripts/test-metadata.sh` with Docker Compose. It creates a separate, temporary PostgreSQL container and two in-process storage nodes. The test connects through a two-host JDBC URL whose first host is unavailable, checks writable readiness, then makes the test database read-only and verifies that readiness drops. The script removes its test containers and temporary database afterward. It does not promote a standby or test automatic failover.
+
 ## Disposable Docker cluster tests
 
-Requires Docker with Compose, Python 3.9 or newer, `curl`, and a free local port 9001. Make a test-only environment file from `.env.cluster.example` and fill in all five blank credentials with test-only values. Keep that file private and out of Git.
+Requires Docker with Compose, Python 3.9 or newer, `curl`, and a free local port 9001. Make a test-only environment file from `.env.cluster.example` and replace all five credential values with test-only values. Keep that file private and out of Git.
 
 ```sh
 cp .env.cluster.example /tmp/objectstore-cluster-tests.env
@@ -50,7 +54,7 @@ COMPOSE_PROJECT_NAME=objectstore-tests docker compose --env-file /tmp/objectstor
 
 If port 9001 is occupied, set `CLUSTER_HOST_PORT` to the same free port in both the environment file and the shell before running the script. The script reads that port from the shell; Compose reads it from the file.
 
-The Docker suite checks signed capability discovery and S3 operations, bucket creation and deletion, metadata and tags, public-read ACLs and anonymous access, copies, upload checksums, multi-segment objects, concurrent overwrites, multipart staging and listings, versioned reads and delete markers, versioned multipart completion, completion after a gateway restart and node loss, reads and writes with a node stopped, refusal to write without a storage quorum, restart recovery, corrupt-replica repair including staged parts, metadata unavailability, and placement on a newly joined node. It then checks rebalance to the fourth node, automatic repair, garbage collection dry run and delayed deletion, and a metadata backup restored to a separate PostgreSQL instance while the primary is stopped. Historical regular and multipart versions are checked after repair and cleanup. It also checks that containers labeled as one physical host cannot satisfy the normal host quorum. Its local-only override permits the remaining phases to use containers as separate test domains.
+The Docker suite checks signed capability discovery and S3 operations, bucket creation and deletion, metadata and tags, public-read ACLs and anonymous access, copies, upload checksums, multi-segment objects, concurrent overwrites, multipart staging and listings, versioned reads and delete markers, versioned multipart completion, completion after a gateway restart and node loss, reads and writes with a node stopped, refusal to write without a storage quorum, restart recovery, corrupt-replica repair including staged parts, metadata unavailability, read-only metadata readiness rejection, and placement on a newly joined node. It then checks rebalance to the fourth node, automatic repair, garbage collection dry run and delayed deletion, and a metadata backup restored to a separate PostgreSQL instance while the original database is stopped. A two-host JDBC URL selects that restored writable database. Historical regular and multipart versions are checked after repair and cleanup. It also checks that containers labeled as one physical host cannot satisfy the normal host quorum. Its local-only override permits the remaining phases to use containers as separate test domains.
 
 `ClusterMigrationTest` is a separate legacy-format fixture and is **not** run by either test script. Do not run its `create` phase against a populated metadata database. The migration procedure is in the [README](README.md#migrating-a-local-cluster).
 

compose.cluster.yaml

@@ -23,7 +23,7 @@ services:
       PUBLIC_TRUSTED_PROXY_IPS: ${PUBLIC_TRUSTED_PROXY_IPS:-}
       CLUSTER_TOKEN: ${CLUSTER_TOKEN:?Set CLUSTER_TOKEN}
       CLUSTER_NODES: ${CLUSTER_NODES:-http://node-a:9100,http://node-b:9100,http://node-c:9100}
-      POSTGRES_JDBC_URL: jdbc:postgresql://metadata:5432/objectstore?connectTimeout=3&socketTimeout=10
+      POSTGRES_JDBC_URL: ${POSTGRES_JDBC_URL:-jdbc:postgresql://metadata:5432/objectstore?connectTimeout=3&socketTimeout=10&targetServerType=primary}
       POSTGRES_USER: objectstore
       POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD}
     ports:
@@ -96,7 +96,7 @@ services:
       CLUSTER_TOKEN: ${CLUSTER_TOKEN:?Set CLUSTER_TOKEN}
       CLUSTER_REPAIR_TOKEN: ${CLUSTER_REPAIR_TOKEN:?Set CLUSTER_REPAIR_TOKEN}
       S3_BUCKET: ${S3_BUCKET:-objects}
-      POSTGRES_JDBC_URL: jdbc:postgresql://metadata:5432/objectstore?connectTimeout=3&socketTimeout=10
+      POSTGRES_JDBC_URL: ${POSTGRES_JDBC_URL:-jdbc:postgresql://metadata:5432/objectstore?connectTimeout=3&socketTimeout=10&targetServerType=primary}
       POSTGRES_USER: objectstore
       POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD}
       CLUSTER_GC_MIN_AGE_SECONDS: ${CLUSTER_GC_MIN_AGE_SECONDS:-1209600}
@@ -131,7 +131,7 @@ services:
       CLUSTER_BACKUP_RETENTION_SECONDS: ${CLUSTER_BACKUP_RETENTION_SECONDS:-0}
       CLUSTER_GC_TEST_MODE: ${CLUSTER_GC_TEST_MODE:-false}
       S3_BUCKET: ${S3_BUCKET:-objects}
-      POSTGRES_JDBC_URL: jdbc:postgresql://metadata:5432/objectstore?connectTimeout=3&socketTimeout=10
+      POSTGRES_JDBC_URL: ${POSTGRES_JDBC_URL:-jdbc:postgresql://metadata:5432/objectstore?connectTimeout=3&socketTimeout=10&targetServerType=primary}
       POSTGRES_USER: objectstore
       POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD}
     depends_on:

scripts/test-cluster.sh

@@ -8,6 +8,8 @@ compose() { docker compose --env-file "$env_file" -f compose.cluster.yaml "$@";
 restore() {
   compose stop maintenance >/dev/null 2>&1 || true
   compose start metadata node-a node-b node-c >/dev/null 2>&1 || true
+  compose exec -T metadata psql -U objectstore -d postgres -c \
+    'ALTER DATABASE objectstore RESET default_transaction_read_only' >/dev/null 2>&1 || true
   if [ -n "$backup_dir" ]; then rm -rf "$backup_dir"; fi
 }
 trap restore EXIT
@@ -59,6 +61,13 @@ expected=$(compose exec -T metadata psql -U objectstore -d objectstore -At -c \
   "SELECT encode(s.sha256,'hex') FROM cluster_segments s JOIN cluster_objects o ON o.generation=s.generation WHERE o.object_key='cluster-test/survivor' LIMIT 1")
 actual=$(compose exec -T node-a sha256sum "/data/segments/$shard/$segment_id" | cut -d' ' -f1)
 [ "$expected" = "$actual" ]
+compose exec -T metadata psql -U objectstore -d postgres -c \
+  'ALTER DATABASE objectstore SET default_transaction_read_only=on' >/dev/null
+status=$(curl -sS -o /dev/null -w '%{http_code}' "http://127.0.0.1:$host_port/ready")
+[ "$status" = 503 ]
+compose exec -T metadata psql -U objectstore -d postgres -c \
+  'ALTER DATABASE objectstore RESET default_transaction_read_only' >/dev/null
+wait_ready
 compose stop metadata
 status=$(curl -sS -o /dev/null -w '%{http_code}' "http://127.0.0.1:$host_port/ready")
 [ "$status" = 503 ]
@@ -124,7 +133,7 @@ compose exec -T metadata-recovery pg_restore -U objectstore -d objectstore --no-
   < "$backup_dir/metadata.dump"
 compose stop metadata
 compose run --rm -T --no-deps \
-  -e 'POSTGRES_JDBC_URL=jdbc:postgresql://metadata-recovery:5432/objectstore?connectTimeout=3&socketTimeout=10' \
+  -e 'POSTGRES_JDBC_URL=jdbc:postgresql://metadata:5432,metadata-recovery:5432/objectstore?connectTimeout=3&socketTimeout=10&targetServerType=primary&hostRecheckSeconds=0' \
   --entrypoint java gateway --add-modules jdk.httpserver,java.net.http \
   -cp /app:/app/postgresql.jar:/app/hash4j.jar cloud.lunarsky.store.ClusterIntegrationTest recovered
 compose start metadata

scripts/test-metadata.sh

@@ -0,0 +1,7 @@
+#!/bin/sh
+set -eu
+cd "$(dirname "$0")/.."
+project="objectstore-metadata-test-$$"
+compose() { docker compose -p "$project" -f tests/metadata/compose.yaml "$@"; }
+trap 'compose down -v --remove-orphans >/dev/null 2>&1 || true' EXIT
+compose up --build --abort-on-container-exit --exit-code-from check check

src/cloud/lunarsky/store/ClusterStore.java

@@ -1347,8 +1347,9 @@ final class ClusterStore implements ObjectStorage, MultipartStorage {
     @Override public boolean ready() {
         if (!nodes.availableHostsAtLeast(2, testNodeDomains)) return false;
         try (Connection connection = connect(); var statement = connection.createStatement();
-             ResultSet result = statement.executeQuery("SELECT 1")) {
+             ResultSet result = statement.executeQuery(
-            return result.next() && result.getInt(1) == 1;
+                 "SELECT NOT pg_is_in_recovery() AND current_setting('transaction_read_only') = 'off'")) {
+            return result.next() && result.getBoolean(1);
         } catch (SQLException error) { return false; }
     }
     RepairReport repairOnce() throws IOException {

test/cloud/lunarsky/store/ClusterMetadataTest.java

@@ -0,0 +1,63 @@
+package cloud.lunarsky.store;
+
+import com.sun.net.httpserver.HttpServer;
+import java.net.InetSocketAddress;
+import java.net.URI;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.sql.DriverManager;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+public final class ClusterMetadataTest {
+    public static void main(String[] args) throws Exception {
+        Map<String, String> env = System.getenv();
+        String token = "metadata-test-cluster-token-0123456789";
+        String repairToken = "metadata-test-repair-token-0123456789";
+        Path root = Files.createTempDirectory("objectstore-metadata-test-");
+        try (ClusterNode first = new ClusterNode(root.resolve("first"), token, repairToken, UUID.randomUUID());
+             ClusterNode second = new ClusterNode(root.resolve("second"), token, repairToken, UUID.randomUUID())) {
+            HttpServer firstServer = server(first);
+            HttpServer secondServer = server(second);
+            try {
+                List<URI> nodes = List.of(url(firstServer), url(secondServer));
+                try (ClusterStore store = new ClusterStore(env.get("POSTGRES_JDBC_URL"),
+                        env.get("POSTGRES_USER"), env.get("POSTGRES_PASSWORD"), "objects",
+                        nodes, token, repairToken, 1024 * 1024, 16 * 1024 * 1024, false)) {
+                    require(store.ready(), "Writable primary was not selected from the multi-host URL");
+                    try (var admin = DriverManager.getConnection(env.get("POSTGRES_ADMIN_JDBC_URL"),
+                            env.get("POSTGRES_USER"), env.get("POSTGRES_PASSWORD"));
+                         var statement = admin.createStatement()) {
+                        statement.execute("ALTER DATABASE objectstore_test SET default_transaction_read_only=on");
+                        try {
+                            require(!store.ready(), "Read-only metadata was reported ready");
+                        } finally {
+                            statement.execute("ALTER DATABASE objectstore_test RESET default_transaction_read_only");
+                        }
+                    }
+                    require(store.ready(), "Writable metadata did not recover after read-only mode ended");
+                }
+            } finally {
+                firstServer.stop(0);
+                secondServer.stop(0);
+            }
+        }
+        System.out.println("Metadata routing tests passed: second JDBC host and writable readiness");
+    }
+
+    private static HttpServer server(ClusterNode node) throws Exception {
+        HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
+        server.createContext("/", node::handle);
+        server.start();
+        return server;
+    }
+
+    private static URI url(HttpServer server) {
+        return URI.create("http://127.0.0.1:" + server.getAddress().getPort());
+    }
+
+    private static void require(boolean condition, String message) {
+        if (!condition) throw new AssertionError(message);
+    }
+}

tests/metadata/compose.yaml

@@ -0,0 +1,30 @@
+services:
+  metadata:
+    image: postgres:17-alpine
+    environment:
+      POSTGRES_DB: objectstore_test
+      POSTGRES_USER: objectstore_test
+      POSTGRES_PASSWORD: local-metadata-test-only
+    tmpfs:
+      - /var/lib/postgresql/data:size=268435456
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U objectstore_test -d objectstore_test"]
+      interval: 2s
+      timeout: 2s
+      retries: 20
+    mem_limit: 256m
+
+  check:
+    build:
+      context: ../..
+    image: lunarsky-objectstore:metadata-test
+    entrypoint: ["java", "--add-modules", "jdk.httpserver,java.net.http", "-cp", "/app:/app/postgresql.jar:/app/hash4j.jar", "cloud.lunarsky.store.ClusterMetadataTest"]
+    environment:
+      POSTGRES_JDBC_URL: jdbc:postgresql://127.0.0.2:5432,metadata:5432/objectstore_test?connectTimeout=1&socketTimeout=10&targetServerType=primary&hostRecheckSeconds=0
+      POSTGRES_ADMIN_JDBC_URL: jdbc:postgresql://metadata:5432/postgres?connectTimeout=3&socketTimeout=10
+      POSTGRES_USER: objectstore_test
+      POSTGRES_PASSWORD: local-metadata-test-only
+    depends_on:
+      metadata:
+        condition: service_healthy
+    mem_limit: 384m

tests/two-host/README.md

@@ -1,6 +1,6 @@
 # Two-machine durability drill
 
-Run this disposable test on two machines. Machine A runs the gateway, PostgreSQL, and one storage node; machine B runs a second storage node. Keep the node connection on a private network: the node protocol uses bearer tokens over HTTP.
+Run this disposable test on two machines. Machine A runs the gateway, PostgreSQL, and one storage node; machine B runs a second storage node. Keep the node connection on a private network: this drill uses bearer tokens over HTTP and does not enable the optional node TLS setup.
 
 Both machines need Docker. Machine A also needs Docker Compose and Python 3. The example uses loopback port 9003 on machine A and private-network port 9103 on machine B; change them if needed. Use separate test volumes, and do not point this drill at an existing ObjectStore cluster.