Route cluster metadata to writable primary
This commit is contained in:
1 parent
a71c4fa5f1
commit
43987bbadb
9 files changed
+137
-12
No files matched your search
@@ -22,6 +22,7 @@ Source: [GitHub](https://github.com/LunarSkyOSS/ObjectStore) · [Gitea mirror](h
|
||||
- [Java client](#java-client)
|
||||
- [Local cluster prototype](#local-cluster-prototype)
|
||||
- [Node transport TLS](#node-transport-tls)
|
||||
- [Metadata primary routing](#metadata-primary-routing)
|
||||
- [Migrating a local cluster](#migrating-a-local-cluster)
|
||||
- [Adding a cluster node](#adding-a-cluster-node)
|
||||
- [Cluster maintenance and recovery](#cluster-maintenance-and-recovery)
|
||||
@@ -119,7 +120,7 @@ The [JDK-only Java client](client/README.md) works with ObjectStore and other S3
|
||||
|
||||
## Local cluster prototype
|
||||
|
||||
The local cluster prototype starts three segment containers and one PostgreSQL container on the same Docker host. Copy `.env.cluster.example` to a private environment file, replace all four credentials, and run:
|
||||
The local cluster prototype starts three segment containers and one PostgreSQL container on the same Docker host. Copy `.env.cluster.example` to a private environment file, replace all five credential values, and run:
|
||||
|
||||
```sh
|
||||
docker compose --env-file /path/to/cluster.env -f compose.cluster.yaml up -d --build
|
||||
@@ -129,6 +130,10 @@ docker compose --env-file /path/to/cluster.env -f compose.cluster.yaml run --rm
|
||||
|
||||
The cluster S3 endpoint binds to `127.0.0.1:9001`; storage nodes and PostgreSQL have no published ports. The separate repair container holds the repair credential and restores missing or corrupt replicas.
|
||||
|
||||
Multipart parts are stored on cluster nodes and indexed in PostgreSQL. Incomplete uploads count toward the logical capacity limit; abort them to release that capacity. Repair includes staged parts. The gateway upgrades the metadata schema when it starts, so back up the database before upgrading an existing cluster.
|
||||
|
||||
Node UUIDs persist on their volumes, and replica manifests use those UUIDs so reordering configured URLs cannot move an existing replica. Each node also has an operator-assigned physical host UUID. New writes require acknowledgements from two different host UUIDs. The optional `CLUSTER_TEST_NODE_DOMAINS=true` override counts containers instead, solely for local process tests; all containers in this Compose file share one physical host.
|
||||
|
||||
## Node transport TLS
|
||||
|
||||
The default local Compose cluster uses HTTP inside its private Docker network. For an HTTPS test, give each node a PKCS#12 keystore containing its private key and a certificate whose DNS subject alternative name matches its `CLUSTER_NODES` hostname. Give the gateway, repair, garbage collection, and maintenance processes a PKCS#12 truststore containing the issuing CA or each node certificate. Mount the files read-only and keep the keystores and password files outside Git.
|
||||
@@ -146,9 +151,15 @@ The files must be readable by container UID 10001 without making private keys or
|
||||
|
||||
This secures node traffic only. The local cluster still lacks automatic PostgreSQL failover, database TLS configuration, encryption at rest, and production multi-server validation. Its HTTP S3 gateway remains bound to localhost; use a separate trusted proxy for external TLS. Do not treat the TLS overlay as a production deployment.
|
||||
|
||||
Multipart parts are stored on cluster nodes and indexed in PostgreSQL. Incomplete uploads count toward the logical capacity limit; abort them to release that capacity. Repair includes staged parts. The gateway upgrades the metadata schema when it starts, so back up the database before upgrading an existing cluster.
|
||||
## Metadata primary routing
|
||||
|
||||
Node UUIDs persist on their volumes, and replica manifests use those UUIDs so reordering configured URLs cannot move an existing replica. Each node also has an operator-assigned physical host UUID. New writes require acknowledgements from two different host UUIDs. The optional `CLUSTER_TEST_NODE_DOMAINS=true` override counts containers instead, solely for local process tests; all containers in this Compose file share one physical host.
|
||||
`POSTGRES_JDBC_URL` can override the database URL for the gateway, repair, garbage collection, and maintenance processes. Its local Compose default now uses `targetServerType=primary`, and `/ready` returns unavailable when the database is read-only or in recovery. The base Compose file still starts and waits for its own single PostgreSQL container; it is not an HA deployment. In an independently managed deployment, list the PostgreSQL hosts in the JDBC URL and keep `targetServerType=primary`:
|
||||
|
||||
```text
|
||||
jdbc:postgresql://db-a:5432,db-b:5432/objectstore?targetServerType=primary&connectTimeout=3&socketTimeout=10
|
||||
```
|
||||
|
||||
ObjectStore opens a new database connection for each operation, so the JDBC driver can select a promoted primary after the old one is stopped. This does **not** promote a standby, fence the old primary, configure synchronous replication, or guarantee that a recently acknowledged write reached the standby. Those jobs belong to a separately operated PostgreSQL HA system. Never allow two writable metadata databases: they can diverge while serving different ObjectStore requests. A request interrupted during failover has an uncertain outcome; verify it before retrying a non-idempotent operation. For remote database connections, configure PostgreSQL TLS and use JDBC `sslmode=verify-full` with a mounted CA certificate. The provided local Compose database does not enable TLS.
|
||||
|
||||
## Migrating a local cluster
|
||||
|
||||
|
||||
Reference in new issue
Block a user