Release ObjectStore 0.0.8

This commit is contained in:
admin committed 2026-10-10 15:22:21 +02:00
1 parent ed7712a8af
commit d6427fbac9
61 files changed
+7082 -267

No files matched your search

+1
View File
@@ -0,0 +1 @@
dist/
+58
View File
@@ -0,0 +1,58 @@
# ObjectStore Java client
A small, synchronous S3-compatible client built with the JDK alone. It uses path-style URLs and AWS Signature Version 4. The code is covered by the repository's MIT license.
## Build and test
Requires JDK 21 or newer. No Maven, Gradle, or third-party Java libraries are needed.
```sh
bash client/scripts/test.sh
bash client/scripts/build.sh
```
Run these commands from the ObjectStore repository root. The second command writes `client/dist/objectstore-client.jar` and `client/dist/javadoc/`.
The [AWT image manager](examples/README.md) is a small drag-and-drop app for trying uploads, listing, previews, downloads, and deletes against an existing bucket.
## Use
```java
import cloud.lunarsky.objectstore.client.ObjectStorageClient;
import cloud.lunarsky.objectstore.client.ObjectStorageClientBuilder;
import java.net.URI;
import java.nio.charset.StandardCharsets;
try (ObjectStorageClient storage = new ObjectStorageClientBuilder()
.endpoint(URI.create("https://storage.example.com"))
.region("us-east-1")
.credentials(accessKey, secretKey)
.build()) {
storage.putObject("photos", "hello.txt", "hello".getBytes(StandardCharsets.UTF_8), "text/plain");
try (ObjectStorageClient.ObjectData data = storage.getObject("photos", "hello.txt")) {
data.body().transferTo(System.out);
}
}
```
`getObject` returns an open stream. Close it even if you do not read every byte. The client requires HTTPS unless you explicitly call `allowInsecureHttp()` for a trusted local endpoint.
## Scope
This version implements single-request object PUT, GET, HEAD, DELETE, ListObjectsV2 pages, standard S3 multipart initiation/part upload/list/complete/abort, unfinished-upload listing, bucket/object ACL GET and PUT, and service detection. ACLs are S3 ACL requests, not a calculation of effective permissions from IAM policies or bucket policies. An S3 service with ACLs disabled can reject them; ObjectStore implements a limited ACL subset.
`getCapabilities()` reads ObjectStore's signed `GET /_objectstore/capabilities` response when available. It reports implemented S3 operation names, service version, storage mode, and configured limits. A listed operation means the service implements it; it does not establish that the current key may perform it or that every AWS option is supported. Older ObjectStore builds fall back to `/health` and leave unverified operations `UNKNOWN`. The response is not cached.
For a generic S3 endpoint, successful operations on this client are recorded as `SUPPORTED`; all untested operations remain `UNKNOWN`. Call `probeReadOnlyCapabilities(bucket, existingObjectKey)` to opt into safe read probes. An S3 denial or missing object leaves that operation `UNKNOWN`, not `UNSUPPORTED`. The client never infers features from a hostname or vendor header. The older `/health` identity fallback is self-reported, not cryptographic attestation.
Multipart upload sessions expose their bucket, key, and upload ID so a caller can persist them and inspect already uploaded parts after a restart. `uploadFileParts` sends a file in sequential parts and leaves completion to the caller. If it fails, the upload remains open for retry or explicit abort; keep the returned upload ID and do not change the source file.
The client does not yet implement presigned URLs, credential providers, automatic retries, region redirects, or streaming uploads of unknown length. A single-request file upload hashes the file before sending it; do not modify the file while the upload runs. The client does not retry writes because a lost response can leave their outcome uncertain.
## Errors
- `ObjectStorageException`: an HTTP error, with status, S3 error code, request ID, and a retryability hint.
- `TransportException`: connection or I/O failure. A write may already have completed.
- `ProtocolException`: an unexpected or malformed successful response.
The client does not include access keys, secret keys, or response bodies in exception messages.
+37
View File
@@ -0,0 +1,37 @@
# Examples
## AWT image manager
![Image manager previewing a cat photo in the local test service](awt-images/screenshot.png)
Cat photo in the screenshot: [IOP Publishing source image](https://ioppublishing.org/wp-content/uploads/2017/03/cat-web-cc0.jpg).
Run from the ObjectStore repository root with JDK 21 or newer:
```sh
bash client/examples/awt-images/run.sh
```
The example uses only the Java client and the JDK. Enter an S3-compatible endpoint, region, existing bucket, and access keys. Use **Connect** to verify listing access. Drag PNG, JPEG, GIF, or BMP files onto the window, or use **Add images**. Select an object to preview it; use **Download** or **Delete** to manage it. Objects go under the specified key prefix with a short random ID, so uploading the same filename does not silently replace an earlier image. The list loads 100 objects at a time.
The example does not create a bucket. When launched directly, it keeps credentials in memory and requires explicit opt-in for plain HTTP. Use HTTP only with a trusted test service. Uploads are limited to 64 MiB per file, and previews to 12 MiB. This is a small interoperability test app, not a production asset manager.
### Local Docker test
Prerequisites: JDK 21 or newer, Bash, Python 3, `curl`, and a graphical desktop session. Docker must be running and accessible to your user, and `127.0.0.1:9002` must be free for the test container.
To try the app with a separate ObjectStore service, run:
```sh
bash client/examples/awt-images/run-test.sh
```
The launcher builds its local image if needed, starts a test container on `127.0.0.1:9002`, and opens the app connected to a `photos` bucket. Set `OBJECTSTORE_TEST_IMAGE` to use a different image. Closing the app leaves the container and its dedicated data volume running; run the launcher again to reconnect. Its generated test credentials are stored in the container's Docker configuration.
When finished, remove only this example's container and volume:
```sh
docker stop objectstore-image-example
docker rm objectstore-image-example
docker volume rm objectstore-image-example-data
```
@@ -0,0 +1,503 @@
import cloud.lunarsky.objectstore.client.Capabilities;
import cloud.lunarsky.objectstore.client.ObjectStorageClient;
import cloud.lunarsky.objectstore.client.ObjectStorageClientBuilder;
import java.awt.BorderLayout;
import java.awt.Button;
import java.awt.Canvas;
import java.awt.Checkbox;
import java.awt.Color;
import java.awt.Dialog;
import java.awt.Dimension;
import java.awt.EventQueue;
import java.awt.FileDialog;
import java.awt.FlowLayout;
import java.awt.Font;
import java.awt.Frame;
import java.awt.Graphics;
import java.awt.GridLayout;
import java.awt.Label;
import java.awt.Panel;
import java.awt.TextField;
import java.awt.dnd.DnDConstants;
import java.awt.dnd.DropTarget;
import java.awt.dnd.DropTargetAdapter;
import java.awt.dnd.DropTargetDropEvent;
import java.awt.datatransfer.DataFlavor;
import java.awt.event.WindowAdapter;
import java.awt.event.WindowEvent;
import java.awt.image.BufferedImage;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.net.URI;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.UUID;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicLong;
import javax.imageio.ImageIO;
import javax.imageio.ImageReader;
import javax.imageio.stream.ImageInputStream;
import javax.imageio.stream.MemoryCacheImageInputStream;
/** A small, dependency-free image browser for testing an S3-compatible endpoint. */
@SuppressWarnings("serial")
public final class ImageManager extends Frame {
private static final long MAX_UPLOAD = 64L * 1024 * 1024;
private static final int MAX_PREVIEW = 12 * 1024 * 1024;
private static final Color BACKGROUND = new Color(29, 27, 38);
private static final Color FOREGROUND = new Color(231, 223, 240);
private final TextField endpoint = new TextField(env("S3_ENDPOINT", "http://localhost:9000"));
private final TextField region = new TextField(env("S3_REGION", "us-east-1"));
private final TextField bucket = new TextField(env("S3_BUCKET", "photos"));
private final TextField prefix = new TextField("images/");
private final TextField access = new TextField(env("S3_ACCESS_KEY", ""));
private final TextField secret = new TextField(env("S3_SECRET_KEY", ""));
private final Checkbox allowHttp = new Checkbox("Allow HTTP for trusted tests", null,
Boolean.parseBoolean(env("S3_ALLOW_HTTP", "false")));
private final java.awt.List images = new java.awt.List(15, false);
private final ImageCanvas preview = new ImageCanvas();
private final Label status = new Label("Enter a bucket and credentials, then connect. Drop images to upload.");
private final Button connectButton = new Button("Connect");
private final Button moreButton = new Button("Load more");
private final ExecutorService io = Executors.newSingleThreadExecutor(r -> {
Thread thread = new Thread(r, "objectstore-image-example");
thread.setDaemon(true);
return thread;
});
private final AtomicLong previewRevision = new AtomicLong();
private final List<ObjectStorageClient.ObjectEntry> entries = new ArrayList<>();
private volatile ObjectStorageClient client;
private volatile String activeBucket;
private volatile String activePrefix;
private String nextToken;
private ImageManager() {
super("ObjectStore image manager");
setLayout(new BorderLayout(8, 8));
setBackground(BACKGROUND);
setForeground(FOREGROUND);
setSize(920, 640);
setMinimumSize(new Dimension(650, 440));
setLocationRelativeTo(null);
secret.setEchoChar('\u2022');
Panel serviceFields = new Panel(new GridLayout(2, 4, 8, 3));
serviceFields.add(label("Endpoint"));
serviceFields.add(label("Region"));
serviceFields.add(label("Bucket"));
serviceFields.add(label("Key prefix"));
serviceFields.add(endpoint);
serviceFields.add(region);
serviceFields.add(bucket);
serviceFields.add(prefix);
Panel credentialFields = new Panel(new GridLayout(2, 2, 8, 3));
credentialFields.add(label("Access key"));
credentialFields.add(label("Secret key"));
credentialFields.add(access);
credentialFields.add(secret);
Panel connection = new Panel(new BorderLayout(0, 5));
connection.add(serviceFields, BorderLayout.NORTH);
connection.add(credentialFields, BorderLayout.CENTER);
Panel actions = new Panel(new FlowLayout(FlowLayout.LEFT, 8, 4));
Button upload = new Button("Add images...");
Button refresh = new Button("Refresh");
Button download = new Button("Download");
Button delete = new Button("Delete");
actions.add(allowHttp);
actions.add(connectButton);
actions.add(upload);
actions.add(refresh);
actions.add(moreButton);
actions.add(download);
actions.add(delete);
moreButton.setEnabled(false);
Panel top = new Panel(new BorderLayout(0, 5));
top.add(connection, BorderLayout.CENTER);
top.add(actions, BorderLayout.SOUTH);
add(top, BorderLayout.NORTH);
Panel listing = new Panel(new BorderLayout(0, 5));
listing.setPreferredSize(new Dimension(285, 400));
listing.add(label("Images in this prefix"), BorderLayout.NORTH);
listing.add(images, BorderLayout.CENTER);
add(listing, BorderLayout.WEST);
add(preview, BorderLayout.CENTER);
add(status, BorderLayout.SOUTH);
connectButton.addActionListener(event -> connect());
upload.addActionListener(event -> chooseImages());
refresh.addActionListener(event -> loadPage(true));
moreButton.addActionListener(event -> loadPage(false));
download.addActionListener(event -> downloadSelected());
delete.addActionListener(event -> deleteSelected());
images.addItemListener(event -> previewSelected());
installDropTarget(this);
installDropTarget(preview);
installDropTarget(images);
addWindowListener(new WindowAdapter() {
@Override public void windowClosing(WindowEvent event) {
io.shutdownNow();
ObjectStorageClient previous = client;
if (previous != null) Thread.ofVirtual().start(previous::close);
dispose();
}
});
if (Boolean.parseBoolean(env("S3_AUTO_CONNECT", "false"))) {
EventQueue.invokeLater(this::connect);
}
}
private void connect() {
String url = endpoint.getText().trim();
String location = region.getText().trim();
String name = bucket.getText().trim();
String folder = normalizePrefix(prefix.getText());
String user = access.getText().trim();
String password = secret.getText();
boolean insecure = allowHttp.getState();
connectButton.setEnabled(false);
setStatus("Connecting...", false);
io.execute(() -> {
ObjectStorageClient candidate = null;
try {
ObjectStorageClientBuilder builder = new ObjectStorageClientBuilder()
.endpoint(URI.create(url)).region(location).credentials(user, password)
.timeout(Duration.ofSeconds(30));
if (insecure) builder.allowInsecureHttp();
candidate = builder.build();
candidate.listObjects(name, folder, null, 1);
ObjectStorageClient previous = client;
client = candidate;
activeBucket = name;
activePrefix = folder;
candidate = null;
if (previous != null) previous.close();
String service = "S3-compatible service";
try {
Capabilities found = client.getCapabilities();
if (found.service() == Capabilities.ServiceKind.OBJECTSTORE) {
service = "ObjectStore" + (found.serviceVersion() == null ? "" : " " + found.serviceVersion());
}
} catch (IOException ignored) {
// Listing already established the connection; service detection is optional.
}
String connectedService = service;
EventQueue.invokeLater(() -> {
connectButton.setEnabled(true);
setStatus("Connected to " + connectedService + ".", false);
loadPage(true);
});
} catch (Exception error) {
if (candidate != null) candidate.close();
showFailure("Connection failed", error);
EventQueue.invokeLater(() -> connectButton.setEnabled(true));
}
});
}
private void loadPage(boolean first) {
if (client == null) {
setStatus("Connect first.", true);
return;
}
if (!first && nextToken == null) return;
String token = first ? null : nextToken;
moreButton.setEnabled(false);
setStatus(first ? "Loading images..." : "Loading more images...", false);
io.execute(() -> {
try {
ObjectStorageClient.ObjectPage page = client.listObjects(activeBucket, activePrefix, token, 100);
EventQueue.invokeLater(() -> {
if (first) {
previewRevision.incrementAndGet();
entries.clear();
images.removeAll();
preview.show(null, "Select an image to preview");
}
for (ObjectStorageClient.ObjectEntry entry : page.objects()) {
if (!isImage(entry.key())) continue;
entries.add(entry);
String name = entry.key().substring(activePrefix.length());
images.add(name + " · " + sizeLabel(entry.size()));
}
nextToken = page.nextContinuationToken();
moreButton.setEnabled(nextToken != null);
setStatus(entries.size() + " image(s) loaded" +
(nextToken == null ? "." : "; more available."), false);
});
} catch (Exception error) {
showFailure("Could not list images", error);
EventQueue.invokeLater(() -> moreButton.setEnabled(token != null));
}
});
}
private void chooseImages() {
FileDialog picker = new FileDialog(this, "Add images", FileDialog.LOAD);
picker.setMultipleMode(true);
picker.setVisible(true);
java.io.File[] selected = picker.getFiles();
if (selected.length > 0) uploadImages(List.of(selected));
}
private void uploadImages(List<java.io.File> files) {
if (client == null) {
setStatus("Connect first.", true);
return;
}
io.execute(() -> {
int uploaded = 0;
for (java.io.File file : files) {
try {
Path path = file.toPath();
String name = path.getFileName().toString();
String type = contentType(name);
if (type == null) throw new IOException("Unsupported image type");
if (!Files.isRegularFile(path) || Files.size(path) > MAX_UPLOAD)
throw new IOException("Image must be a file of at most 64 MiB");
String key = activePrefix + UUID.randomUUID() + "-" + name;
client.putObject(activeBucket, key, path, type);
uploaded++;
int completed = uploaded;
EventQueue.invokeLater(() -> setStatus("Uploaded " + completed + " of " + files.size() + ".", false));
} catch (Exception error) {
showFailure("Could not upload " + file.getName(), error);
}
}
if (uploaded > 0) EventQueue.invokeLater(() -> loadPage(true));
});
}
private void previewSelected() {
ObjectStorageClient.ObjectEntry entry = selectedEntry();
long revision = previewRevision.incrementAndGet();
if (entry == null) {
preview.show(null, "Select an image to preview");
return;
}
preview.show(null, "Loading preview...");
io.execute(() -> {
try (ObjectStorageClient.ObjectData data = client.getObject(activeBucket, entry.key())) {
if (data.length() > MAX_PREVIEW) throw new IOException("Preview exceeds 12 MiB");
byte[] bytes = data.body().readNBytes(MAX_PREVIEW + 1);
if (bytes.length > MAX_PREVIEW) throw new IOException("Preview exceeds 12 MiB");
BufferedImage image = decodePreview(bytes);
EventQueue.invokeLater(() -> {
if (previewRevision.get() == revision) preview.show(image, null);
});
} catch (Exception error) {
EventQueue.invokeLater(() -> {
if (previewRevision.get() == revision) preview.show(null, "Preview unavailable");
});
showFailure("Could not preview image", error);
}
});
}
private void downloadSelected() {
ObjectStorageClient.ObjectEntry entry = selectedEntry();
if (entry == null) {
setStatus("Select an image first.", true);
return;
}
FileDialog picker = new FileDialog(this, "Save image", FileDialog.SAVE);
picker.setFile(Path.of(entry.key()).getFileName().toString());
picker.setVisible(true);
if (picker.getFile() == null) return;
Path destination = Path.of(picker.getDirectory(), picker.getFile());
if (Files.exists(destination) && !confirm("Replace this file?", destination.toString())) return;
io.execute(() -> {
Path temporary = null;
try {
Path parent = destination.toAbsolutePath().getParent();
temporary = Files.createTempFile(parent, ".objectstore-image-", ".part");
try (ObjectStorageClient.ObjectData data = client.getObject(activeBucket, entry.key())) {
Files.copy(data.body(), temporary, StandardCopyOption.REPLACE_EXISTING);
}
Files.move(temporary, destination, StandardCopyOption.REPLACE_EXISTING);
EventQueue.invokeLater(() -> setStatus("Saved " + destination.getFileName() + ".", false));
} catch (Exception error) {
showFailure("Download failed", error);
} finally {
if (temporary != null) {
try { Files.deleteIfExists(temporary); } catch (IOException ignored) { }
}
}
});
}
private void deleteSelected() {
ObjectStorageClient.ObjectEntry entry = selectedEntry();
if (entry == null) {
setStatus("Select an image first.", true);
return;
}
if (!confirm("Delete this image?", entry.key())) return;
io.execute(() -> {
try {
client.deleteObject(activeBucket, entry.key());
EventQueue.invokeLater(() -> {
previewRevision.incrementAndGet();
loadPage(true);
});
} catch (Exception error) { showFailure("Delete failed", error); }
});
}
private ObjectStorageClient.ObjectEntry selectedEntry() {
int index = images.getSelectedIndex();
return index < 0 || index >= entries.size() ? null : entries.get(index);
}
private void installDropTarget(java.awt.Component target) {
new DropTarget(target, DnDConstants.ACTION_COPY, new DropTargetAdapter() {
@Override public void drop(DropTargetDropEvent event) {
if (!event.isDataFlavorSupported(DataFlavor.javaFileListFlavor)) {
event.rejectDrop();
return;
}
try {
event.acceptDrop(DnDConstants.ACTION_COPY);
Object value = event.getTransferable().getTransferData(DataFlavor.javaFileListFlavor);
List<?> dropped = (List<?>) value;
List<java.io.File> files = new ArrayList<>();
for (Object item : dropped) if (item instanceof java.io.File file) files.add(file);
event.dropComplete(true);
EventQueue.invokeLater(() -> uploadImages(files));
} catch (Exception error) {
event.dropComplete(false);
showFailure("Drop failed", error);
}
}
}, true);
}
private boolean confirm(String title, String detail) {
Dialog dialog = new Dialog(this, title, true);
dialog.setLayout(new BorderLayout(12, 12));
dialog.add(new Label(detail), BorderLayout.CENTER);
Panel buttons = new Panel(new FlowLayout(FlowLayout.RIGHT));
boolean[] accepted = { false };
Button cancel = new Button("Cancel");
Button proceed = new Button("Continue");
cancel.addActionListener(event -> dialog.dispose());
proceed.addActionListener(event -> { accepted[0] = true; dialog.dispose(); });
buttons.add(cancel);
buttons.add(proceed);
dialog.add(buttons, BorderLayout.SOUTH);
dialog.setSize(490, 120);
dialog.setLocationRelativeTo(this);
dialog.setVisible(true);
return accepted[0];
}
private void showFailure(String action, Exception error) {
EventQueue.invokeLater(() -> setStatus(action + ": " + error.getMessage(), true));
}
private void setStatus(String message, boolean failed) {
status.setForeground(failed ? new Color(245, 143, 157) : FOREGROUND);
status.setText(message);
}
private static Label label(String text) {
return new Label(text);
}
private static String normalizePrefix(String value) {
String cleaned = value.trim().replace('\\', '/');
while (cleaned.startsWith("/")) cleaned = cleaned.substring(1);
return cleaned.isEmpty() || cleaned.endsWith("/") ? cleaned : cleaned + "/";
}
private static String contentType(String name) {
String lower = name.toLowerCase(Locale.ROOT);
if (lower.endsWith(".png")) return "image/png";
if (lower.endsWith(".jpg") || lower.endsWith(".jpeg")) return "image/jpeg";
if (lower.endsWith(".gif")) return "image/gif";
if (lower.endsWith(".bmp")) return "image/bmp";
return null;
}
private static boolean isImage(String name) {
return contentType(name) != null;
}
private static String sizeLabel(long bytes) {
return bytes < 1024 ? bytes + " B" : String.format(Locale.ROOT, "%.1f KiB", bytes / 1024.0);
}
private static BufferedImage decodePreview(byte[] bytes) throws IOException {
try (ImageInputStream stream = new MemoryCacheImageInputStream(new ByteArrayInputStream(bytes))) {
var readers = ImageIO.getImageReaders(stream);
if (!readers.hasNext()) throw new IOException("Unsupported image data");
ImageReader reader = readers.next();
try {
reader.setInput(stream, true, true);
int width = reader.getWidth(0);
int height = reader.getHeight(0);
if (width < 1 || height < 1 || width > 16000 || height > 16000 ||
(long) width * height > 40_000_000)
throw new IOException("Image dimensions are too large for preview");
int step = Math.max(1, (Math.max(width, height) + 1199) / 1200);
var parameters = reader.getDefaultReadParam();
parameters.setSourceSubsampling(step, step, 0, 0);
return reader.read(0, parameters);
} finally { reader.dispose(); }
}
}
private static String env(String name, String fallback) {
String value = System.getenv(name);
return value == null ? fallback : value;
}
@SuppressWarnings("serial")
private static final class ImageCanvas extends Canvas {
private BufferedImage image;
private String message = "Select an image to preview";
private ImageCanvas() {
setBackground(new Color(21, 20, 29));
setForeground(FOREGROUND);
setFont(new Font(Font.SANS_SERIF, Font.PLAIN, 16));
}
private void show(BufferedImage value, String text) {
image = value;
message = text;
repaint();
}
@Override public void paint(Graphics graphics) {
int width = getWidth();
int height = getHeight();
if (image == null) {
graphics.setColor(FOREGROUND);
graphics.drawString(message, 20, Math.max(35, height / 2));
return;
}
double scale = Math.min((width - 24.0) / image.getWidth(),
(height - 24.0) / image.getHeight());
scale = Math.max(0.01, Math.min(scale, 1.0));
int drawWidth = (int) Math.round(image.getWidth() * scale);
int drawHeight = (int) Math.round(image.getHeight() * scale);
graphics.drawImage(image, (width - drawWidth) / 2, (height - drawHeight) / 2,
drawWidth, drawHeight, null);
}
}
public static void main(String[] args) {
EventQueue.invokeLater(() -> new ImageManager().setVisible(true));
}
}
+51
View File
@@ -0,0 +1,51 @@
#!/usr/bin/env bash
set -euo pipefail
client_dir="$(cd "$(dirname "$0")/../.." && pwd)"
project_dir="$(cd "$client_dir/.." && pwd)"
container="objectstore-image-example"
volume="objectstore-image-example-data"
image="${OBJECTSTORE_TEST_IMAGE:-objectstore-image-example:local}"
if ! docker container inspect "$container" >/dev/null 2>&1; then
if ! docker image inspect "$image" >/dev/null 2>&1; then
docker build --tag "$image" "$project_dir"
fi
export S3_ACCESS_KEY="ImageExampleTest1"
export S3_SECRET_KEY="$(python3 -c 'import secrets; print(secrets.token_hex(32))')"
docker volume create "$volume" >/dev/null
docker run --detach --name "$container" \
--publish 127.0.0.1:9002:9000 \
--volume "$volume:/data" \
--env S3_ACCESS_KEY --env S3_SECRET_KEY \
--env S3_BUCKET=photos --env S3_REGION=us-east-1 \
--env MAX_OBJECT_BYTES=67108864 --env MAX_TOTAL_BYTES=1073741824 \
"$image" >/dev/null
else
while IFS='=' read -r key value; do
case "$key" in
S3_ACCESS_KEY) export S3_ACCESS_KEY="$value" ;;
S3_SECRET_KEY) export S3_SECRET_KEY="$value" ;;
esac
done < <(docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' "$container")
if [[ "$(docker inspect --format '{{.State.Running}}' "$container")" != true ]]; then
docker start "$container" >/dev/null
fi
fi
for attempt in {1..40}; do
if curl --silent --fail --output /dev/null http://127.0.0.1:9002/health; then
break
fi
sleep 0.25
done
if ! curl --silent --fail --output /dev/null http://127.0.0.1:9002/health; then
echo "The isolated ObjectStore container did not become healthy; check docker logs $container." >&2
exit 1
fi
export S3_ENDPOINT=http://127.0.0.1:9002
export S3_REGION=us-east-1
export S3_BUCKET=photos
export S3_ALLOW_HTTP=true
export S3_AUTO_CONNECT=true
exec "$client_dir/examples/awt-images/run.sh"
+11
View File
@@ -0,0 +1,11 @@
#!/usr/bin/env bash
set -euo pipefail
client_dir="$(cd "$(dirname "$0")/../.." && pwd)"
if [[ ! -f "$client_dir/dist/objectstore-client.jar" ]]; then
bash "$client_dir/scripts/build.sh"
fi
classes="$client_dir/dist/examples/awt-images"
mkdir -p "$classes"
javac --release 21 -cp "$client_dir/dist/objectstore-client.jar" \
-d "$classes" "$client_dir/examples/awt-images/ImageManager.java"
exec java -cp "$classes:$client_dir/dist/objectstore-client.jar" ImageManager
Binary file not shown.

After

Width:  |  Height:  |  Size: 381 KiB

+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
rm -rf dist/classes
mkdir -p dist/classes
find src/main/java -name '*.java' -print0 |
xargs -0 javac --release 21 -d dist/classes
mkdir -p dist/classes/META-INF
cp ../LICENSE dist/classes/META-INF/LICENSE
jar --create --file dist/objectstore-client.jar -C dist/classes .
javadoc --release 21 -quiet -Xdoclint:reference,syntax,html -d dist/javadoc \
$(find src/main/java -name '*.java' -print)
echo "Built dist/objectstore-client.jar"
+9
View File
@@ -0,0 +1,9 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
build_dir="$(mktemp -d)"
trap 'rm -rf "$build_dir"' EXIT
find src/main/java src/test/java -name '*.java' -print0 |
xargs -0 javac --release 21 -d "$build_dir"
java -cp "$build_dir" cloud.lunarsky.objectstore.client.ClientTest
java -cp "$build_dir" cloud.lunarsky.objectstore.client.MultipartClientTest
@@ -0,0 +1,27 @@
package cloud.lunarsky.objectstore.client;
import java.util.List;
import java.util.Objects;
/** A bucket or object ACL. This is not an effective-permissions calculation. */
public record AclPolicy(String ownerId, List<Grant> grants) {
/** A canonical user or predefined S3 group. */
public enum GranteeType { CANONICAL_USER, GROUP }
/** An S3 ACL permission. */
public enum Permission { READ, WRITE, READ_ACP, WRITE_ACP, FULL_CONTROL }
/** One ACL grant to a canonical user ID or S3 group URI. */
public record Grant(GranteeType type, String grantee, Permission permission) {
public Grant {
Objects.requireNonNull(type, "type");
if (grantee == null || grantee.isBlank()) throw new IllegalArgumentException("grantee is required");
Objects.requireNonNull(permission, "permission");
}
}
public AclPolicy {
if (ownerId == null || ownerId.isBlank()) throw new IllegalArgumentException("owner ID is required");
grants = List.copyOf(grants);
}
}
@@ -0,0 +1,43 @@
package cloud.lunarsky.objectstore.client;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
/** Reported or observed operation support, separate from the caller's authorization. */
public record Capabilities(ServiceKind service, Map<String, Support> operations, Map<String, Long> limits,
String serviceVersion, String storageMode, boolean completeOperationInventory) {
private static final Set<String> KNOWN_OPERATIONS = Set.of(
"ListBuckets", "CreateBucket", "HeadBucket", "DeleteBucket", "ListObjectsV2",
"PutObject", "GetObject", "HeadObject", "DeleteObject", "CopyObject",
"GetObjectTagging", "PutObjectTagging", "DeleteObjectTagging",
"CreateMultipartUpload", "UploadPart", "ListParts", "CompleteMultipartUpload",
"AbortMultipartUpload", "ListMultipartUploads", "GetBucketVersioning",
"PutBucketVersioning", "ListObjectVersions", "GetBucketAcl", "PutBucketAcl",
"GetObjectAcl", "PutObjectAcl");
/** Identifies a self-reported ObjectStore service or an unrecognized S3-compatible service. */
public enum ServiceKind { OBJECTSTORE, UNKNOWN_S3 }
/** Three-state feature support; a generic S3 server cannot be inferred from its hostname. */
public enum Support { SUPPORTED, UNSUPPORTED, UNKNOWN }
public Capabilities {
Objects.requireNonNull(service, "service");
operations = Map.copyOf(operations);
limits = Map.copyOf(limits);
}
/** Constructs a result without a server manifest or configured limits. */
public Capabilities(ServiceKind service, Map<String, Support> operations) {
this(service, operations, Map.of(), null, null, false);
}
/** Returns support for a named operation. A missing manifest or denied probe stays UNKNOWN. */
public Support support(String operation) {
Support observed = operations.get(operation);
if (observed != null) return observed;
if (completeOperationInventory && KNOWN_OPERATIONS.contains(operation)) return Support.UNSUPPORTED;
return Support.UNKNOWN;
}
}
@@ -0,0 +1,194 @@
package cloud.lunarsky.objectstore.client;
import java.math.BigDecimal;
import java.nio.ByteBuffer;
import java.nio.charset.CharacterCodingException;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
final class Json {
private Json() { }
static Object parse(byte[] bytes) throws ProtocolException {
String source;
try { source = StandardCharsets.UTF_8.newDecoder().decode(ByteBuffer.wrap(bytes)).toString(); }
catch (CharacterCodingException e) { throw new ProtocolException("Capability JSON is not UTF-8", e); }
Parser parser = new Parser(source);
Object value = parser.value(0);
parser.space();
if (parser.index != source.length()) throw new ProtocolException("Trailing capability JSON data");
return value;
}
@SuppressWarnings("unchecked")
static Map<String, Object> object(Object value, String name) throws ProtocolException {
if (!(value instanceof Map<?, ?>)) throw new ProtocolException(name + " must be an object");
return (Map<String, Object>) value;
}
static List<?> array(Object value, String name) throws ProtocolException {
if (!(value instanceof List<?> list)) throw new ProtocolException(name + " must be an array");
return list;
}
static String string(Object value, String name) throws ProtocolException {
if (!(value instanceof String text) || text.isBlank())
throw new ProtocolException(name + " must be a nonempty string");
return text;
}
static long integer(Object value, String name) throws ProtocolException {
if (!(value instanceof BigDecimal number)) throw new ProtocolException(name + " must be an integer");
try { return number.longValueExact(); }
catch (ArithmeticException e) { throw new ProtocolException(name + " is out of range", e); }
}
private static final class Parser {
private final String source;
private int index;
private Parser(String source) { this.source = source; }
private void space() {
while (index < source.length() && (source.charAt(index) == ' ' || source.charAt(index) == '\n' ||
source.charAt(index) == '\r' || source.charAt(index) == '\t')) index++;
}
private Object value(int depth) throws ProtocolException {
if (depth > 16) throw new ProtocolException("Capability JSON is too deeply nested");
space();
if (index >= source.length()) throw new ProtocolException("Incomplete capability JSON");
return switch (source.charAt(index)) {
case '{' -> object(depth + 1);
case '[' -> array(depth + 1);
case '"' -> string();
case 't' -> literal("true", Boolean.TRUE);
case 'f' -> literal("false", Boolean.FALSE);
case 'n' -> literal("null", null);
default -> number();
};
}
private Map<String, Object> object(int depth) throws ProtocolException {
index++;
space();
Map<String, Object> result = new HashMap<>();
if (take('}')) return result;
do {
space();
if (index >= source.length() || source.charAt(index) != '"')
throw new ProtocolException("Capability JSON object key is missing");
String key = string();
space();
require(':');
if (result.containsKey(key)) throw new ProtocolException("Duplicate capability JSON key");
result.put(key, value(depth));
if (result.size() > 256) throw new ProtocolException("Capability JSON object is too large");
space();
if (take('}')) return result;
require(',');
} while (true);
}
private List<Object> array(int depth) throws ProtocolException {
index++;
space();
List<Object> result = new ArrayList<>();
if (take(']')) return result;
do {
result.add(value(depth));
if (result.size() > 2048) throw new ProtocolException("Capability JSON array is too large");
space();
if (take(']')) return result;
require(',');
} while (true);
}
private String string() throws ProtocolException {
index++;
StringBuilder result = new StringBuilder();
while (index < source.length()) {
char current = source.charAt(index++);
if (current == '"') {
for (int i = 0; i < result.length(); i++) {
char unit = result.charAt(i);
if (Character.isHighSurrogate(unit)) {
if (++i >= result.length() || !Character.isLowSurrogate(result.charAt(i)))
throw new ProtocolException("Invalid JSON Unicode surrogate");
} else if (Character.isLowSurrogate(unit))
throw new ProtocolException("Invalid JSON Unicode surrogate");
}
return result.toString();
}
if (current < 0x20) throw new ProtocolException("Unescaped control character in JSON string");
if (current != '\\') { result.append(current); continue; }
if (index >= source.length()) throw new ProtocolException("Incomplete JSON escape");
char escaped = source.charAt(index++);
switch (escaped) {
case '"', '\\', '/' -> result.append(escaped);
case 'b' -> result.append('\b');
case 'f' -> result.append('\f');
case 'n' -> result.append('\n');
case 'r' -> result.append('\r');
case 't' -> result.append('\t');
case 'u' -> {
if (index + 4 > source.length()) throw new ProtocolException("Incomplete Unicode escape");
int unit = 0;
for (int i = 0; i < 4; i++) {
int digit = Character.digit(source.charAt(index++), 16);
if (digit < 0) throw new ProtocolException("Invalid Unicode escape");
unit = unit * 16 + digit;
}
result.append((char) unit);
}
default -> throw new ProtocolException("Invalid JSON escape");
}
}
throw new ProtocolException("Unterminated JSON string");
}
private Object literal(String expected, Object value) throws ProtocolException {
if (!source.startsWith(expected, index)) throw new ProtocolException("Invalid JSON literal");
index += expected.length();
return value;
}
private BigDecimal number() throws ProtocolException {
int start = index;
if (take('-') && index >= source.length()) throw new ProtocolException("Invalid JSON number");
if (take('0')) {
if (index < source.length() && Character.isDigit(source.charAt(index)))
throw new ProtocolException("Invalid JSON number");
} else {
if (index >= source.length() || source.charAt(index) < '1' || source.charAt(index) > '9')
throw new ProtocolException("Invalid JSON number");
while (index < source.length() && source.charAt(index) >= '0' && source.charAt(index) <= '9') index++;
}
if (take('.')) {
int first = index;
while (index < source.length() && source.charAt(index) >= '0' && source.charAt(index) <= '9') index++;
if (first == index) throw new ProtocolException("Invalid JSON fraction");
}
if (take('e') || take('E')) {
if (!take('+')) take('-');
int first = index;
while (index < source.length() && source.charAt(index) >= '0' && source.charAt(index) <= '9') index++;
if (first == index) throw new ProtocolException("Invalid JSON exponent");
}
try { return new BigDecimal(source.substring(start, index)); }
catch (NumberFormatException e) { throw new ProtocolException("Invalid JSON number", e); }
}
private boolean take(char value) {
if (index < source.length() && source.charAt(index) == value) { index++; return true; }
return false;
}
private void require(char value) throws ProtocolException {
if (!take(value)) throw new ProtocolException("Expected '" + value + "' in capability JSON");
}
}
}
@@ -0,0 +1,633 @@
package cloud.lunarsky.objectstore.client;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.net.URLDecoder;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.time.Clock;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Base64;
import java.util.Comparator;
import java.util.HashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import org.w3c.dom.Element;
/** A synchronous, path-style S3 client. Instances are safe to share between threads. */
public final class ObjectStorageClient implements AutoCloseable {
private static final int MAX_XML = 4 * 1024 * 1024;
private static final int MAX_ERROR = 64 * 1024;
private static final String EMPTY_HASH = SigV4.hash(new byte[0]);
private final URI endpoint;
private final String region;
private final String accessKey;
private final String secretKey;
private final Duration timeout;
private final Clock clock;
private final HttpClient http;
private final Set<String> observedOperations = ConcurrentHashMap.newKeySet();
ObjectStorageClient(URI endpoint, String region, String accessKey, String secretKey,
Duration timeout, Clock clock) {
this.endpoint = endpoint;
this.region = region;
this.accessKey = accessKey;
this.secretKey = secretKey;
this.timeout = timeout;
this.clock = clock;
this.http = HttpClient.newBuilder().followRedirects(HttpClient.Redirect.NEVER)
.connectTimeout(timeout).version(HttpClient.Version.HTTP_1_1).build();
}
/** Returns one open object stream. The caller must close the returned result. */
public ObjectData getObject(String bucket, String key) throws IOException {
HttpResponse<InputStream> response = send("GET", objectPath(bucket, key), Map.of(), Map.of(),
HttpRequest.BodyPublishers.noBody(), EMPTY_HASH);
if (!successful(response)) {
try (InputStream ignored = response.body()) { throw failure(response); }
}
observedOperations.add("GetObject");
return new ObjectData(response.body(), length(response), response.headers().firstValue("content-type").orElse(null),
response.headers().firstValue("etag").orElse(null));
}
/** Returns object headers without downloading content. */
public ObjectMetadata headObject(String bucket, String key) throws IOException {
HttpResponse<InputStream> response = send("HEAD", objectPath(bucket, key), Map.of(), Map.of(),
HttpRequest.BodyPublishers.noBody(), EMPTY_HASH);
try (InputStream ignored = response.body()) {
if (!successful(response)) throw failure(response);
observedOperations.add("HeadObject");
return new ObjectMetadata(length(response), response.headers().firstValue("content-type").orElse(null),
response.headers().firstValue("etag").orElse(null));
}
}
/** Uploads an immutable byte array as one object. No automatic write retry is performed. */
public void putObject(String bucket, String key, byte[] content, String contentType) throws IOException {
Objects.requireNonNull(content, "content");
put(bucket, key, HttpRequest.BodyPublishers.ofByteArray(content), SigV4.hash(content), contentType);
}
/** Uploads a file. Do not change the file between hashing and completion of the upload. */
public void putObject(String bucket, String key, Path file, String contentType) throws IOException {
Objects.requireNonNull(file, "file");
String hash;
try (InputStream input = Files.newInputStream(file)) {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
byte[] buffer = new byte[65536];
for (int count; (count = input.read(buffer)) >= 0; ) digest.update(buffer, 0, count);
hash = java.util.HexFormat.of().formatHex(digest.digest());
} catch (NoSuchAlgorithmException e) { throw new IllegalStateException(e); }
put(bucket, key, HttpRequest.BodyPublishers.ofFile(file), hash, contentType);
}
private void put(String bucket, String key, HttpRequest.BodyPublisher body, String hash,
String contentType) throws IOException {
Map<String, String> headers = contentType == null ? Map.of() : Map.of("content-type", contentType);
HttpResponse<InputStream> response = send("PUT", objectPath(bucket, key), Map.of(), headers, body, hash);
try (InputStream ignored = response.body()) {
if (!successful(response)) throw failure(response);
observedOperations.add("PutObject");
}
}
/** Deletes the current object. A versioned backend may create a delete marker. */
public void deleteObject(String bucket, String key) throws IOException {
HttpResponse<InputStream> response = send("DELETE", objectPath(bucket, key), Map.of(), Map.of(),
HttpRequest.BodyPublishers.noBody(), EMPTY_HASH);
try (InputStream ignored = response.body()) {
if (!successful(response)) throw failure(response);
observedOperations.add("DeleteObject");
}
}
/** Lists one page of objects using S3 ListObjectsV2. Pass the returned token to get the next page. */
public ObjectPage listObjects(String bucket, String prefix, String continuationToken, int maxKeys)
throws IOException {
if (maxKeys < 1 || maxKeys > 1000) throw new IllegalArgumentException("maxKeys must be 1..1000");
Map<String, String> query = new HashMap<>();
query.put("list-type", "2");
query.put("encoding-type", "url");
query.put("max-keys", Integer.toString(maxKeys));
if (prefix != null) query.put("prefix", prefix);
if (continuationToken != null) query.put("continuation-token", continuationToken);
Element root = xml("GET", bucketPath(bucket), query);
requireRoot(root, "ListBucketResult");
observedOperations.add("ListObjectsV2");
List<ObjectEntry> entries = new ArrayList<>();
var nodes = Xml.descendants(root, "Contents");
for (int i = 0; i < nodes.getLength(); i++) {
Element entry = (Element) nodes.item(i);
String key = Xml.text(entry, "Key");
String size = Xml.text(entry, "Size");
if (key == null || size == null) throw new ProtocolException("Incomplete object listing");
try {
entries.add(new ObjectEntry(URLDecoder.decode(key.replace("+", "%2B"), StandardCharsets.UTF_8),
Long.parseLong(size), Xml.text(entry, "ETag")));
} catch (IllegalArgumentException e) { throw new ProtocolException("Invalid object listing entry", e); }
}
String next = Xml.text(root, "NextContinuationToken");
if ("true".equalsIgnoreCase(Xml.text(root, "IsTruncated")) && (next == null || next.isEmpty()))
throw new ProtocolException("Truncated listing has no continuation token");
return new ObjectPage(entries, next);
}
/** Starts a standard S3 multipart upload. Keep the returned upload ID to resume or abort later. */
public MultipartUpload createMultipartUpload(String bucket, String key, String contentType) throws IOException {
String path = objectPath(bucket, key);
Map<String, String> headers = contentType == null ? Map.of() : Map.of("content-type", contentType);
HttpResponse<InputStream> response = send("POST", path, Map.of("uploads", ""), headers,
HttpRequest.BodyPublishers.noBody(), EMPTY_HASH);
Element root = responseXml(response, "InitiateMultipartUploadResult");
String id = Xml.text(root, "UploadId");
if (id == null || id.isBlank()) throw new ProtocolException("Multipart initiation has no upload ID");
observedOperations.add("CreateMultipartUpload");
return new MultipartUpload(bucket, key, id);
}
/** Uploads one part. The ETag in the result must be supplied when completing the upload. */
public Part uploadPart(MultipartUpload upload, int partNumber, byte[] content) throws IOException {
Objects.requireNonNull(upload, "upload");
Objects.requireNonNull(content, "content");
checkPartNumber(partNumber);
HttpResponse<InputStream> response = send("PUT", objectPath(upload.bucket(), upload.key()),
Map.of("uploadId", upload.uploadId(), "partNumber", Integer.toString(partNumber)), Map.of(),
HttpRequest.BodyPublishers.ofByteArray(content), SigV4.hash(content));
try (InputStream ignored = response.body()) {
if (!successful(response)) throw failure(response);
String etag = response.headers().firstValue("etag").orElse(null);
if (etag == null || etag.isBlank()) throw new ProtocolException("Part upload has no ETag");
observedOperations.add("UploadPart");
return new Part(partNumber, etag, content.length);
}
}
/**
* Uploads an immutable file as sequential parts without completing it. The caller owns the
* upload session and can retry, list parts, complete, or abort it after any failure.
* Each part is held in memory once; partSize must be 5 to 128 MiB.
*/
public List<Part> uploadFileParts(MultipartUpload upload, Path file, int partSize) throws IOException {
Objects.requireNonNull(upload, "upload");
Objects.requireNonNull(file, "file");
if (partSize < 5 * 1024 * 1024 || partSize > 128 * 1024 * 1024)
throw new IllegalArgumentException("partSize must be 5..128 MiB");
long size = Files.size(file);
if (size < 1 || (size + partSize - 1L) / partSize > 10000)
throw new IllegalArgumentException("file requires 1..10000 parts");
List<Part> parts = new ArrayList<>();
try (InputStream input = Files.newInputStream(file)) {
long remaining = size;
for (int number = 1; remaining > 0; number++) {
int length = (int) Math.min(remaining, partSize);
byte[] bytes = input.readNBytes(length);
if (bytes.length != length) throw new IOException("File changed during multipart upload");
parts.add(uploadPart(upload, number, bytes));
remaining -= length;
}
if (input.read() != -1) throw new IOException("File changed during multipart upload");
}
return List.copyOf(parts);
}
/** Lists one page of uploaded parts for resume or verification. */
public PartPage listParts(MultipartUpload upload, int partNumberMarker, int maxParts) throws IOException {
Objects.requireNonNull(upload, "upload");
if (partNumberMarker < 0 || partNumberMarker > 10000 || maxParts < 1 || maxParts > 1000)
throw new IllegalArgumentException("invalid part marker or page size");
Element root = xml("GET", objectPath(upload.bucket(), upload.key()),
Map.of("uploadId", upload.uploadId(), "part-number-marker", Integer.toString(partNumberMarker),
"max-parts", Integer.toString(maxParts)));
requireRoot(root, "ListPartsResult");
observedOperations.add("ListParts");
List<Part> parts = new ArrayList<>();
var nodes = Xml.descendants(root, "Part");
for (int i = 0; i < nodes.getLength(); i++) {
Element entry = (Element) nodes.item(i);
try {
parts.add(new Part(Integer.parseInt(requiredText(entry, "PartNumber")),
requiredText(entry, "ETag"), Long.parseLong(requiredText(entry, "Size"))));
} catch (NumberFormatException e) { throw new ProtocolException("Invalid part listing entry", e); }
}
boolean truncated = Boolean.parseBoolean(Xml.text(root, "IsTruncated"));
int next = partNumberMarker;
if (truncated) {
try { next = Integer.parseInt(requiredText(root, "NextPartNumberMarker")); }
catch (NumberFormatException e) { throw new ProtocolException("Invalid next part marker", e); }
if (next <= partNumberMarker) throw new ProtocolException("Part listing did not advance");
}
return new PartPage(parts, truncated, next);
}
/** Completes the upload using the exact part numbers and ETags returned by the service. */
public String completeMultipartUpload(MultipartUpload upload, List<Part> parts) throws IOException {
Objects.requireNonNull(upload, "upload");
Objects.requireNonNull(parts, "parts");
if (parts.isEmpty() || parts.size() > 10000) throw new IllegalArgumentException("1..10000 parts required");
List<Part> ordered = new ArrayList<>(parts);
ordered.sort(Comparator.comparingInt(Part::number));
StringBuilder xml = new StringBuilder("<CompleteMultipartUpload>");
int previous = 0;
for (Part part : ordered) {
Objects.requireNonNull(part, "part");
checkPartNumber(part.number());
if (part.number() == previous) throw new IllegalArgumentException("duplicate part number");
previous = part.number();
xml.append("<Part><PartNumber>").append(part.number()).append("</PartNumber><ETag>")
.append(Xml.escape(part.etag())).append("</ETag></Part>");
}
byte[] bytes = xml.append("</CompleteMultipartUpload>").toString().getBytes(StandardCharsets.UTF_8);
HttpResponse<InputStream> response = send("POST", objectPath(upload.bucket(), upload.key()),
Map.of("uploadId", upload.uploadId()), Map.of("content-type", "application/xml"),
HttpRequest.BodyPublishers.ofByteArray(bytes), SigV4.hash(bytes));
Element root = responseXml(response, "CompleteMultipartUploadResult");
observedOperations.add("CompleteMultipartUpload");
return requiredText(root, "ETag");
}
/** Aborts an unfinished upload. A completed upload cannot be aborted. */
public void abortMultipartUpload(MultipartUpload upload) throws IOException {
Objects.requireNonNull(upload, "upload");
HttpResponse<InputStream> response = send("DELETE", objectPath(upload.bucket(), upload.key()),
Map.of("uploadId", upload.uploadId()), Map.of(), HttpRequest.BodyPublishers.noBody(), EMPTY_HASH);
try (InputStream ignored = response.body()) {
if (!successful(response)) throw failure(response);
observedOperations.add("AbortMultipartUpload");
}
}
/** Lists one page of unfinished uploads in a bucket. */
public MultipartUploadPage listMultipartUploads(String bucket, String prefix, String keyMarker,
String uploadIdMarker, int maxUploads) throws IOException {
if (maxUploads < 1 || maxUploads > 1000) throw new IllegalArgumentException("maxUploads must be 1..1000");
if (uploadIdMarker != null && keyMarker == null)
throw new IllegalArgumentException("upload ID marker requires key marker");
Map<String, String> query = new HashMap<>();
query.put("uploads", "");
query.put("max-uploads", Integer.toString(maxUploads));
if (prefix != null) query.put("prefix", prefix);
if (keyMarker != null) query.put("key-marker", keyMarker);
if (uploadIdMarker != null) query.put("upload-id-marker", uploadIdMarker);
Element root = xml("GET", bucketPath(bucket), query);
requireRoot(root, "ListMultipartUploadsResult");
observedOperations.add("ListMultipartUploads");
List<MultipartUpload> uploads = new ArrayList<>();
var nodes = Xml.descendants(root, "Upload");
for (int i = 0; i < nodes.getLength(); i++) {
Element entry = (Element) nodes.item(i);
uploads.add(new MultipartUpload(bucket, requiredText(entry, "Key"), requiredText(entry, "UploadId")));
}
boolean truncated = Boolean.parseBoolean(Xml.text(root, "IsTruncated"));
String nextKey = Xml.text(root, "NextKeyMarker");
String nextId = Xml.text(root, "NextUploadIdMarker");
if (truncated && (nextKey == null || nextId == null))
throw new ProtocolException("Truncated upload listing has no markers");
return new MultipartUploadPage(uploads, truncated, nextKey, nextId);
}
private static String requiredText(Element element, String name) throws ProtocolException {
String value = Xml.text(element, name);
if (value == null || value.isBlank()) throw new ProtocolException("Multipart response missing " + name);
return value;
}
private static void checkPartNumber(int number) {
if (number < 1 || number > 10000) throw new IllegalArgumentException("part number must be 1..10000");
}
/** Gets a bucket ACL. The backend may reject ACL operations when ACLs are disabled. */
public AclPolicy getBucketAcl(String bucket) throws IOException {
AclPolicy policy = readAcl(bucketPath(bucket));
observedOperations.add("GetBucketAcl");
return policy;
}
/** Gets an object ACL. This does not calculate permissions from policies or other grants. */
public AclPolicy getObjectAcl(String bucket, String key) throws IOException {
AclPolicy policy = readAcl(objectPath(bucket, key));
observedOperations.add("GetObjectAcl");
return policy;
}
/** Replaces a bucket ACL. No automatic retry is performed. */
public void putBucketAcl(String bucket, AclPolicy policy) throws IOException {
writeAcl(bucketPath(bucket), policy);
observedOperations.add("PutBucketAcl");
}
/** Replaces an object ACL. No automatic retry is performed. */
public void putObjectAcl(String bucket, String key, AclPolicy policy) throws IOException {
writeAcl(objectPath(bucket, key), policy);
observedOperations.add("PutObjectAcl");
}
private AclPolicy readAcl(String path) throws IOException {
Element root = xml("GET", path, Map.of("acl", ""));
requireRoot(root, "AccessControlPolicy");
Element owner = Xml.child(root, "Owner");
String ownerId = owner == null ? null : Xml.text(owner, "ID");
if (ownerId == null || ownerId.isBlank()) throw new ProtocolException("ACL response has no owner ID");
List<AclPolicy.Grant> grants = new ArrayList<>();
var nodes = Xml.descendants(root, "Grant");
for (int i = 0; i < nodes.getLength(); i++) {
Element grant = (Element) nodes.item(i);
Element grantee = Xml.child(grant, "Grantee");
if (grantee == null) throw new ProtocolException("ACL grant has no grantee");
String type = grantee.getAttributeNS("http://www.w3.org/2001/XMLSchema-instance", "type");
if (type.isEmpty()) type = grantee.getAttribute("xsi:type");
AclPolicy.GranteeType kind = switch (type) {
case "CanonicalUser" -> AclPolicy.GranteeType.CANONICAL_USER;
case "Group" -> AclPolicy.GranteeType.GROUP;
default -> throw new ProtocolException("Unsupported ACL grantee type");
};
String id = Xml.text(grantee, kind == AclPolicy.GranteeType.GROUP ? "URI" : "ID");
String permission = Xml.text(grant, "Permission");
if (id == null || permission == null) throw new ProtocolException("Incomplete ACL grant");
try { grants.add(new AclPolicy.Grant(kind, id, AclPolicy.Permission.valueOf(permission))); }
catch (IllegalArgumentException e) { throw new ProtocolException("Unsupported ACL permission", e); }
}
return new AclPolicy(ownerId, grants);
}
private void writeAcl(String path, AclPolicy policy) throws IOException {
Objects.requireNonNull(policy, "policy");
StringBuilder xml = new StringBuilder("<AccessControlPolicy xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\" ")
.append("xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"><Owner><ID>")
.append(Xml.escape(policy.ownerId())).append("</ID></Owner><AccessControlList>");
for (AclPolicy.Grant grant : policy.grants()) {
boolean group = grant.type() == AclPolicy.GranteeType.GROUP;
xml.append("<Grant><Grantee xsi:type=\"").append(group ? "Group" : "CanonicalUser")
.append("\"><").append(group ? "URI" : "ID").append('>')
.append(Xml.escape(grant.grantee())).append("</").append(group ? "URI" : "ID")
.append("></Grantee><Permission>").append(grant.permission()).append("</Permission></Grant>");
}
xml.append("</AccessControlList></AccessControlPolicy>");
byte[] bytes = xml.toString().getBytes(StandardCharsets.UTF_8);
String md5;
try { md5 = Base64.getEncoder().encodeToString(MessageDigest.getInstance("MD5").digest(bytes)); }
catch (NoSuchAlgorithmException e) { throw new IllegalStateException(e); }
HttpResponse<InputStream> response = send("PUT", path, Map.of("acl", ""),
Map.of("content-type", "application/xml", "content-md5", md5),
HttpRequest.BodyPublishers.ofByteArray(bytes), SigV4.hash(bytes));
try (InputStream ignored = response.body()) {
if (!successful(response)) throw failure(response);
}
}
/**
* Reads ObjectStore's signed manifest when present. Other S3 endpoints report only operations
* observed succeeding on this client. A denied or missing manifest does not imply a feature is
* unsupported. Service identity from /health is self-reported, not cryptographic attestation.
*/
public Capabilities getCapabilities() throws IOException {
HttpResponse<InputStream> response = send("GET", "/_objectstore/capabilities", Map.of(), Map.of(),
HttpRequest.BodyPublishers.noBody(), EMPTY_HASH);
ProtocolException invalidManifest = null;
try (InputStream body = response.body()) {
if (successful(response)) {
try { return withObserved(parseManifest(readLimited(body, 64 * 1024))); }
catch (ProtocolException e) { invalidManifest = e; }
}
}
boolean objectStore = probeHealth();
if (invalidManifest != null && objectStore) throw invalidManifest;
return withObserved(new Capabilities(objectStore ? Capabilities.ServiceKind.OBJECTSTORE :
Capabilities.ServiceKind.UNKNOWN_S3, Map.of()));
}
/**
* Opts into read-only probes on a known bucket and, optionally, an existing object key.
* Successful calls establish support for this principal. S3 error responses leave support
* UNKNOWN; transport and malformed-response errors are still reported to the caller.
*/
public Capabilities probeReadOnlyCapabilities(String bucket, String existingObjectKey) throws IOException {
Capabilities base = getCapabilities();
probe(base, "ListObjectsV2", () -> listObjects(bucket, null, null, 1));
probe(base, "ListMultipartUploads", () -> listMultipartUploads(bucket, null, null, null, 1));
probe(base, "GetBucketAcl", () -> getBucketAcl(bucket));
if (existingObjectKey != null) {
probe(base, "HeadObject", () -> headObject(bucket, existingObjectKey));
probe(base, "GetObjectAcl", () -> getObjectAcl(bucket, existingObjectKey));
}
return withObserved(base);
}
private void probe(Capabilities base, String operation, Probe call) throws IOException {
if (base.support(operation) == Capabilities.Support.UNSUPPORTED) return;
try { call.run(); }
catch (ObjectStorageException ignored) { }
}
@FunctionalInterface private interface Probe { void run() throws IOException; }
private Capabilities withObserved(Capabilities base) {
Map<String, Capabilities.Support> operations = new HashMap<>(base.operations());
observedOperations.forEach(name -> operations.put(name, Capabilities.Support.SUPPORTED));
return new Capabilities(base.service(), operations, base.limits(), base.serviceVersion(),
base.storageMode(), base.completeOperationInventory());
}
private static Capabilities parseManifest(byte[] bytes) throws ProtocolException {
Map<String, Object> document = Json.object(Json.parse(bytes), "capability manifest");
if (Json.integer(document.get("schemaVersion"), "schemaVersion") != 1 ||
!"lunarsky-objectstore".equals(Json.string(document.get("service"), "service")))
throw new ProtocolException("Unsupported capability manifest");
String version = Json.string(document.get("serviceVersion"), "serviceVersion");
String mode = Json.string(document.get("storageMode"), "storageMode");
Map<String, Capabilities.Support> operations = new HashMap<>();
for (Object value : Json.array(document.get("operations"), "operations")) {
String name = Json.string(value, "operation name");
if (operations.put(name, Capabilities.Support.SUPPORTED) != null)
throw new ProtocolException("Duplicate capability operation");
}
Map<String, Object> rawLimits = Json.object(document.get("limits"), "limits");
Map<String, Long> limits = new HashMap<>();
for (var entry : rawLimits.entrySet()) {
long value = Json.integer(entry.getValue(), "limit " + entry.getKey());
if (value < 0) throw new ProtocolException("Negative capability limit");
limits.put(entry.getKey(), value);
}
for (String required : List.of("maxObjectBytes", "maxTotalBytes", "maxParts"))
if (!limits.containsKey(required)) throw new ProtocolException("Missing capability limit " + required);
return new Capabilities(Capabilities.ServiceKind.OBJECTSTORE, operations, limits, version, mode, true);
}
private boolean probeHealth() throws IOException {
URI uri = URI.create(origin() + "/health");
HttpRequest request = HttpRequest.newBuilder(uri).timeout(timeout).GET().build();
HttpResponse<InputStream> response = execute(request);
try (InputStream body = response.body()) {
if (response.statusCode() != 200) return false;
String value = new String(readLimited(body, 1024), StandardCharsets.UTF_8);
return value.matches("(?s)\\s*\\{\\s*\"status\"\\s*:\\s*\"ok\"\\s*,\\s*\"service\"\\s*:\\s*\"lunarsky-objectstore\"\\s*}\\s*");
}
}
private Element xml(String method, String path, Map<String, String> query) throws IOException {
HttpResponse<InputStream> response = send(method, path, query, Map.of(),
HttpRequest.BodyPublishers.noBody(), EMPTY_HASH);
return responseXml(response, null);
}
private static Element responseXml(HttpResponse<InputStream> response, String expectedRoot) throws IOException {
try (InputStream body = response.body()) {
if (!successful(response)) throw failure(response);
Element root = Xml.parse(readLimited(body, MAX_XML)).getDocumentElement();
if ("Error".equals(root.getLocalName())) {
String code = Xml.text(root, "Code");
throw new ObjectStorageException(response.statusCode(), code,
response.headers().firstValue("x-amz-request-id").orElse(null));
}
if (expectedRoot != null) requireRoot(root, expectedRoot);
return root;
}
}
private HttpResponse<InputStream> send(String method, String path, Map<String, String> query,
Map<String, String> extra, HttpRequest.BodyPublisher body,
String payloadHash) throws IOException {
String queryString = SigV4.query(query);
URI uri = URI.create(origin() + path + (queryString.isEmpty() ? "" : "?" + queryString));
Instant now = clock.instant();
Map<String, String> headers = new HashMap<>(extra);
headers.put("host", uri.getRawAuthority().toLowerCase(Locale.ROOT));
headers.put("x-amz-date", SigV4.timestamp(now));
headers.put("x-amz-content-sha256", payloadHash);
String authorization = SigV4.authorization(method, uri, headers, payloadHash, now,
region, accessKey, secretKey);
HttpRequest.Builder request = HttpRequest.newBuilder(uri).timeout(timeout);
headers.forEach((name, value) -> {
if (!"host".equals(name)) request.header(name, value);
});
request.header("Authorization", authorization);
return execute(request.method(method, body).build());
}
private HttpResponse<InputStream> execute(HttpRequest request) throws IOException {
try { return http.send(request, HttpResponse.BodyHandlers.ofInputStream()); }
catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new IOException("Storage request interrupted", e);
} catch (IOException e) { throw new TransportException(e); }
}
private static boolean successful(HttpResponse<?> response) {
return response.statusCode() >= 200 && response.statusCode() < 300;
}
private static ObjectStorageException failure(HttpResponse<InputStream> response) throws IOException {
String code = null;
String requestId = response.headers().firstValue("x-amz-request-id").orElse(null);
try {
byte[] bytes = readLimited(response.body(), MAX_ERROR);
if (bytes.length > 0) {
Element root = Xml.parse(bytes).getDocumentElement();
if ("Error".equals(root.getLocalName())) {
code = Xml.text(root, "Code");
if (requestId == null) requestId = Xml.text(root, "RequestId");
}
}
} catch (IOException ignored) {
}
return new ObjectStorageException(response.statusCode(), code, requestId);
}
private static byte[] readLimited(InputStream stream, int limit) throws IOException {
byte[] bytes = stream.readNBytes(limit + 1);
if (bytes.length > limit) throw new ProtocolException("Storage response exceeds size limit");
return bytes;
}
private static long length(HttpResponse<?> response) {
return response.headers().firstValueAsLong("content-length").orElse(-1);
}
private static void requireRoot(Element root, String name) throws IOException {
if (!name.equals(root.getLocalName())) throw new ProtocolException("Unexpected storage XML response");
}
private String origin() {
return endpoint.getScheme() + "://" + endpoint.getRawAuthority();
}
private static String bucketPath(String bucket) {
validateBucket(bucket);
return "/" + SigV4.encode(bucket, false);
}
private static String objectPath(String bucket, String key) {
if (key == null || key.isEmpty() || key.indexOf('\0') >= 0 ||
key.getBytes(StandardCharsets.UTF_8).length > 1024)
throw new IllegalArgumentException("object key must be 1..1024 UTF-8 bytes without NUL");
return bucketPath(bucket) + "/" + SigV4.encode(key, true);
}
private static void validateBucket(String bucket) {
if (bucket == null || !bucket.matches("[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]"))
throw new IllegalArgumentException("bucket must be a 3..63 character DNS-style name");
}
/** Closes the underlying HTTP client. Do not use this instance afterward. */
@Override public void close() { http.close(); }
/** An open response stream and selected object headers. Close this result after reading. */
public record ObjectData(InputStream body, long length, String contentType, String etag)
implements AutoCloseable {
@Override public void close() throws IOException { body.close(); }
}
/** Headers returned by a HEAD request. A negative length means it was not supplied. */
public record ObjectMetadata(long length, String contentType, String etag) { }
/** A listed object. */
public record ObjectEntry(String key, long size, String etag) { }
/** One listing page. A null next token means no next page was reported. */
public record ObjectPage(List<ObjectEntry> objects, String nextContinuationToken) {
public ObjectPage { objects = List.copyOf(objects); }
}
/** An unfinished multipart upload. Persist all three fields if resuming in a later process. */
public record MultipartUpload(String bucket, String key, String uploadId) {
public MultipartUpload {
validateBucket(bucket);
if (key == null || key.isEmpty()) throw new IllegalArgumentException("key is required");
if (uploadId == null || uploadId.isBlank()) throw new IllegalArgumentException("upload ID is required");
}
}
/** A successfully uploaded part, including the service-provided ETag. */
public record Part(int number, String etag, long size) {
public Part {
checkPartNumber(number);
if (etag == null || etag.isBlank()) throw new IllegalArgumentException("ETag is required");
if (size < 0) throw new IllegalArgumentException("part size must not be negative");
}
}
/** A page of uploaded parts. Pass nextPartNumberMarker to the next call when truncated. */
public record PartPage(List<Part> parts, boolean truncated, int nextPartNumberMarker) {
public PartPage { parts = List.copyOf(parts); }
}
/** A page of unfinished uploads. Pass both next markers to the next call when truncated. */
public record MultipartUploadPage(List<MultipartUpload> uploads, boolean truncated,
String nextKeyMarker, String nextUploadIdMarker) {
public MultipartUploadPage { uploads = List.copyOf(uploads); }
}
}
@@ -0,0 +1,70 @@
package cloud.lunarsky.objectstore.client;
import java.net.URI;
import java.time.Clock;
import java.time.Duration;
import java.util.Objects;
/** Configures a path-style, Signature V4 S3 client without third-party dependencies. */
public final class ObjectStorageClientBuilder {
private URI endpoint;
private String region = "us-east-1";
private String accessKey;
private String secretKey;
private Duration timeout = Duration.ofSeconds(30);
private boolean allowInsecureHttp;
/** Sets the service root, for example {@code https://storage.example.com}. */
public ObjectStorageClientBuilder endpoint(URI value) {
endpoint = Objects.requireNonNull(value, "endpoint");
return this;
}
/** Sets the Signature V4 region. The default is {@code us-east-1}. */
public ObjectStorageClientBuilder region(String value) {
region = requireText(value, "region");
return this;
}
/** Sets credentials used only in request signatures. Keep them out of logs. */
public ObjectStorageClientBuilder credentials(String access, String secret) {
accessKey = requireText(access, "access key");
secretKey = requireText(secret, "secret key");
return this;
}
/** Sets the per-request timeout. */
public ObjectStorageClientBuilder timeout(Duration value) {
if (Objects.requireNonNull(value, "timeout").isNegative() || value.isZero())
throw new IllegalArgumentException("timeout must be positive");
timeout = value;
return this;
}
/** Allows plain HTTP for a trusted local test endpoint. HTTPS is required by default. */
public ObjectStorageClientBuilder allowInsecureHttp() {
allowInsecureHttp = true;
return this;
}
/** Builds an independent client. */
public ObjectStorageClient build() {
if (endpoint == null) throw new IllegalStateException("endpoint is required");
if (accessKey == null || secretKey == null) throw new IllegalStateException("credentials are required");
if (!region.matches("[a-z0-9-]+")) throw new IllegalArgumentException("invalid region");
if (!accessKey.matches("[A-Za-z0-9_+=./@-]+")) throw new IllegalArgumentException("invalid access key");
String scheme = endpoint.getScheme();
if (!"https".equalsIgnoreCase(scheme) && !(allowInsecureHttp && "http".equalsIgnoreCase(scheme)))
throw new IllegalArgumentException("HTTPS endpoint required unless insecure HTTP is explicitly allowed");
if (endpoint.getHost() == null || endpoint.getUserInfo() != null || endpoint.getRawQuery() != null ||
endpoint.getRawFragment() != null || !(endpoint.getRawPath() == null || endpoint.getRawPath().isEmpty() ||
"/".equals(endpoint.getRawPath())))
throw new IllegalArgumentException("endpoint must be an origin without path, query, fragment, or user info");
return new ObjectStorageClient(endpoint, region, accessKey, secretKey, timeout, Clock.systemUTC());
}
private static String requireText(String value, String name) {
if (value == null || value.isBlank()) throw new IllegalArgumentException(name + " is required");
return value;
}
}
@@ -0,0 +1,30 @@
package cloud.lunarsky.objectstore.client;
import java.io.IOException;
/** An S3 error response with stable fields for callers to inspect. */
public final class ObjectStorageException extends IOException {
private final int statusCode;
private final String errorCode;
private final String requestId;
ObjectStorageException(int statusCode, String errorCode, String requestId) {
super("Storage request failed: HTTP " + statusCode + (errorCode == null ? "" : " (" + errorCode + ")"));
this.statusCode = statusCode;
this.errorCode = errorCode;
this.requestId = requestId;
}
/** Returns the HTTP status. */
public int statusCode() { return statusCode; }
/** Returns the S3 error code, or {@code null} if the server supplied none. */
public String errorCode() { return errorCode; }
/** Returns the request ID, or {@code null}. */
public String requestId() { return requestId; }
/** Returns whether retry might help. A failed write may already have reached the server. */
public boolean retryable() { return statusCode == 429 || statusCode == 500 || statusCode == 502 ||
statusCode == 503 || statusCode == 504; }
}
@@ -0,0 +1,9 @@
package cloud.lunarsky.objectstore.client;
import java.io.IOException;
/** A successful HTTP response that does not match the expected storage protocol. */
public final class ProtocolException extends IOException {
ProtocolException(String message) { super(message); }
ProtocolException(String message, Throwable cause) { super(message, cause); }
}
@@ -0,0 +1,83 @@
package cloud.lunarsky.objectstore.client;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.time.Instant;
import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.HexFormat;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.TreeMap;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
final class SigV4 {
private static final DateTimeFormatter DATE = DateTimeFormatter.ofPattern("yyyyMMdd", Locale.ROOT)
.withZone(ZoneOffset.UTC);
private static final DateTimeFormatter TIME = DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'", Locale.ROOT)
.withZone(ZoneOffset.UTC);
private SigV4() { }
static String hash(byte[] bytes) {
try { return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(bytes)); }
catch (NoSuchAlgorithmException e) { throw new IllegalStateException(e); }
}
static String encode(String text, boolean keepSlash) {
StringBuilder result = new StringBuilder();
for (byte value : text.getBytes(StandardCharsets.UTF_8)) {
int b = value & 0xff;
if (b >= 'A' && b <= 'Z' || b >= 'a' && b <= 'z' || b >= '0' && b <= '9' ||
b == '-' || b == '_' || b == '.' || b == '~' || keepSlash && b == '/') result.append((char) b);
else result.append('%').append(Character.toUpperCase(Character.forDigit(b >>> 4, 16)))
.append(Character.toUpperCase(Character.forDigit(b & 15, 16)));
}
return result.toString();
}
static String query(Map<String, String> values) {
List<String> entries = new ArrayList<>();
values.forEach((key, value) -> entries.add(encode(key, false) + "=" + encode(value, false)));
entries.sort(String::compareTo);
return String.join("&", entries);
}
static String authorization(String method, URI uri, Map<String, String> headers, String payloadHash,
Instant now, String region, String accessKey, String secretKey) {
TreeMap<String, String> signed = new TreeMap<>();
headers.forEach((name, value) -> signed.put(name.toLowerCase(Locale.ROOT),
value.trim().replaceAll("\\s+", " ")));
String names = String.join(";", signed.keySet());
StringBuilder canonicalHeaders = new StringBuilder();
signed.forEach((name, value) -> canonicalHeaders.append(name).append(':').append(value).append('\n'));
String canonical = method + '\n' + uri.getRawPath() + '\n' +
(uri.getRawQuery() == null ? "" : uri.getRawQuery()) + '\n' + canonicalHeaders + '\n' +
names + '\n' + payloadHash;
String day = DATE.format(now);
String scope = day + '/' + region + "/s3/aws4_request";
String toSign = "AWS4-HMAC-SHA256\n" + TIME.format(now) + '\n' + scope + '\n' +
hash(canonical.getBytes(StandardCharsets.UTF_8));
byte[] key = hmac(("AWS4" + secretKey).getBytes(StandardCharsets.UTF_8), day);
key = hmac(key, region);
key = hmac(key, "s3");
key = hmac(key, "aws4_request");
return "AWS4-HMAC-SHA256 Credential=" + accessKey + '/' + scope + ",SignedHeaders=" + names +
",Signature=" + HexFormat.of().formatHex(hmac(key, toSign));
}
private static byte[] hmac(byte[] key, String value) {
try {
Mac mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(key, "HmacSHA256"));
return mac.doFinal(value.getBytes(StandardCharsets.UTF_8));
} catch (Exception e) { throw new IllegalStateException("HMAC-SHA256 unavailable", e); }
}
static String timestamp(Instant now) { return TIME.format(now); }
}
@@ -0,0 +1,8 @@
package cloud.lunarsky.objectstore.client;
import java.io.IOException;
/** A connection or I/O failure. A write may have completed before this was observed. */
public final class TransportException extends IOException {
TransportException(IOException cause) { super("Storage transport failed", cause); }
}
@@ -0,0 +1,54 @@
package cloud.lunarsky.objectstore.client;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import javax.xml.XMLConstants;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.SAXException;
final class Xml {
private Xml() { }
static Document parse(byte[] bytes) throws IOException {
try {
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
factory.setNamespaceAware(true);
factory.setFeature("http://apache.org/xml/features/disallow-doctype-decl", true);
factory.setFeature("http://xml.org/sax/features/external-general-entities", false);
factory.setFeature("http://xml.org/sax/features/external-parameter-entities", false);
factory.setFeature(XMLConstants.FEATURE_SECURE_PROCESSING, true);
factory.setXIncludeAware(false);
factory.setExpandEntityReferences(false);
factory.setAttribute(XMLConstants.ACCESS_EXTERNAL_DTD, "");
factory.setAttribute(XMLConstants.ACCESS_EXTERNAL_SCHEMA, "");
return factory.newDocumentBuilder().parse(new ByteArrayInputStream(bytes));
} catch (ParserConfigurationException | SAXException | IllegalArgumentException e) {
throw new ProtocolException("Malformed or unsafe XML response", e);
}
}
static Element child(Element element, String name) {
for (Node node = element.getFirstChild(); node != null; node = node.getNextSibling())
if (node instanceof Element found && name.equals(found.getLocalName())) return found;
return null;
}
static String text(Element element, String name) {
Element found = child(element, name);
return found == null ? null : found.getTextContent();
}
static NodeList descendants(Element element, String name) {
return element.getElementsByTagNameNS("*", name);
}
static String escape(String text) {
return text.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;")
.replace("\"", "&quot;").replace("'", "&apos;");
}
}
@@ -0,0 +1,190 @@
package cloud.lunarsky.objectstore.client;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.time.Instant;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
public final class ClientTest {
private static final String S3_NS = "http://s3.amazonaws.com/doc/2006-03-01/";
public static void main(String[] args) throws Exception {
signingVector();
protocol();
System.out.println("Client tests passed");
}
private static void signingVector() {
URI uri = URI.create("https://examplebucket.s3.amazonaws.com/test.txt");
String empty = SigV4.hash(new byte[0]);
String actual = SigV4.authorization("GET", uri, Map.of(
"host", "examplebucket.s3.amazonaws.com", "range", "bytes=0-9",
"x-amz-content-sha256", empty, "x-amz-date", "20130524T000000Z"), empty,
Instant.parse("2013-05-24T00:00:00Z"), "us-east-1", "AKIAIOSFODNN7EXAMPLE",
"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY");
check(actual.endsWith("Signature=f0e8bdb87c964420e857bd35b5d6ed310bd44f0170aba48dd91039c6036bdb41"),
"AWS Signature V4 vector");
check("a=%20&z=%2F".equals(SigV4.query(Map.of("z", "/", "a", " "))), "query encoding");
}
private static void protocol() throws Exception {
AtomicReference<String> observedHash = new AtomicReference<>();
AtomicReference<String> observedAcl = new AtomicReference<>();
AtomicReference<String> manifest = new AtomicReference<>();
AtomicInteger manifestStatus = new AtomicInteger(200);
AtomicBoolean health = new AtomicBoolean(true);
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/", exchange -> {
try { handle(exchange, observedHash, observedAcl, manifest, manifestStatus, health); }
finally { exchange.close(); }
});
server.start();
try (ObjectStorageClient client = new ObjectStorageClientBuilder()
.endpoint(URI.create("http://127.0.0.1:" + server.getAddress().getPort()))
.allowInsecureHttp().credentials("test-key", "test-secret").build()) {
client.putObject("mybucket", "hello world.txt", "hello".getBytes(StandardCharsets.UTF_8), "text/plain");
check(SigV4.hash("hello".getBytes(StandardCharsets.UTF_8)).equals(observedHash.get()), "payload hash");
try (ObjectStorageClient.ObjectData data = client.getObject("mybucket", "hello world.txt")) {
check("hello".equals(new String(data.body().readAllBytes(), StandardCharsets.UTF_8)), "GET body");
}
check("text/plain".equals(client.headObject("mybucket", "hello world.txt").contentType()),
"HEAD content type");
var page = client.listObjects("mybucket", "hello", null, 10);
check(page.objects().size() == 1 && "hello world.txt".equals(page.objects().getFirst().key()),
"list object key");
check("next-token".equals(page.nextContinuationToken()), "list continuation");
AclPolicy policy = client.getBucketAcl("mybucket");
check(policy.grants().size() == 1 && policy.grants().getFirst().permission() ==
AclPolicy.Permission.FULL_CONTROL, "GET ACL");
client.putObjectAcl("mybucket", "hello world.txt", new AclPolicy("owner-id", List.of(
new AclPolicy.Grant(AclPolicy.GranteeType.GROUP,
"http://acs.amazonaws.com/groups/global/AllUsers", AclPolicy.Permission.READ))));
check(observedAcl.get().contains("<Permission>READ</Permission>"), "PUT ACL");
check(client.getCapabilities().service() == Capabilities.ServiceKind.OBJECTSTORE, "service discovery");
check(client.getCapabilities().support("versioning") == Capabilities.Support.UNKNOWN,
"unknown support remains unknown");
manifest.set("{\"schemaVersion\":1,\"service\":\"lunarsky-objectstore\"," +
"\"serviceVersion\":\"0.0.5\",\"storageMode\":\"disk\"," +
"\"operations\":[\"PutObject\",\"ListParts\"]," +
"\"limits\":{\"maxObjectBytes\":1024,\"maxTotalBytes\":4096,\"maxParts\":10000}}");
var capabilities = client.getCapabilities();
check(capabilities.service() == Capabilities.ServiceKind.OBJECTSTORE &&
capabilities.support("ListParts") == Capabilities.Support.SUPPORTED &&
capabilities.support("PutBucketAcl") == Capabilities.Support.UNSUPPORTED &&
capabilities.limits().get("maxParts") == 10000 &&
"0.0.5".equals(capabilities.serviceVersion()), "signed manifest");
manifest.set("{\"schemaVersion\":1,\"schemaVersion\":1}");
try {
client.getCapabilities();
throw new AssertionError("Expected malformed manifest");
} catch (ProtocolException expected) { }
health.set(false);
check(client.getCapabilities().service() == Capabilities.ServiceKind.UNKNOWN_S3,
"generic S3 with unrelated matching path");
manifestStatus.set(503);
check(client.getCapabilities().support("CopyObject") == Capabilities.Support.UNKNOWN,
"ambiguous manifest failure stays unknown");
manifestStatus.set(200);
manifest.set(null);
check(client.getCapabilities().service() == Capabilities.ServiceKind.UNKNOWN_S3, "generic S3 fallback");
check(client.getCapabilities().support("PutObject") == Capabilities.Support.SUPPORTED &&
client.getCapabilities().support("CopyObject") == Capabilities.Support.UNKNOWN,
"generic S3 observed support");
capabilities = client.probeReadOnlyCapabilities("mybucket", "hello world.txt");
check(capabilities.support("ListMultipartUploads") == Capabilities.Support.UNKNOWN &&
capabilities.support("ListObjectsV2") == Capabilities.Support.SUPPORTED,
"denied generic probe remains unknown");
client.deleteObject("mybucket", "hello world.txt");
try {
client.getObject("mybucket", "denied");
throw new AssertionError("Expected AccessDenied");
} catch (ObjectStorageException error) {
check(error.statusCode() == 403 && "AccessDenied".equals(error.errorCode()) &&
"request-123".equals(error.requestId()), "typed S3 error");
}
} finally { server.stop(0); }
}
private static void handle(HttpExchange exchange, AtomicReference<String> hash,
AtomicReference<String> acl, AtomicReference<String> manifest,
AtomicInteger manifestStatus, AtomicBoolean health) throws IOException {
String path = exchange.getRequestURI().getRawPath();
String query = exchange.getRequestURI().getRawQuery();
if ("/health".equals(path)) {
respond(exchange, health.get() ? 200 : 404,
health.get() ? "{\"status\":\"ok\",\"service\":\"lunarsky-objectstore\"}" : "");
return;
}
check(exchange.getRequestHeaders().getFirst("Authorization") != null, "signed request");
if ("/_objectstore/capabilities".equals(path)) {
String value = manifest.get();
respond(exchange, value == null ? 404 : manifestStatus.get(), value == null ? "" : value);
return;
}
if ("/mybucket".equals(path) && query != null && query.contains("uploads=")) {
respond(exchange, 403, "<Error><Code>AccessDenied</Code></Error>");
return;
}
if ("/mybucket".equals(path) && query != null && query.contains("list-type=2")) {
respond(exchange, 200, "<ListBucketResult xmlns=\"" + S3_NS + "\"><Contents><Key>hello%20world.txt</Key>" +
"<Size>5</Size><ETag>\"abc\"</ETag></Contents><NextContinuationToken>next-token" +
"</NextContinuationToken></ListBucketResult>");
return;
}
if (query != null && query.equals("acl=")) {
if ("GET".equals(exchange.getRequestMethod())) {
respond(exchange, 200, "<AccessControlPolicy xmlns=\"" + S3_NS + "\" " +
"xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"><Owner><ID>owner-id</ID></Owner>" +
"<AccessControlList><Grant><Grantee xsi:type=\"CanonicalUser\"><ID>owner-id</ID>" +
"</Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>");
} else {
acl.set(new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8));
check(exchange.getRequestHeaders().getFirst("Content-MD5") != null, "ACL Content-MD5");
respond(exchange, 200, "");
}
return;
}
if ("/mybucket/denied".equals(path)) {
exchange.getResponseHeaders().add("x-amz-request-id", "request-123");
respond(exchange, 403, "<Error><Code>AccessDenied</Code></Error>");
return;
}
check("/mybucket/hello%20world.txt".equals(path), "path-style key encoding");
switch (exchange.getRequestMethod()) {
case "PUT" -> {
byte[] body = exchange.getRequestBody().readAllBytes();
hash.set(exchange.getRequestHeaders().getFirst("x-amz-content-sha256"));
check(SigV4.hash(body).equals(hash.get()), "uploaded body hash");
respond(exchange, 200, "");
}
case "GET" -> respond(exchange, 200, "hello");
case "HEAD" -> {
exchange.getResponseHeaders().add("Content-Type", "text/plain");
exchange.sendResponseHeaders(200, -1);
}
case "DELETE" -> respond(exchange, 204, "");
default -> throw new AssertionError("Unexpected method");
}
}
private static void respond(HttpExchange exchange, int status, String content) throws IOException {
byte[] bytes = content.getBytes(StandardCharsets.UTF_8);
if (status == 204 || bytes.length == 0) exchange.sendResponseHeaders(status, -1);
else {
exchange.sendResponseHeaders(status, bytes.length);
exchange.getResponseBody().write(bytes);
}
}
private static void check(boolean condition, String description) {
if (!condition) throw new AssertionError(description);
}
}
@@ -0,0 +1,119 @@
package cloud.lunarsky.objectstore.client;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
public final class MultipartClientTest {
public static void main(String[] args) throws Exception {
AtomicReference<String> completion = new AtomicReference<>();
AtomicBoolean completionError = new AtomicBoolean();
List<Integer> sizes = new ArrayList<>();
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/", exchange -> {
try { handle(exchange, completion, completionError, sizes); }
finally { exchange.close(); }
});
server.start();
try (ObjectStorageClient client = new ObjectStorageClientBuilder()
.endpoint(URI.create("http://127.0.0.1:" + server.getAddress().getPort()))
.allowInsecureHttp().credentials("test-key", "test-secret").build()) {
var upload = client.createMultipartUpload("mybucket", "large file.bin", "application/octet-stream");
check("u-1".equals(upload.uploadId()), "initiation ID");
var first = client.uploadPart(upload, 1, "first".getBytes(StandardCharsets.UTF_8));
var second = client.uploadPart(upload, 2, "second".getBytes(StandardCharsets.UTF_8));
check(first.number() == 1 && "\"etag-1\"".equals(first.etag()), "part result");
var page = client.listParts(upload, 0, 1);
check(page.truncated() && page.nextPartNumberMarker() == 1 && page.parts().size() == 1,
"first part page");
page = client.listParts(upload, page.nextPartNumberMarker(), 1);
check(!page.truncated() && page.parts().getFirst().number() == 2, "second part page");
var uploads = client.listMultipartUploads("mybucket", "large", null, null, 10);
check(uploads.uploads().size() == 1 && "u-1".equals(uploads.uploads().getFirst().uploadId()),
"unfinished upload listing");
String etag = client.completeMultipartUpload(upload, List.of(second, first));
check("\"final-etag\"".equals(etag), "completion ETag");
check(completion.get().indexOf("<PartNumber>1</PartNumber>") <
completion.get().indexOf("<PartNumber>2</PartNumber>"), "completion part order");
completionError.set(true);
try {
client.completeMultipartUpload(upload, List.of(first));
throw new AssertionError("Expected embedded S3 error");
} catch (ObjectStorageException error) {
check("InvalidPart".equals(error.errorCode()), "HTTP 200 completion error");
}
completionError.set(false);
Path file = Files.createTempFile("objectstore-client-multipart", ".bin");
try {
byte[] content = new byte[5 * 1024 * 1024 + 3];
content[content.length - 1] = 42;
Files.write(file, content);
var fileParts = client.uploadFileParts(upload, file, 5 * 1024 * 1024);
check(fileParts.size() == 2 && fileParts.getLast().size() == 3, "file part slicing");
check(sizes.contains(5 * 1024 * 1024) && sizes.contains(3), "part request sizes");
} finally { Files.deleteIfExists(file); }
client.abortMultipartUpload(upload);
System.out.println("Multipart client tests passed");
} finally { server.stop(0); }
}
private static void handle(HttpExchange exchange, AtomicReference<String> completion,
AtomicBoolean completionError, List<Integer> sizes) throws IOException {
check(exchange.getRequestHeaders().getFirst("Authorization") != null, "signed multipart request");
String path = exchange.getRequestURI().getRawPath();
String query = exchange.getRequestURI().getRawQuery();
String method = exchange.getRequestMethod();
if ("/mybucket".equals(path) && "GET".equals(method) && query.contains("uploads=")) {
reply(exchange, 200, "<ListMultipartUploadsResult><Upload><Key>large file.bin</Key>" +
"<UploadId>u-1</UploadId></Upload><IsTruncated>false</IsTruncated></ListMultipartUploadsResult>");
return;
}
check("/mybucket/large%20file.bin".equals(path), "multipart path");
if ("POST".equals(method) && "uploads=".equals(query)) {
reply(exchange, 200, "<InitiateMultipartUploadResult><UploadId>u-1" +
"</UploadId></InitiateMultipartUploadResult>");
} else if ("PUT".equals(method) && query.contains("partNumber=")) {
int number = query.contains("partNumber=1") ? 1 : 2;
byte[] body = exchange.getRequestBody().readAllBytes();
check(SigV4.hash(body).equals(exchange.getRequestHeaders().getFirst("x-amz-content-sha256")),
"part hash");
sizes.add(body.length);
exchange.getResponseHeaders().set("ETag", "\"etag-" + number + "\"");
reply(exchange, 200, "");
} else if ("GET".equals(method) && query.contains("uploadId=")) {
boolean first = query.contains("part-number-marker=0");
reply(exchange, 200, "<ListPartsResult><Part><PartNumber>" + (first ? 1 : 2) +
"</PartNumber><ETag>\"etag-" + (first ? 1 : 2) + "\"</ETag><Size>" +
(first ? 5 : 6) + "</Size></Part><IsTruncated>" + first + "</IsTruncated>" +
"<NextPartNumberMarker>" + (first ? 1 : 2) + "</NextPartNumberMarker></ListPartsResult>");
} else if ("POST".equals(method) && query.contains("uploadId=")) {
completion.set(new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8));
reply(exchange, 200, completionError.get() ? "<Error><Code>InvalidPart</Code></Error>" :
"<CompleteMultipartUploadResult><ETag>\"final-etag\"</ETag></CompleteMultipartUploadResult>");
} else if ("DELETE".equals(method) && query.contains("uploadId=")) {
reply(exchange, 204, "");
} else throw new AssertionError("Unexpected multipart operation: " + method + " " + query);
}
private static void reply(HttpExchange exchange, int status, String content) throws IOException {
byte[] bytes = content.getBytes(StandardCharsets.UTF_8);
if (bytes.length == 0) exchange.sendResponseHeaders(status, -1);
else {
exchange.sendResponseHeaders(status, bytes.length);
exchange.getResponseBody().write(bytes);
}
}
private static void check(boolean condition, String description) {
if (!condition) throw new AssertionError(description);
}
}