Release ObjectStore 0.0.8
This commit is contained in:
1 parent
ed7712a8af
commit
d6427fbac9
61 files changed
+7082
-267
No files matched your search
@@ -0,0 +1 @@
|
||||
dist/
|
||||
@@ -0,0 +1,58 @@
|
||||
# ObjectStore Java client
|
||||
|
||||
A small, synchronous S3-compatible client built with the JDK alone. It uses path-style URLs and AWS Signature Version 4. The code is covered by the repository's MIT license.
|
||||
|
||||
## Build and test
|
||||
|
||||
Requires JDK 21 or newer. No Maven, Gradle, or third-party Java libraries are needed.
|
||||
|
||||
```sh
|
||||
bash client/scripts/test.sh
|
||||
bash client/scripts/build.sh
|
||||
```
|
||||
|
||||
Run these commands from the ObjectStore repository root. The second command writes `client/dist/objectstore-client.jar` and `client/dist/javadoc/`.
|
||||
|
||||
The [AWT image manager](examples/README.md) is a small drag-and-drop app for trying uploads, listing, previews, downloads, and deletes against an existing bucket.
|
||||
|
||||
## Use
|
||||
|
||||
```java
|
||||
import cloud.lunarsky.objectstore.client.ObjectStorageClient;
|
||||
import cloud.lunarsky.objectstore.client.ObjectStorageClientBuilder;
|
||||
import java.net.URI;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
|
||||
try (ObjectStorageClient storage = new ObjectStorageClientBuilder()
|
||||
.endpoint(URI.create("https://storage.example.com"))
|
||||
.region("us-east-1")
|
||||
.credentials(accessKey, secretKey)
|
||||
.build()) {
|
||||
storage.putObject("photos", "hello.txt", "hello".getBytes(StandardCharsets.UTF_8), "text/plain");
|
||||
try (ObjectStorageClient.ObjectData data = storage.getObject("photos", "hello.txt")) {
|
||||
data.body().transferTo(System.out);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`getObject` returns an open stream. Close it even if you do not read every byte. The client requires HTTPS unless you explicitly call `allowInsecureHttp()` for a trusted local endpoint.
|
||||
|
||||
## Scope
|
||||
|
||||
This version implements single-request object PUT, GET, HEAD, DELETE, ListObjectsV2 pages, standard S3 multipart initiation/part upload/list/complete/abort, unfinished-upload listing, bucket/object ACL GET and PUT, and service detection. ACLs are S3 ACL requests, not a calculation of effective permissions from IAM policies or bucket policies. An S3 service with ACLs disabled can reject them; ObjectStore implements a limited ACL subset.
|
||||
|
||||
`getCapabilities()` reads ObjectStore's signed `GET /_objectstore/capabilities` response when available. It reports implemented S3 operation names, service version, storage mode, and configured limits. A listed operation means the service implements it; it does not establish that the current key may perform it or that every AWS option is supported. Older ObjectStore builds fall back to `/health` and leave unverified operations `UNKNOWN`. The response is not cached.
|
||||
|
||||
For a generic S3 endpoint, successful operations on this client are recorded as `SUPPORTED`; all untested operations remain `UNKNOWN`. Call `probeReadOnlyCapabilities(bucket, existingObjectKey)` to opt into safe read probes. An S3 denial or missing object leaves that operation `UNKNOWN`, not `UNSUPPORTED`. The client never infers features from a hostname or vendor header. The older `/health` identity fallback is self-reported, not cryptographic attestation.
|
||||
|
||||
Multipart upload sessions expose their bucket, key, and upload ID so a caller can persist them and inspect already uploaded parts after a restart. `uploadFileParts` sends a file in sequential parts and leaves completion to the caller. If it fails, the upload remains open for retry or explicit abort; keep the returned upload ID and do not change the source file.
|
||||
|
||||
The client does not yet implement presigned URLs, credential providers, automatic retries, region redirects, or streaming uploads of unknown length. A single-request file upload hashes the file before sending it; do not modify the file while the upload runs. The client does not retry writes because a lost response can leave their outcome uncertain.
|
||||
|
||||
## Errors
|
||||
|
||||
- `ObjectStorageException`: an HTTP error, with status, S3 error code, request ID, and a retryability hint.
|
||||
- `TransportException`: connection or I/O failure. A write may already have completed.
|
||||
- `ProtocolException`: an unexpected or malformed successful response.
|
||||
|
||||
The client does not include access keys, secret keys, or response bodies in exception messages.
|
||||
@@ -0,0 +1,37 @@
|
||||
# Examples
|
||||
|
||||
## AWT image manager
|
||||
|
||||

|
||||
|
||||
Cat photo in the screenshot: [IOP Publishing source image](https://ioppublishing.org/wp-content/uploads/2017/03/cat-web-cc0.jpg).
|
||||
|
||||
Run from the ObjectStore repository root with JDK 21 or newer:
|
||||
|
||||
```sh
|
||||
bash client/examples/awt-images/run.sh
|
||||
```
|
||||
|
||||
The example uses only the Java client and the JDK. Enter an S3-compatible endpoint, region, existing bucket, and access keys. Use **Connect** to verify listing access. Drag PNG, JPEG, GIF, or BMP files onto the window, or use **Add images**. Select an object to preview it; use **Download** or **Delete** to manage it. Objects go under the specified key prefix with a short random ID, so uploading the same filename does not silently replace an earlier image. The list loads 100 objects at a time.
|
||||
|
||||
The example does not create a bucket. When launched directly, it keeps credentials in memory and requires explicit opt-in for plain HTTP. Use HTTP only with a trusted test service. Uploads are limited to 64 MiB per file, and previews to 12 MiB. This is a small interoperability test app, not a production asset manager.
|
||||
|
||||
### Local Docker test
|
||||
|
||||
Prerequisites: JDK 21 or newer, Bash, Python 3, `curl`, and a graphical desktop session. Docker must be running and accessible to your user, and `127.0.0.1:9002` must be free for the test container.
|
||||
|
||||
To try the app with a separate ObjectStore service, run:
|
||||
|
||||
```sh
|
||||
bash client/examples/awt-images/run-test.sh
|
||||
```
|
||||
|
||||
The launcher builds its local image if needed, starts a test container on `127.0.0.1:9002`, and opens the app connected to a `photos` bucket. Set `OBJECTSTORE_TEST_IMAGE` to use a different image. Closing the app leaves the container and its dedicated data volume running; run the launcher again to reconnect. Its generated test credentials are stored in the container's Docker configuration.
|
||||
|
||||
When finished, remove only this example's container and volume:
|
||||
|
||||
```sh
|
||||
docker stop objectstore-image-example
|
||||
docker rm objectstore-image-example
|
||||
docker volume rm objectstore-image-example-data
|
||||
```
|
||||
@@ -0,0 +1,503 @@
|
||||
import cloud.lunarsky.objectstore.client.Capabilities;
|
||||
import cloud.lunarsky.objectstore.client.ObjectStorageClient;
|
||||
import cloud.lunarsky.objectstore.client.ObjectStorageClientBuilder;
|
||||
|
||||
import java.awt.BorderLayout;
|
||||
import java.awt.Button;
|
||||
import java.awt.Canvas;
|
||||
import java.awt.Checkbox;
|
||||
import java.awt.Color;
|
||||
import java.awt.Dialog;
|
||||
import java.awt.Dimension;
|
||||
import java.awt.EventQueue;
|
||||
import java.awt.FileDialog;
|
||||
import java.awt.FlowLayout;
|
||||
import java.awt.Font;
|
||||
import java.awt.Frame;
|
||||
import java.awt.Graphics;
|
||||
import java.awt.GridLayout;
|
||||
import java.awt.Label;
|
||||
import java.awt.Panel;
|
||||
import java.awt.TextField;
|
||||
import java.awt.dnd.DnDConstants;
|
||||
import java.awt.dnd.DropTarget;
|
||||
import java.awt.dnd.DropTargetAdapter;
|
||||
import java.awt.dnd.DropTargetDropEvent;
|
||||
import java.awt.datatransfer.DataFlavor;
|
||||
import java.awt.event.WindowAdapter;
|
||||
import java.awt.event.WindowEvent;
|
||||
import java.awt.image.BufferedImage;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.IOException;
|
||||
import java.net.URI;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.StandardCopyOption;
|
||||
import java.time.Duration;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.atomic.AtomicLong;
|
||||
import javax.imageio.ImageIO;
|
||||
import javax.imageio.ImageReader;
|
||||
import javax.imageio.stream.ImageInputStream;
|
||||
import javax.imageio.stream.MemoryCacheImageInputStream;
|
||||
|
||||
/** A small, dependency-free image browser for testing an S3-compatible endpoint. */
|
||||
@SuppressWarnings("serial")
|
||||
public final class ImageManager extends Frame {
|
||||
private static final long MAX_UPLOAD = 64L * 1024 * 1024;
|
||||
private static final int MAX_PREVIEW = 12 * 1024 * 1024;
|
||||
private static final Color BACKGROUND = new Color(29, 27, 38);
|
||||
private static final Color FOREGROUND = new Color(231, 223, 240);
|
||||
|
||||
private final TextField endpoint = new TextField(env("S3_ENDPOINT", "http://localhost:9000"));
|
||||
private final TextField region = new TextField(env("S3_REGION", "us-east-1"));
|
||||
private final TextField bucket = new TextField(env("S3_BUCKET", "photos"));
|
||||
private final TextField prefix = new TextField("images/");
|
||||
private final TextField access = new TextField(env("S3_ACCESS_KEY", ""));
|
||||
private final TextField secret = new TextField(env("S3_SECRET_KEY", ""));
|
||||
private final Checkbox allowHttp = new Checkbox("Allow HTTP for trusted tests", null,
|
||||
Boolean.parseBoolean(env("S3_ALLOW_HTTP", "false")));
|
||||
private final java.awt.List images = new java.awt.List(15, false);
|
||||
private final ImageCanvas preview = new ImageCanvas();
|
||||
private final Label status = new Label("Enter a bucket and credentials, then connect. Drop images to upload.");
|
||||
private final Button connectButton = new Button("Connect");
|
||||
private final Button moreButton = new Button("Load more");
|
||||
private final ExecutorService io = Executors.newSingleThreadExecutor(r -> {
|
||||
Thread thread = new Thread(r, "objectstore-image-example");
|
||||
thread.setDaemon(true);
|
||||
return thread;
|
||||
});
|
||||
private final AtomicLong previewRevision = new AtomicLong();
|
||||
private final List<ObjectStorageClient.ObjectEntry> entries = new ArrayList<>();
|
||||
private volatile ObjectStorageClient client;
|
||||
private volatile String activeBucket;
|
||||
private volatile String activePrefix;
|
||||
private String nextToken;
|
||||
|
||||
private ImageManager() {
|
||||
super("ObjectStore image manager");
|
||||
setLayout(new BorderLayout(8, 8));
|
||||
setBackground(BACKGROUND);
|
||||
setForeground(FOREGROUND);
|
||||
setSize(920, 640);
|
||||
setMinimumSize(new Dimension(650, 440));
|
||||
setLocationRelativeTo(null);
|
||||
secret.setEchoChar('\u2022');
|
||||
|
||||
Panel serviceFields = new Panel(new GridLayout(2, 4, 8, 3));
|
||||
serviceFields.add(label("Endpoint"));
|
||||
serviceFields.add(label("Region"));
|
||||
serviceFields.add(label("Bucket"));
|
||||
serviceFields.add(label("Key prefix"));
|
||||
serviceFields.add(endpoint);
|
||||
serviceFields.add(region);
|
||||
serviceFields.add(bucket);
|
||||
serviceFields.add(prefix);
|
||||
Panel credentialFields = new Panel(new GridLayout(2, 2, 8, 3));
|
||||
credentialFields.add(label("Access key"));
|
||||
credentialFields.add(label("Secret key"));
|
||||
credentialFields.add(access);
|
||||
credentialFields.add(secret);
|
||||
Panel connection = new Panel(new BorderLayout(0, 5));
|
||||
connection.add(serviceFields, BorderLayout.NORTH);
|
||||
connection.add(credentialFields, BorderLayout.CENTER);
|
||||
|
||||
Panel actions = new Panel(new FlowLayout(FlowLayout.LEFT, 8, 4));
|
||||
Button upload = new Button("Add images...");
|
||||
Button refresh = new Button("Refresh");
|
||||
Button download = new Button("Download");
|
||||
Button delete = new Button("Delete");
|
||||
actions.add(allowHttp);
|
||||
actions.add(connectButton);
|
||||
actions.add(upload);
|
||||
actions.add(refresh);
|
||||
actions.add(moreButton);
|
||||
actions.add(download);
|
||||
actions.add(delete);
|
||||
moreButton.setEnabled(false);
|
||||
|
||||
Panel top = new Panel(new BorderLayout(0, 5));
|
||||
top.add(connection, BorderLayout.CENTER);
|
||||
top.add(actions, BorderLayout.SOUTH);
|
||||
add(top, BorderLayout.NORTH);
|
||||
|
||||
Panel listing = new Panel(new BorderLayout(0, 5));
|
||||
listing.setPreferredSize(new Dimension(285, 400));
|
||||
listing.add(label("Images in this prefix"), BorderLayout.NORTH);
|
||||
listing.add(images, BorderLayout.CENTER);
|
||||
add(listing, BorderLayout.WEST);
|
||||
add(preview, BorderLayout.CENTER);
|
||||
add(status, BorderLayout.SOUTH);
|
||||
|
||||
connectButton.addActionListener(event -> connect());
|
||||
upload.addActionListener(event -> chooseImages());
|
||||
refresh.addActionListener(event -> loadPage(true));
|
||||
moreButton.addActionListener(event -> loadPage(false));
|
||||
download.addActionListener(event -> downloadSelected());
|
||||
delete.addActionListener(event -> deleteSelected());
|
||||
images.addItemListener(event -> previewSelected());
|
||||
installDropTarget(this);
|
||||
installDropTarget(preview);
|
||||
installDropTarget(images);
|
||||
addWindowListener(new WindowAdapter() {
|
||||
@Override public void windowClosing(WindowEvent event) {
|
||||
io.shutdownNow();
|
||||
ObjectStorageClient previous = client;
|
||||
if (previous != null) Thread.ofVirtual().start(previous::close);
|
||||
dispose();
|
||||
}
|
||||
});
|
||||
if (Boolean.parseBoolean(env("S3_AUTO_CONNECT", "false"))) {
|
||||
EventQueue.invokeLater(this::connect);
|
||||
}
|
||||
}
|
||||
|
||||
private void connect() {
|
||||
String url = endpoint.getText().trim();
|
||||
String location = region.getText().trim();
|
||||
String name = bucket.getText().trim();
|
||||
String folder = normalizePrefix(prefix.getText());
|
||||
String user = access.getText().trim();
|
||||
String password = secret.getText();
|
||||
boolean insecure = allowHttp.getState();
|
||||
connectButton.setEnabled(false);
|
||||
setStatus("Connecting...", false);
|
||||
io.execute(() -> {
|
||||
ObjectStorageClient candidate = null;
|
||||
try {
|
||||
ObjectStorageClientBuilder builder = new ObjectStorageClientBuilder()
|
||||
.endpoint(URI.create(url)).region(location).credentials(user, password)
|
||||
.timeout(Duration.ofSeconds(30));
|
||||
if (insecure) builder.allowInsecureHttp();
|
||||
candidate = builder.build();
|
||||
candidate.listObjects(name, folder, null, 1);
|
||||
ObjectStorageClient previous = client;
|
||||
client = candidate;
|
||||
activeBucket = name;
|
||||
activePrefix = folder;
|
||||
candidate = null;
|
||||
if (previous != null) previous.close();
|
||||
String service = "S3-compatible service";
|
||||
try {
|
||||
Capabilities found = client.getCapabilities();
|
||||
if (found.service() == Capabilities.ServiceKind.OBJECTSTORE) {
|
||||
service = "ObjectStore" + (found.serviceVersion() == null ? "" : " " + found.serviceVersion());
|
||||
}
|
||||
} catch (IOException ignored) {
|
||||
// Listing already established the connection; service detection is optional.
|
||||
}
|
||||
String connectedService = service;
|
||||
EventQueue.invokeLater(() -> {
|
||||
connectButton.setEnabled(true);
|
||||
setStatus("Connected to " + connectedService + ".", false);
|
||||
loadPage(true);
|
||||
});
|
||||
} catch (Exception error) {
|
||||
if (candidate != null) candidate.close();
|
||||
showFailure("Connection failed", error);
|
||||
EventQueue.invokeLater(() -> connectButton.setEnabled(true));
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
private void loadPage(boolean first) {
|
||||
if (client == null) {
|
||||
setStatus("Connect first.", true);
|
||||
return;
|
||||
}
|
||||
if (!first && nextToken == null) return;
|
||||
String token = first ? null : nextToken;
|
||||
moreButton.setEnabled(false);
|
||||
setStatus(first ? "Loading images..." : "Loading more images...", false);
|
||||
io.execute(() -> {
|
||||
try {
|
||||
ObjectStorageClient.ObjectPage page = client.listObjects(activeBucket, activePrefix, token, 100);
|
||||
EventQueue.invokeLater(() -> {
|
||||
if (first) {
|
||||
previewRevision.incrementAndGet();
|
||||
entries.clear();
|
||||
images.removeAll();
|
||||
preview.show(null, "Select an image to preview");
|
||||
}
|
||||
for (ObjectStorageClient.ObjectEntry entry : page.objects()) {
|
||||
if (!isImage(entry.key())) continue;
|
||||
entries.add(entry);
|
||||
String name = entry.key().substring(activePrefix.length());
|
||||
images.add(name + " · " + sizeLabel(entry.size()));
|
||||
}
|
||||
nextToken = page.nextContinuationToken();
|
||||
moreButton.setEnabled(nextToken != null);
|
||||
setStatus(entries.size() + " image(s) loaded" +
|
||||
(nextToken == null ? "." : "; more available."), false);
|
||||
});
|
||||
} catch (Exception error) {
|
||||
showFailure("Could not list images", error);
|
||||
EventQueue.invokeLater(() -> moreButton.setEnabled(token != null));
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
private void chooseImages() {
|
||||
FileDialog picker = new FileDialog(this, "Add images", FileDialog.LOAD);
|
||||
picker.setMultipleMode(true);
|
||||
picker.setVisible(true);
|
||||
java.io.File[] selected = picker.getFiles();
|
||||
if (selected.length > 0) uploadImages(List.of(selected));
|
||||
}
|
||||
|
||||
private void uploadImages(List<java.io.File> files) {
|
||||
if (client == null) {
|
||||
setStatus("Connect first.", true);
|
||||
return;
|
||||
}
|
||||
io.execute(() -> {
|
||||
int uploaded = 0;
|
||||
for (java.io.File file : files) {
|
||||
try {
|
||||
Path path = file.toPath();
|
||||
String name = path.getFileName().toString();
|
||||
String type = contentType(name);
|
||||
if (type == null) throw new IOException("Unsupported image type");
|
||||
if (!Files.isRegularFile(path) || Files.size(path) > MAX_UPLOAD)
|
||||
throw new IOException("Image must be a file of at most 64 MiB");
|
||||
String key = activePrefix + UUID.randomUUID() + "-" + name;
|
||||
client.putObject(activeBucket, key, path, type);
|
||||
uploaded++;
|
||||
int completed = uploaded;
|
||||
EventQueue.invokeLater(() -> setStatus("Uploaded " + completed + " of " + files.size() + ".", false));
|
||||
} catch (Exception error) {
|
||||
showFailure("Could not upload " + file.getName(), error);
|
||||
}
|
||||
}
|
||||
if (uploaded > 0) EventQueue.invokeLater(() -> loadPage(true));
|
||||
});
|
||||
}
|
||||
|
||||
private void previewSelected() {
|
||||
ObjectStorageClient.ObjectEntry entry = selectedEntry();
|
||||
long revision = previewRevision.incrementAndGet();
|
||||
if (entry == null) {
|
||||
preview.show(null, "Select an image to preview");
|
||||
return;
|
||||
}
|
||||
preview.show(null, "Loading preview...");
|
||||
io.execute(() -> {
|
||||
try (ObjectStorageClient.ObjectData data = client.getObject(activeBucket, entry.key())) {
|
||||
if (data.length() > MAX_PREVIEW) throw new IOException("Preview exceeds 12 MiB");
|
||||
byte[] bytes = data.body().readNBytes(MAX_PREVIEW + 1);
|
||||
if (bytes.length > MAX_PREVIEW) throw new IOException("Preview exceeds 12 MiB");
|
||||
BufferedImage image = decodePreview(bytes);
|
||||
EventQueue.invokeLater(() -> {
|
||||
if (previewRevision.get() == revision) preview.show(image, null);
|
||||
});
|
||||
} catch (Exception error) {
|
||||
EventQueue.invokeLater(() -> {
|
||||
if (previewRevision.get() == revision) preview.show(null, "Preview unavailable");
|
||||
});
|
||||
showFailure("Could not preview image", error);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
private void downloadSelected() {
|
||||
ObjectStorageClient.ObjectEntry entry = selectedEntry();
|
||||
if (entry == null) {
|
||||
setStatus("Select an image first.", true);
|
||||
return;
|
||||
}
|
||||
FileDialog picker = new FileDialog(this, "Save image", FileDialog.SAVE);
|
||||
picker.setFile(Path.of(entry.key()).getFileName().toString());
|
||||
picker.setVisible(true);
|
||||
if (picker.getFile() == null) return;
|
||||
Path destination = Path.of(picker.getDirectory(), picker.getFile());
|
||||
if (Files.exists(destination) && !confirm("Replace this file?", destination.toString())) return;
|
||||
io.execute(() -> {
|
||||
Path temporary = null;
|
||||
try {
|
||||
Path parent = destination.toAbsolutePath().getParent();
|
||||
temporary = Files.createTempFile(parent, ".objectstore-image-", ".part");
|
||||
try (ObjectStorageClient.ObjectData data = client.getObject(activeBucket, entry.key())) {
|
||||
Files.copy(data.body(), temporary, StandardCopyOption.REPLACE_EXISTING);
|
||||
}
|
||||
Files.move(temporary, destination, StandardCopyOption.REPLACE_EXISTING);
|
||||
EventQueue.invokeLater(() -> setStatus("Saved " + destination.getFileName() + ".", false));
|
||||
} catch (Exception error) {
|
||||
showFailure("Download failed", error);
|
||||
} finally {
|
||||
if (temporary != null) {
|
||||
try { Files.deleteIfExists(temporary); } catch (IOException ignored) { }
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
private void deleteSelected() {
|
||||
ObjectStorageClient.ObjectEntry entry = selectedEntry();
|
||||
if (entry == null) {
|
||||
setStatus("Select an image first.", true);
|
||||
return;
|
||||
}
|
||||
if (!confirm("Delete this image?", entry.key())) return;
|
||||
io.execute(() -> {
|
||||
try {
|
||||
client.deleteObject(activeBucket, entry.key());
|
||||
EventQueue.invokeLater(() -> {
|
||||
previewRevision.incrementAndGet();
|
||||
loadPage(true);
|
||||
});
|
||||
} catch (Exception error) { showFailure("Delete failed", error); }
|
||||
});
|
||||
}
|
||||
|
||||
private ObjectStorageClient.ObjectEntry selectedEntry() {
|
||||
int index = images.getSelectedIndex();
|
||||
return index < 0 || index >= entries.size() ? null : entries.get(index);
|
||||
}
|
||||
|
||||
private void installDropTarget(java.awt.Component target) {
|
||||
new DropTarget(target, DnDConstants.ACTION_COPY, new DropTargetAdapter() {
|
||||
@Override public void drop(DropTargetDropEvent event) {
|
||||
if (!event.isDataFlavorSupported(DataFlavor.javaFileListFlavor)) {
|
||||
event.rejectDrop();
|
||||
return;
|
||||
}
|
||||
try {
|
||||
event.acceptDrop(DnDConstants.ACTION_COPY);
|
||||
Object value = event.getTransferable().getTransferData(DataFlavor.javaFileListFlavor);
|
||||
List<?> dropped = (List<?>) value;
|
||||
List<java.io.File> files = new ArrayList<>();
|
||||
for (Object item : dropped) if (item instanceof java.io.File file) files.add(file);
|
||||
event.dropComplete(true);
|
||||
EventQueue.invokeLater(() -> uploadImages(files));
|
||||
} catch (Exception error) {
|
||||
event.dropComplete(false);
|
||||
showFailure("Drop failed", error);
|
||||
}
|
||||
}
|
||||
}, true);
|
||||
}
|
||||
|
||||
private boolean confirm(String title, String detail) {
|
||||
Dialog dialog = new Dialog(this, title, true);
|
||||
dialog.setLayout(new BorderLayout(12, 12));
|
||||
dialog.add(new Label(detail), BorderLayout.CENTER);
|
||||
Panel buttons = new Panel(new FlowLayout(FlowLayout.RIGHT));
|
||||
boolean[] accepted = { false };
|
||||
Button cancel = new Button("Cancel");
|
||||
Button proceed = new Button("Continue");
|
||||
cancel.addActionListener(event -> dialog.dispose());
|
||||
proceed.addActionListener(event -> { accepted[0] = true; dialog.dispose(); });
|
||||
buttons.add(cancel);
|
||||
buttons.add(proceed);
|
||||
dialog.add(buttons, BorderLayout.SOUTH);
|
||||
dialog.setSize(490, 120);
|
||||
dialog.setLocationRelativeTo(this);
|
||||
dialog.setVisible(true);
|
||||
return accepted[0];
|
||||
}
|
||||
|
||||
private void showFailure(String action, Exception error) {
|
||||
EventQueue.invokeLater(() -> setStatus(action + ": " + error.getMessage(), true));
|
||||
}
|
||||
|
||||
private void setStatus(String message, boolean failed) {
|
||||
status.setForeground(failed ? new Color(245, 143, 157) : FOREGROUND);
|
||||
status.setText(message);
|
||||
}
|
||||
|
||||
private static Label label(String text) {
|
||||
return new Label(text);
|
||||
}
|
||||
|
||||
private static String normalizePrefix(String value) {
|
||||
String cleaned = value.trim().replace('\\', '/');
|
||||
while (cleaned.startsWith("/")) cleaned = cleaned.substring(1);
|
||||
return cleaned.isEmpty() || cleaned.endsWith("/") ? cleaned : cleaned + "/";
|
||||
}
|
||||
|
||||
private static String contentType(String name) {
|
||||
String lower = name.toLowerCase(Locale.ROOT);
|
||||
if (lower.endsWith(".png")) return "image/png";
|
||||
if (lower.endsWith(".jpg") || lower.endsWith(".jpeg")) return "image/jpeg";
|
||||
if (lower.endsWith(".gif")) return "image/gif";
|
||||
if (lower.endsWith(".bmp")) return "image/bmp";
|
||||
return null;
|
||||
}
|
||||
|
||||
private static boolean isImage(String name) {
|
||||
return contentType(name) != null;
|
||||
}
|
||||
|
||||
private static String sizeLabel(long bytes) {
|
||||
return bytes < 1024 ? bytes + " B" : String.format(Locale.ROOT, "%.1f KiB", bytes / 1024.0);
|
||||
}
|
||||
|
||||
private static BufferedImage decodePreview(byte[] bytes) throws IOException {
|
||||
try (ImageInputStream stream = new MemoryCacheImageInputStream(new ByteArrayInputStream(bytes))) {
|
||||
var readers = ImageIO.getImageReaders(stream);
|
||||
if (!readers.hasNext()) throw new IOException("Unsupported image data");
|
||||
ImageReader reader = readers.next();
|
||||
try {
|
||||
reader.setInput(stream, true, true);
|
||||
int width = reader.getWidth(0);
|
||||
int height = reader.getHeight(0);
|
||||
if (width < 1 || height < 1 || width > 16000 || height > 16000 ||
|
||||
(long) width * height > 40_000_000)
|
||||
throw new IOException("Image dimensions are too large for preview");
|
||||
int step = Math.max(1, (Math.max(width, height) + 1199) / 1200);
|
||||
var parameters = reader.getDefaultReadParam();
|
||||
parameters.setSourceSubsampling(step, step, 0, 0);
|
||||
return reader.read(0, parameters);
|
||||
} finally { reader.dispose(); }
|
||||
}
|
||||
}
|
||||
|
||||
private static String env(String name, String fallback) {
|
||||
String value = System.getenv(name);
|
||||
return value == null ? fallback : value;
|
||||
}
|
||||
|
||||
@SuppressWarnings("serial")
|
||||
private static final class ImageCanvas extends Canvas {
|
||||
private BufferedImage image;
|
||||
private String message = "Select an image to preview";
|
||||
|
||||
private ImageCanvas() {
|
||||
setBackground(new Color(21, 20, 29));
|
||||
setForeground(FOREGROUND);
|
||||
setFont(new Font(Font.SANS_SERIF, Font.PLAIN, 16));
|
||||
}
|
||||
|
||||
private void show(BufferedImage value, String text) {
|
||||
image = value;
|
||||
message = text;
|
||||
repaint();
|
||||
}
|
||||
|
||||
@Override public void paint(Graphics graphics) {
|
||||
int width = getWidth();
|
||||
int height = getHeight();
|
||||
if (image == null) {
|
||||
graphics.setColor(FOREGROUND);
|
||||
graphics.drawString(message, 20, Math.max(35, height / 2));
|
||||
return;
|
||||
}
|
||||
double scale = Math.min((width - 24.0) / image.getWidth(),
|
||||
(height - 24.0) / image.getHeight());
|
||||
scale = Math.max(0.01, Math.min(scale, 1.0));
|
||||
int drawWidth = (int) Math.round(image.getWidth() * scale);
|
||||
int drawHeight = (int) Math.round(image.getHeight() * scale);
|
||||
graphics.drawImage(image, (width - drawWidth) / 2, (height - drawHeight) / 2,
|
||||
drawWidth, drawHeight, null);
|
||||
}
|
||||
}
|
||||
|
||||
public static void main(String[] args) {
|
||||
EventQueue.invokeLater(() -> new ImageManager().setVisible(true));
|
||||
}
|
||||
}
|
||||
Executable
+51
@@ -0,0 +1,51 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
client_dir="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
project_dir="$(cd "$client_dir/.." && pwd)"
|
||||
container="objectstore-image-example"
|
||||
volume="objectstore-image-example-data"
|
||||
image="${OBJECTSTORE_TEST_IMAGE:-objectstore-image-example:local}"
|
||||
|
||||
if ! docker container inspect "$container" >/dev/null 2>&1; then
|
||||
if ! docker image inspect "$image" >/dev/null 2>&1; then
|
||||
docker build --tag "$image" "$project_dir"
|
||||
fi
|
||||
export S3_ACCESS_KEY="ImageExampleTest1"
|
||||
export S3_SECRET_KEY="$(python3 -c 'import secrets; print(secrets.token_hex(32))')"
|
||||
docker volume create "$volume" >/dev/null
|
||||
docker run --detach --name "$container" \
|
||||
--publish 127.0.0.1:9002:9000 \
|
||||
--volume "$volume:/data" \
|
||||
--env S3_ACCESS_KEY --env S3_SECRET_KEY \
|
||||
--env S3_BUCKET=photos --env S3_REGION=us-east-1 \
|
||||
--env MAX_OBJECT_BYTES=67108864 --env MAX_TOTAL_BYTES=1073741824 \
|
||||
"$image" >/dev/null
|
||||
else
|
||||
while IFS='=' read -r key value; do
|
||||
case "$key" in
|
||||
S3_ACCESS_KEY) export S3_ACCESS_KEY="$value" ;;
|
||||
S3_SECRET_KEY) export S3_SECRET_KEY="$value" ;;
|
||||
esac
|
||||
done < <(docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' "$container")
|
||||
if [[ "$(docker inspect --format '{{.State.Running}}' "$container")" != true ]]; then
|
||||
docker start "$container" >/dev/null
|
||||
fi
|
||||
fi
|
||||
|
||||
for attempt in {1..40}; do
|
||||
if curl --silent --fail --output /dev/null http://127.0.0.1:9002/health; then
|
||||
break
|
||||
fi
|
||||
sleep 0.25
|
||||
done
|
||||
if ! curl --silent --fail --output /dev/null http://127.0.0.1:9002/health; then
|
||||
echo "The isolated ObjectStore container did not become healthy; check docker logs $container." >&2
|
||||
exit 1
|
||||
fi
|
||||
export S3_ENDPOINT=http://127.0.0.1:9002
|
||||
export S3_REGION=us-east-1
|
||||
export S3_BUCKET=photos
|
||||
export S3_ALLOW_HTTP=true
|
||||
export S3_AUTO_CONNECT=true
|
||||
exec "$client_dir/examples/awt-images/run.sh"
|
||||
Executable
+11
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
client_dir="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
if [[ ! -f "$client_dir/dist/objectstore-client.jar" ]]; then
|
||||
bash "$client_dir/scripts/build.sh"
|
||||
fi
|
||||
classes="$client_dir/dist/examples/awt-images"
|
||||
mkdir -p "$classes"
|
||||
javac --release 21 -cp "$client_dir/dist/objectstore-client.jar" \
|
||||
-d "$classes" "$client_dir/examples/awt-images/ImageManager.java"
|
||||
exec java -cp "$classes:$client_dir/dist/objectstore-client.jar" ImageManager
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 381 KiB |
@@ -0,0 +1,13 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
rm -rf dist/classes
|
||||
mkdir -p dist/classes
|
||||
find src/main/java -name '*.java' -print0 |
|
||||
xargs -0 javac --release 21 -d dist/classes
|
||||
mkdir -p dist/classes/META-INF
|
||||
cp ../LICENSE dist/classes/META-INF/LICENSE
|
||||
jar --create --file dist/objectstore-client.jar -C dist/classes .
|
||||
javadoc --release 21 -quiet -Xdoclint:reference,syntax,html -d dist/javadoc \
|
||||
$(find src/main/java -name '*.java' -print)
|
||||
echo "Built dist/objectstore-client.jar"
|
||||
@@ -0,0 +1,9 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
build_dir="$(mktemp -d)"
|
||||
trap 'rm -rf "$build_dir"' EXIT
|
||||
find src/main/java src/test/java -name '*.java' -print0 |
|
||||
xargs -0 javac --release 21 -d "$build_dir"
|
||||
java -cp "$build_dir" cloud.lunarsky.objectstore.client.ClientTest
|
||||
java -cp "$build_dir" cloud.lunarsky.objectstore.client.MultipartClientTest
|
||||
@@ -0,0 +1,27 @@
|
||||
package cloud.lunarsky.objectstore.client;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Objects;
|
||||
|
||||
/** A bucket or object ACL. This is not an effective-permissions calculation. */
|
||||
public record AclPolicy(String ownerId, List<Grant> grants) {
|
||||
/** A canonical user or predefined S3 group. */
|
||||
public enum GranteeType { CANONICAL_USER, GROUP }
|
||||
|
||||
/** An S3 ACL permission. */
|
||||
public enum Permission { READ, WRITE, READ_ACP, WRITE_ACP, FULL_CONTROL }
|
||||
|
||||
/** One ACL grant to a canonical user ID or S3 group URI. */
|
||||
public record Grant(GranteeType type, String grantee, Permission permission) {
|
||||
public Grant {
|
||||
Objects.requireNonNull(type, "type");
|
||||
if (grantee == null || grantee.isBlank()) throw new IllegalArgumentException("grantee is required");
|
||||
Objects.requireNonNull(permission, "permission");
|
||||
}
|
||||
}
|
||||
|
||||
public AclPolicy {
|
||||
if (ownerId == null || ownerId.isBlank()) throw new IllegalArgumentException("owner ID is required");
|
||||
grants = List.copyOf(grants);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
package cloud.lunarsky.objectstore.client;
|
||||
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.Set;
|
||||
|
||||
/** Reported or observed operation support, separate from the caller's authorization. */
|
||||
public record Capabilities(ServiceKind service, Map<String, Support> operations, Map<String, Long> limits,
|
||||
String serviceVersion, String storageMode, boolean completeOperationInventory) {
|
||||
private static final Set<String> KNOWN_OPERATIONS = Set.of(
|
||||
"ListBuckets", "CreateBucket", "HeadBucket", "DeleteBucket", "ListObjectsV2",
|
||||
"PutObject", "GetObject", "HeadObject", "DeleteObject", "CopyObject",
|
||||
"GetObjectTagging", "PutObjectTagging", "DeleteObjectTagging",
|
||||
"CreateMultipartUpload", "UploadPart", "ListParts", "CompleteMultipartUpload",
|
||||
"AbortMultipartUpload", "ListMultipartUploads", "GetBucketVersioning",
|
||||
"PutBucketVersioning", "ListObjectVersions", "GetBucketAcl", "PutBucketAcl",
|
||||
"GetObjectAcl", "PutObjectAcl");
|
||||
|
||||
/** Identifies a self-reported ObjectStore service or an unrecognized S3-compatible service. */
|
||||
public enum ServiceKind { OBJECTSTORE, UNKNOWN_S3 }
|
||||
|
||||
/** Three-state feature support; a generic S3 server cannot be inferred from its hostname. */
|
||||
public enum Support { SUPPORTED, UNSUPPORTED, UNKNOWN }
|
||||
|
||||
public Capabilities {
|
||||
Objects.requireNonNull(service, "service");
|
||||
operations = Map.copyOf(operations);
|
||||
limits = Map.copyOf(limits);
|
||||
}
|
||||
|
||||
/** Constructs a result without a server manifest or configured limits. */
|
||||
public Capabilities(ServiceKind service, Map<String, Support> operations) {
|
||||
this(service, operations, Map.of(), null, null, false);
|
||||
}
|
||||
|
||||
/** Returns support for a named operation. A missing manifest or denied probe stays UNKNOWN. */
|
||||
public Support support(String operation) {
|
||||
Support observed = operations.get(operation);
|
||||
if (observed != null) return observed;
|
||||
if (completeOperationInventory && KNOWN_OPERATIONS.contains(operation)) return Support.UNSUPPORTED;
|
||||
return Support.UNKNOWN;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
package cloud.lunarsky.objectstore.client;
|
||||
|
||||
import java.math.BigDecimal;
|
||||
import java.nio.ByteBuffer;
|
||||
import java.nio.charset.CharacterCodingException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
final class Json {
|
||||
private Json() { }
|
||||
|
||||
static Object parse(byte[] bytes) throws ProtocolException {
|
||||
String source;
|
||||
try { source = StandardCharsets.UTF_8.newDecoder().decode(ByteBuffer.wrap(bytes)).toString(); }
|
||||
catch (CharacterCodingException e) { throw new ProtocolException("Capability JSON is not UTF-8", e); }
|
||||
Parser parser = new Parser(source);
|
||||
Object value = parser.value(0);
|
||||
parser.space();
|
||||
if (parser.index != source.length()) throw new ProtocolException("Trailing capability JSON data");
|
||||
return value;
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
static Map<String, Object> object(Object value, String name) throws ProtocolException {
|
||||
if (!(value instanceof Map<?, ?>)) throw new ProtocolException(name + " must be an object");
|
||||
return (Map<String, Object>) value;
|
||||
}
|
||||
|
||||
static List<?> array(Object value, String name) throws ProtocolException {
|
||||
if (!(value instanceof List<?> list)) throw new ProtocolException(name + " must be an array");
|
||||
return list;
|
||||
}
|
||||
|
||||
static String string(Object value, String name) throws ProtocolException {
|
||||
if (!(value instanceof String text) || text.isBlank())
|
||||
throw new ProtocolException(name + " must be a nonempty string");
|
||||
return text;
|
||||
}
|
||||
|
||||
static long integer(Object value, String name) throws ProtocolException {
|
||||
if (!(value instanceof BigDecimal number)) throw new ProtocolException(name + " must be an integer");
|
||||
try { return number.longValueExact(); }
|
||||
catch (ArithmeticException e) { throw new ProtocolException(name + " is out of range", e); }
|
||||
}
|
||||
|
||||
private static final class Parser {
|
||||
private final String source;
|
||||
private int index;
|
||||
|
||||
private Parser(String source) { this.source = source; }
|
||||
|
||||
private void space() {
|
||||
while (index < source.length() && (source.charAt(index) == ' ' || source.charAt(index) == '\n' ||
|
||||
source.charAt(index) == '\r' || source.charAt(index) == '\t')) index++;
|
||||
}
|
||||
|
||||
private Object value(int depth) throws ProtocolException {
|
||||
if (depth > 16) throw new ProtocolException("Capability JSON is too deeply nested");
|
||||
space();
|
||||
if (index >= source.length()) throw new ProtocolException("Incomplete capability JSON");
|
||||
return switch (source.charAt(index)) {
|
||||
case '{' -> object(depth + 1);
|
||||
case '[' -> array(depth + 1);
|
||||
case '"' -> string();
|
||||
case 't' -> literal("true", Boolean.TRUE);
|
||||
case 'f' -> literal("false", Boolean.FALSE);
|
||||
case 'n' -> literal("null", null);
|
||||
default -> number();
|
||||
};
|
||||
}
|
||||
|
||||
private Map<String, Object> object(int depth) throws ProtocolException {
|
||||
index++;
|
||||
space();
|
||||
Map<String, Object> result = new HashMap<>();
|
||||
if (take('}')) return result;
|
||||
do {
|
||||
space();
|
||||
if (index >= source.length() || source.charAt(index) != '"')
|
||||
throw new ProtocolException("Capability JSON object key is missing");
|
||||
String key = string();
|
||||
space();
|
||||
require(':');
|
||||
if (result.containsKey(key)) throw new ProtocolException("Duplicate capability JSON key");
|
||||
result.put(key, value(depth));
|
||||
if (result.size() > 256) throw new ProtocolException("Capability JSON object is too large");
|
||||
space();
|
||||
if (take('}')) return result;
|
||||
require(',');
|
||||
} while (true);
|
||||
}
|
||||
|
||||
private List<Object> array(int depth) throws ProtocolException {
|
||||
index++;
|
||||
space();
|
||||
List<Object> result = new ArrayList<>();
|
||||
if (take(']')) return result;
|
||||
do {
|
||||
result.add(value(depth));
|
||||
if (result.size() > 2048) throw new ProtocolException("Capability JSON array is too large");
|
||||
space();
|
||||
if (take(']')) return result;
|
||||
require(',');
|
||||
} while (true);
|
||||
}
|
||||
|
||||
private String string() throws ProtocolException {
|
||||
index++;
|
||||
StringBuilder result = new StringBuilder();
|
||||
while (index < source.length()) {
|
||||
char current = source.charAt(index++);
|
||||
if (current == '"') {
|
||||
for (int i = 0; i < result.length(); i++) {
|
||||
char unit = result.charAt(i);
|
||||
if (Character.isHighSurrogate(unit)) {
|
||||
if (++i >= result.length() || !Character.isLowSurrogate(result.charAt(i)))
|
||||
throw new ProtocolException("Invalid JSON Unicode surrogate");
|
||||
} else if (Character.isLowSurrogate(unit))
|
||||
throw new ProtocolException("Invalid JSON Unicode surrogate");
|
||||
}
|
||||
return result.toString();
|
||||
}
|
||||
if (current < 0x20) throw new ProtocolException("Unescaped control character in JSON string");
|
||||
if (current != '\\') { result.append(current); continue; }
|
||||
if (index >= source.length()) throw new ProtocolException("Incomplete JSON escape");
|
||||
char escaped = source.charAt(index++);
|
||||
switch (escaped) {
|
||||
case '"', '\\', '/' -> result.append(escaped);
|
||||
case 'b' -> result.append('\b');
|
||||
case 'f' -> result.append('\f');
|
||||
case 'n' -> result.append('\n');
|
||||
case 'r' -> result.append('\r');
|
||||
case 't' -> result.append('\t');
|
||||
case 'u' -> {
|
||||
if (index + 4 > source.length()) throw new ProtocolException("Incomplete Unicode escape");
|
||||
int unit = 0;
|
||||
for (int i = 0; i < 4; i++) {
|
||||
int digit = Character.digit(source.charAt(index++), 16);
|
||||
if (digit < 0) throw new ProtocolException("Invalid Unicode escape");
|
||||
unit = unit * 16 + digit;
|
||||
}
|
||||
result.append((char) unit);
|
||||
}
|
||||
default -> throw new ProtocolException("Invalid JSON escape");
|
||||
}
|
||||
}
|
||||
throw new ProtocolException("Unterminated JSON string");
|
||||
}
|
||||
|
||||
private Object literal(String expected, Object value) throws ProtocolException {
|
||||
if (!source.startsWith(expected, index)) throw new ProtocolException("Invalid JSON literal");
|
||||
index += expected.length();
|
||||
return value;
|
||||
}
|
||||
|
||||
private BigDecimal number() throws ProtocolException {
|
||||
int start = index;
|
||||
if (take('-') && index >= source.length()) throw new ProtocolException("Invalid JSON number");
|
||||
if (take('0')) {
|
||||
if (index < source.length() && Character.isDigit(source.charAt(index)))
|
||||
throw new ProtocolException("Invalid JSON number");
|
||||
} else {
|
||||
if (index >= source.length() || source.charAt(index) < '1' || source.charAt(index) > '9')
|
||||
throw new ProtocolException("Invalid JSON number");
|
||||
while (index < source.length() && source.charAt(index) >= '0' && source.charAt(index) <= '9') index++;
|
||||
}
|
||||
if (take('.')) {
|
||||
int first = index;
|
||||
while (index < source.length() && source.charAt(index) >= '0' && source.charAt(index) <= '9') index++;
|
||||
if (first == index) throw new ProtocolException("Invalid JSON fraction");
|
||||
}
|
||||
if (take('e') || take('E')) {
|
||||
if (!take('+')) take('-');
|
||||
int first = index;
|
||||
while (index < source.length() && source.charAt(index) >= '0' && source.charAt(index) <= '9') index++;
|
||||
if (first == index) throw new ProtocolException("Invalid JSON exponent");
|
||||
}
|
||||
try { return new BigDecimal(source.substring(start, index)); }
|
||||
catch (NumberFormatException e) { throw new ProtocolException("Invalid JSON number", e); }
|
||||
}
|
||||
|
||||
private boolean take(char value) {
|
||||
if (index < source.length() && source.charAt(index) == value) { index++; return true; }
|
||||
return false;
|
||||
}
|
||||
|
||||
private void require(char value) throws ProtocolException {
|
||||
if (!take(value)) throw new ProtocolException("Expected '" + value + "' in capability JSON");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,633 @@
|
||||
package cloud.lunarsky.objectstore.client;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.net.URI;
|
||||
import java.net.URLDecoder;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.security.MessageDigest;
|
||||
import java.security.NoSuchAlgorithmException;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Base64;
|
||||
import java.util.Comparator;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import org.w3c.dom.Element;
|
||||
|
||||
/** A synchronous, path-style S3 client. Instances are safe to share between threads. */
|
||||
public final class ObjectStorageClient implements AutoCloseable {
|
||||
private static final int MAX_XML = 4 * 1024 * 1024;
|
||||
private static final int MAX_ERROR = 64 * 1024;
|
||||
private static final String EMPTY_HASH = SigV4.hash(new byte[0]);
|
||||
private final URI endpoint;
|
||||
private final String region;
|
||||
private final String accessKey;
|
||||
private final String secretKey;
|
||||
private final Duration timeout;
|
||||
private final Clock clock;
|
||||
private final HttpClient http;
|
||||
private final Set<String> observedOperations = ConcurrentHashMap.newKeySet();
|
||||
|
||||
ObjectStorageClient(URI endpoint, String region, String accessKey, String secretKey,
|
||||
Duration timeout, Clock clock) {
|
||||
this.endpoint = endpoint;
|
||||
this.region = region;
|
||||
this.accessKey = accessKey;
|
||||
this.secretKey = secretKey;
|
||||
this.timeout = timeout;
|
||||
this.clock = clock;
|
||||
this.http = HttpClient.newBuilder().followRedirects(HttpClient.Redirect.NEVER)
|
||||
.connectTimeout(timeout).version(HttpClient.Version.HTTP_1_1).build();
|
||||
}
|
||||
|
||||
/** Returns one open object stream. The caller must close the returned result. */
|
||||
public ObjectData getObject(String bucket, String key) throws IOException {
|
||||
HttpResponse<InputStream> response = send("GET", objectPath(bucket, key), Map.of(), Map.of(),
|
||||
HttpRequest.BodyPublishers.noBody(), EMPTY_HASH);
|
||||
if (!successful(response)) {
|
||||
try (InputStream ignored = response.body()) { throw failure(response); }
|
||||
}
|
||||
observedOperations.add("GetObject");
|
||||
return new ObjectData(response.body(), length(response), response.headers().firstValue("content-type").orElse(null),
|
||||
response.headers().firstValue("etag").orElse(null));
|
||||
}
|
||||
|
||||
/** Returns object headers without downloading content. */
|
||||
public ObjectMetadata headObject(String bucket, String key) throws IOException {
|
||||
HttpResponse<InputStream> response = send("HEAD", objectPath(bucket, key), Map.of(), Map.of(),
|
||||
HttpRequest.BodyPublishers.noBody(), EMPTY_HASH);
|
||||
try (InputStream ignored = response.body()) {
|
||||
if (!successful(response)) throw failure(response);
|
||||
observedOperations.add("HeadObject");
|
||||
return new ObjectMetadata(length(response), response.headers().firstValue("content-type").orElse(null),
|
||||
response.headers().firstValue("etag").orElse(null));
|
||||
}
|
||||
}
|
||||
|
||||
/** Uploads an immutable byte array as one object. No automatic write retry is performed. */
|
||||
public void putObject(String bucket, String key, byte[] content, String contentType) throws IOException {
|
||||
Objects.requireNonNull(content, "content");
|
||||
put(bucket, key, HttpRequest.BodyPublishers.ofByteArray(content), SigV4.hash(content), contentType);
|
||||
}
|
||||
|
||||
/** Uploads a file. Do not change the file between hashing and completion of the upload. */
|
||||
public void putObject(String bucket, String key, Path file, String contentType) throws IOException {
|
||||
Objects.requireNonNull(file, "file");
|
||||
String hash;
|
||||
try (InputStream input = Files.newInputStream(file)) {
|
||||
MessageDigest digest = MessageDigest.getInstance("SHA-256");
|
||||
byte[] buffer = new byte[65536];
|
||||
for (int count; (count = input.read(buffer)) >= 0; ) digest.update(buffer, 0, count);
|
||||
hash = java.util.HexFormat.of().formatHex(digest.digest());
|
||||
} catch (NoSuchAlgorithmException e) { throw new IllegalStateException(e); }
|
||||
put(bucket, key, HttpRequest.BodyPublishers.ofFile(file), hash, contentType);
|
||||
}
|
||||
|
||||
private void put(String bucket, String key, HttpRequest.BodyPublisher body, String hash,
|
||||
String contentType) throws IOException {
|
||||
Map<String, String> headers = contentType == null ? Map.of() : Map.of("content-type", contentType);
|
||||
HttpResponse<InputStream> response = send("PUT", objectPath(bucket, key), Map.of(), headers, body, hash);
|
||||
try (InputStream ignored = response.body()) {
|
||||
if (!successful(response)) throw failure(response);
|
||||
observedOperations.add("PutObject");
|
||||
}
|
||||
}
|
||||
|
||||
/** Deletes the current object. A versioned backend may create a delete marker. */
|
||||
public void deleteObject(String bucket, String key) throws IOException {
|
||||
HttpResponse<InputStream> response = send("DELETE", objectPath(bucket, key), Map.of(), Map.of(),
|
||||
HttpRequest.BodyPublishers.noBody(), EMPTY_HASH);
|
||||
try (InputStream ignored = response.body()) {
|
||||
if (!successful(response)) throw failure(response);
|
||||
observedOperations.add("DeleteObject");
|
||||
}
|
||||
}
|
||||
|
||||
/** Lists one page of objects using S3 ListObjectsV2. Pass the returned token to get the next page. */
|
||||
public ObjectPage listObjects(String bucket, String prefix, String continuationToken, int maxKeys)
|
||||
throws IOException {
|
||||
if (maxKeys < 1 || maxKeys > 1000) throw new IllegalArgumentException("maxKeys must be 1..1000");
|
||||
Map<String, String> query = new HashMap<>();
|
||||
query.put("list-type", "2");
|
||||
query.put("encoding-type", "url");
|
||||
query.put("max-keys", Integer.toString(maxKeys));
|
||||
if (prefix != null) query.put("prefix", prefix);
|
||||
if (continuationToken != null) query.put("continuation-token", continuationToken);
|
||||
Element root = xml("GET", bucketPath(bucket), query);
|
||||
requireRoot(root, "ListBucketResult");
|
||||
observedOperations.add("ListObjectsV2");
|
||||
List<ObjectEntry> entries = new ArrayList<>();
|
||||
var nodes = Xml.descendants(root, "Contents");
|
||||
for (int i = 0; i < nodes.getLength(); i++) {
|
||||
Element entry = (Element) nodes.item(i);
|
||||
String key = Xml.text(entry, "Key");
|
||||
String size = Xml.text(entry, "Size");
|
||||
if (key == null || size == null) throw new ProtocolException("Incomplete object listing");
|
||||
try {
|
||||
entries.add(new ObjectEntry(URLDecoder.decode(key.replace("+", "%2B"), StandardCharsets.UTF_8),
|
||||
Long.parseLong(size), Xml.text(entry, "ETag")));
|
||||
} catch (IllegalArgumentException e) { throw new ProtocolException("Invalid object listing entry", e); }
|
||||
}
|
||||
String next = Xml.text(root, "NextContinuationToken");
|
||||
if ("true".equalsIgnoreCase(Xml.text(root, "IsTruncated")) && (next == null || next.isEmpty()))
|
||||
throw new ProtocolException("Truncated listing has no continuation token");
|
||||
return new ObjectPage(entries, next);
|
||||
}
|
||||
|
||||
/** Starts a standard S3 multipart upload. Keep the returned upload ID to resume or abort later. */
|
||||
public MultipartUpload createMultipartUpload(String bucket, String key, String contentType) throws IOException {
|
||||
String path = objectPath(bucket, key);
|
||||
Map<String, String> headers = contentType == null ? Map.of() : Map.of("content-type", contentType);
|
||||
HttpResponse<InputStream> response = send("POST", path, Map.of("uploads", ""), headers,
|
||||
HttpRequest.BodyPublishers.noBody(), EMPTY_HASH);
|
||||
Element root = responseXml(response, "InitiateMultipartUploadResult");
|
||||
String id = Xml.text(root, "UploadId");
|
||||
if (id == null || id.isBlank()) throw new ProtocolException("Multipart initiation has no upload ID");
|
||||
observedOperations.add("CreateMultipartUpload");
|
||||
return new MultipartUpload(bucket, key, id);
|
||||
}
|
||||
|
||||
/** Uploads one part. The ETag in the result must be supplied when completing the upload. */
|
||||
public Part uploadPart(MultipartUpload upload, int partNumber, byte[] content) throws IOException {
|
||||
Objects.requireNonNull(upload, "upload");
|
||||
Objects.requireNonNull(content, "content");
|
||||
checkPartNumber(partNumber);
|
||||
HttpResponse<InputStream> response = send("PUT", objectPath(upload.bucket(), upload.key()),
|
||||
Map.of("uploadId", upload.uploadId(), "partNumber", Integer.toString(partNumber)), Map.of(),
|
||||
HttpRequest.BodyPublishers.ofByteArray(content), SigV4.hash(content));
|
||||
try (InputStream ignored = response.body()) {
|
||||
if (!successful(response)) throw failure(response);
|
||||
String etag = response.headers().firstValue("etag").orElse(null);
|
||||
if (etag == null || etag.isBlank()) throw new ProtocolException("Part upload has no ETag");
|
||||
observedOperations.add("UploadPart");
|
||||
return new Part(partNumber, etag, content.length);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Uploads an immutable file as sequential parts without completing it. The caller owns the
|
||||
* upload session and can retry, list parts, complete, or abort it after any failure.
|
||||
* Each part is held in memory once; partSize must be 5 to 128 MiB.
|
||||
*/
|
||||
public List<Part> uploadFileParts(MultipartUpload upload, Path file, int partSize) throws IOException {
|
||||
Objects.requireNonNull(upload, "upload");
|
||||
Objects.requireNonNull(file, "file");
|
||||
if (partSize < 5 * 1024 * 1024 || partSize > 128 * 1024 * 1024)
|
||||
throw new IllegalArgumentException("partSize must be 5..128 MiB");
|
||||
long size = Files.size(file);
|
||||
if (size < 1 || (size + partSize - 1L) / partSize > 10000)
|
||||
throw new IllegalArgumentException("file requires 1..10000 parts");
|
||||
List<Part> parts = new ArrayList<>();
|
||||
try (InputStream input = Files.newInputStream(file)) {
|
||||
long remaining = size;
|
||||
for (int number = 1; remaining > 0; number++) {
|
||||
int length = (int) Math.min(remaining, partSize);
|
||||
byte[] bytes = input.readNBytes(length);
|
||||
if (bytes.length != length) throw new IOException("File changed during multipart upload");
|
||||
parts.add(uploadPart(upload, number, bytes));
|
||||
remaining -= length;
|
||||
}
|
||||
if (input.read() != -1) throw new IOException("File changed during multipart upload");
|
||||
}
|
||||
return List.copyOf(parts);
|
||||
}
|
||||
|
||||
/** Lists one page of uploaded parts for resume or verification. */
|
||||
public PartPage listParts(MultipartUpload upload, int partNumberMarker, int maxParts) throws IOException {
|
||||
Objects.requireNonNull(upload, "upload");
|
||||
if (partNumberMarker < 0 || partNumberMarker > 10000 || maxParts < 1 || maxParts > 1000)
|
||||
throw new IllegalArgumentException("invalid part marker or page size");
|
||||
Element root = xml("GET", objectPath(upload.bucket(), upload.key()),
|
||||
Map.of("uploadId", upload.uploadId(), "part-number-marker", Integer.toString(partNumberMarker),
|
||||
"max-parts", Integer.toString(maxParts)));
|
||||
requireRoot(root, "ListPartsResult");
|
||||
observedOperations.add("ListParts");
|
||||
List<Part> parts = new ArrayList<>();
|
||||
var nodes = Xml.descendants(root, "Part");
|
||||
for (int i = 0; i < nodes.getLength(); i++) {
|
||||
Element entry = (Element) nodes.item(i);
|
||||
try {
|
||||
parts.add(new Part(Integer.parseInt(requiredText(entry, "PartNumber")),
|
||||
requiredText(entry, "ETag"), Long.parseLong(requiredText(entry, "Size"))));
|
||||
} catch (NumberFormatException e) { throw new ProtocolException("Invalid part listing entry", e); }
|
||||
}
|
||||
boolean truncated = Boolean.parseBoolean(Xml.text(root, "IsTruncated"));
|
||||
int next = partNumberMarker;
|
||||
if (truncated) {
|
||||
try { next = Integer.parseInt(requiredText(root, "NextPartNumberMarker")); }
|
||||
catch (NumberFormatException e) { throw new ProtocolException("Invalid next part marker", e); }
|
||||
if (next <= partNumberMarker) throw new ProtocolException("Part listing did not advance");
|
||||
}
|
||||
return new PartPage(parts, truncated, next);
|
||||
}
|
||||
|
||||
/** Completes the upload using the exact part numbers and ETags returned by the service. */
|
||||
public String completeMultipartUpload(MultipartUpload upload, List<Part> parts) throws IOException {
|
||||
Objects.requireNonNull(upload, "upload");
|
||||
Objects.requireNonNull(parts, "parts");
|
||||
if (parts.isEmpty() || parts.size() > 10000) throw new IllegalArgumentException("1..10000 parts required");
|
||||
List<Part> ordered = new ArrayList<>(parts);
|
||||
ordered.sort(Comparator.comparingInt(Part::number));
|
||||
StringBuilder xml = new StringBuilder("<CompleteMultipartUpload>");
|
||||
int previous = 0;
|
||||
for (Part part : ordered) {
|
||||
Objects.requireNonNull(part, "part");
|
||||
checkPartNumber(part.number());
|
||||
if (part.number() == previous) throw new IllegalArgumentException("duplicate part number");
|
||||
previous = part.number();
|
||||
xml.append("<Part><PartNumber>").append(part.number()).append("</PartNumber><ETag>")
|
||||
.append(Xml.escape(part.etag())).append("</ETag></Part>");
|
||||
}
|
||||
byte[] bytes = xml.append("</CompleteMultipartUpload>").toString().getBytes(StandardCharsets.UTF_8);
|
||||
HttpResponse<InputStream> response = send("POST", objectPath(upload.bucket(), upload.key()),
|
||||
Map.of("uploadId", upload.uploadId()), Map.of("content-type", "application/xml"),
|
||||
HttpRequest.BodyPublishers.ofByteArray(bytes), SigV4.hash(bytes));
|
||||
Element root = responseXml(response, "CompleteMultipartUploadResult");
|
||||
observedOperations.add("CompleteMultipartUpload");
|
||||
return requiredText(root, "ETag");
|
||||
}
|
||||
|
||||
/** Aborts an unfinished upload. A completed upload cannot be aborted. */
|
||||
public void abortMultipartUpload(MultipartUpload upload) throws IOException {
|
||||
Objects.requireNonNull(upload, "upload");
|
||||
HttpResponse<InputStream> response = send("DELETE", objectPath(upload.bucket(), upload.key()),
|
||||
Map.of("uploadId", upload.uploadId()), Map.of(), HttpRequest.BodyPublishers.noBody(), EMPTY_HASH);
|
||||
try (InputStream ignored = response.body()) {
|
||||
if (!successful(response)) throw failure(response);
|
||||
observedOperations.add("AbortMultipartUpload");
|
||||
}
|
||||
}
|
||||
|
||||
/** Lists one page of unfinished uploads in a bucket. */
|
||||
public MultipartUploadPage listMultipartUploads(String bucket, String prefix, String keyMarker,
|
||||
String uploadIdMarker, int maxUploads) throws IOException {
|
||||
if (maxUploads < 1 || maxUploads > 1000) throw new IllegalArgumentException("maxUploads must be 1..1000");
|
||||
if (uploadIdMarker != null && keyMarker == null)
|
||||
throw new IllegalArgumentException("upload ID marker requires key marker");
|
||||
Map<String, String> query = new HashMap<>();
|
||||
query.put("uploads", "");
|
||||
query.put("max-uploads", Integer.toString(maxUploads));
|
||||
if (prefix != null) query.put("prefix", prefix);
|
||||
if (keyMarker != null) query.put("key-marker", keyMarker);
|
||||
if (uploadIdMarker != null) query.put("upload-id-marker", uploadIdMarker);
|
||||
Element root = xml("GET", bucketPath(bucket), query);
|
||||
requireRoot(root, "ListMultipartUploadsResult");
|
||||
observedOperations.add("ListMultipartUploads");
|
||||
List<MultipartUpload> uploads = new ArrayList<>();
|
||||
var nodes = Xml.descendants(root, "Upload");
|
||||
for (int i = 0; i < nodes.getLength(); i++) {
|
||||
Element entry = (Element) nodes.item(i);
|
||||
uploads.add(new MultipartUpload(bucket, requiredText(entry, "Key"), requiredText(entry, "UploadId")));
|
||||
}
|
||||
boolean truncated = Boolean.parseBoolean(Xml.text(root, "IsTruncated"));
|
||||
String nextKey = Xml.text(root, "NextKeyMarker");
|
||||
String nextId = Xml.text(root, "NextUploadIdMarker");
|
||||
if (truncated && (nextKey == null || nextId == null))
|
||||
throw new ProtocolException("Truncated upload listing has no markers");
|
||||
return new MultipartUploadPage(uploads, truncated, nextKey, nextId);
|
||||
}
|
||||
|
||||
private static String requiredText(Element element, String name) throws ProtocolException {
|
||||
String value = Xml.text(element, name);
|
||||
if (value == null || value.isBlank()) throw new ProtocolException("Multipart response missing " + name);
|
||||
return value;
|
||||
}
|
||||
|
||||
private static void checkPartNumber(int number) {
|
||||
if (number < 1 || number > 10000) throw new IllegalArgumentException("part number must be 1..10000");
|
||||
}
|
||||
|
||||
/** Gets a bucket ACL. The backend may reject ACL operations when ACLs are disabled. */
|
||||
public AclPolicy getBucketAcl(String bucket) throws IOException {
|
||||
AclPolicy policy = readAcl(bucketPath(bucket));
|
||||
observedOperations.add("GetBucketAcl");
|
||||
return policy;
|
||||
}
|
||||
|
||||
/** Gets an object ACL. This does not calculate permissions from policies or other grants. */
|
||||
public AclPolicy getObjectAcl(String bucket, String key) throws IOException {
|
||||
AclPolicy policy = readAcl(objectPath(bucket, key));
|
||||
observedOperations.add("GetObjectAcl");
|
||||
return policy;
|
||||
}
|
||||
|
||||
/** Replaces a bucket ACL. No automatic retry is performed. */
|
||||
public void putBucketAcl(String bucket, AclPolicy policy) throws IOException {
|
||||
writeAcl(bucketPath(bucket), policy);
|
||||
observedOperations.add("PutBucketAcl");
|
||||
}
|
||||
|
||||
/** Replaces an object ACL. No automatic retry is performed. */
|
||||
public void putObjectAcl(String bucket, String key, AclPolicy policy) throws IOException {
|
||||
writeAcl(objectPath(bucket, key), policy);
|
||||
observedOperations.add("PutObjectAcl");
|
||||
}
|
||||
|
||||
private AclPolicy readAcl(String path) throws IOException {
|
||||
Element root = xml("GET", path, Map.of("acl", ""));
|
||||
requireRoot(root, "AccessControlPolicy");
|
||||
Element owner = Xml.child(root, "Owner");
|
||||
String ownerId = owner == null ? null : Xml.text(owner, "ID");
|
||||
if (ownerId == null || ownerId.isBlank()) throw new ProtocolException("ACL response has no owner ID");
|
||||
List<AclPolicy.Grant> grants = new ArrayList<>();
|
||||
var nodes = Xml.descendants(root, "Grant");
|
||||
for (int i = 0; i < nodes.getLength(); i++) {
|
||||
Element grant = (Element) nodes.item(i);
|
||||
Element grantee = Xml.child(grant, "Grantee");
|
||||
if (grantee == null) throw new ProtocolException("ACL grant has no grantee");
|
||||
String type = grantee.getAttributeNS("http://www.w3.org/2001/XMLSchema-instance", "type");
|
||||
if (type.isEmpty()) type = grantee.getAttribute("xsi:type");
|
||||
AclPolicy.GranteeType kind = switch (type) {
|
||||
case "CanonicalUser" -> AclPolicy.GranteeType.CANONICAL_USER;
|
||||
case "Group" -> AclPolicy.GranteeType.GROUP;
|
||||
default -> throw new ProtocolException("Unsupported ACL grantee type");
|
||||
};
|
||||
String id = Xml.text(grantee, kind == AclPolicy.GranteeType.GROUP ? "URI" : "ID");
|
||||
String permission = Xml.text(grant, "Permission");
|
||||
if (id == null || permission == null) throw new ProtocolException("Incomplete ACL grant");
|
||||
try { grants.add(new AclPolicy.Grant(kind, id, AclPolicy.Permission.valueOf(permission))); }
|
||||
catch (IllegalArgumentException e) { throw new ProtocolException("Unsupported ACL permission", e); }
|
||||
}
|
||||
return new AclPolicy(ownerId, grants);
|
||||
}
|
||||
|
||||
private void writeAcl(String path, AclPolicy policy) throws IOException {
|
||||
Objects.requireNonNull(policy, "policy");
|
||||
StringBuilder xml = new StringBuilder("<AccessControlPolicy xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\" ")
|
||||
.append("xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"><Owner><ID>")
|
||||
.append(Xml.escape(policy.ownerId())).append("</ID></Owner><AccessControlList>");
|
||||
for (AclPolicy.Grant grant : policy.grants()) {
|
||||
boolean group = grant.type() == AclPolicy.GranteeType.GROUP;
|
||||
xml.append("<Grant><Grantee xsi:type=\"").append(group ? "Group" : "CanonicalUser")
|
||||
.append("\"><").append(group ? "URI" : "ID").append('>')
|
||||
.append(Xml.escape(grant.grantee())).append("</").append(group ? "URI" : "ID")
|
||||
.append("></Grantee><Permission>").append(grant.permission()).append("</Permission></Grant>");
|
||||
}
|
||||
xml.append("</AccessControlList></AccessControlPolicy>");
|
||||
byte[] bytes = xml.toString().getBytes(StandardCharsets.UTF_8);
|
||||
String md5;
|
||||
try { md5 = Base64.getEncoder().encodeToString(MessageDigest.getInstance("MD5").digest(bytes)); }
|
||||
catch (NoSuchAlgorithmException e) { throw new IllegalStateException(e); }
|
||||
HttpResponse<InputStream> response = send("PUT", path, Map.of("acl", ""),
|
||||
Map.of("content-type", "application/xml", "content-md5", md5),
|
||||
HttpRequest.BodyPublishers.ofByteArray(bytes), SigV4.hash(bytes));
|
||||
try (InputStream ignored = response.body()) {
|
||||
if (!successful(response)) throw failure(response);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads ObjectStore's signed manifest when present. Other S3 endpoints report only operations
|
||||
* observed succeeding on this client. A denied or missing manifest does not imply a feature is
|
||||
* unsupported. Service identity from /health is self-reported, not cryptographic attestation.
|
||||
*/
|
||||
public Capabilities getCapabilities() throws IOException {
|
||||
HttpResponse<InputStream> response = send("GET", "/_objectstore/capabilities", Map.of(), Map.of(),
|
||||
HttpRequest.BodyPublishers.noBody(), EMPTY_HASH);
|
||||
ProtocolException invalidManifest = null;
|
||||
try (InputStream body = response.body()) {
|
||||
if (successful(response)) {
|
||||
try { return withObserved(parseManifest(readLimited(body, 64 * 1024))); }
|
||||
catch (ProtocolException e) { invalidManifest = e; }
|
||||
}
|
||||
}
|
||||
boolean objectStore = probeHealth();
|
||||
if (invalidManifest != null && objectStore) throw invalidManifest;
|
||||
return withObserved(new Capabilities(objectStore ? Capabilities.ServiceKind.OBJECTSTORE :
|
||||
Capabilities.ServiceKind.UNKNOWN_S3, Map.of()));
|
||||
}
|
||||
|
||||
/**
|
||||
* Opts into read-only probes on a known bucket and, optionally, an existing object key.
|
||||
* Successful calls establish support for this principal. S3 error responses leave support
|
||||
* UNKNOWN; transport and malformed-response errors are still reported to the caller.
|
||||
*/
|
||||
public Capabilities probeReadOnlyCapabilities(String bucket, String existingObjectKey) throws IOException {
|
||||
Capabilities base = getCapabilities();
|
||||
probe(base, "ListObjectsV2", () -> listObjects(bucket, null, null, 1));
|
||||
probe(base, "ListMultipartUploads", () -> listMultipartUploads(bucket, null, null, null, 1));
|
||||
probe(base, "GetBucketAcl", () -> getBucketAcl(bucket));
|
||||
if (existingObjectKey != null) {
|
||||
probe(base, "HeadObject", () -> headObject(bucket, existingObjectKey));
|
||||
probe(base, "GetObjectAcl", () -> getObjectAcl(bucket, existingObjectKey));
|
||||
}
|
||||
return withObserved(base);
|
||||
}
|
||||
|
||||
private void probe(Capabilities base, String operation, Probe call) throws IOException {
|
||||
if (base.support(operation) == Capabilities.Support.UNSUPPORTED) return;
|
||||
try { call.run(); }
|
||||
catch (ObjectStorageException ignored) { }
|
||||
}
|
||||
|
||||
@FunctionalInterface private interface Probe { void run() throws IOException; }
|
||||
|
||||
private Capabilities withObserved(Capabilities base) {
|
||||
Map<String, Capabilities.Support> operations = new HashMap<>(base.operations());
|
||||
observedOperations.forEach(name -> operations.put(name, Capabilities.Support.SUPPORTED));
|
||||
return new Capabilities(base.service(), operations, base.limits(), base.serviceVersion(),
|
||||
base.storageMode(), base.completeOperationInventory());
|
||||
}
|
||||
|
||||
private static Capabilities parseManifest(byte[] bytes) throws ProtocolException {
|
||||
Map<String, Object> document = Json.object(Json.parse(bytes), "capability manifest");
|
||||
if (Json.integer(document.get("schemaVersion"), "schemaVersion") != 1 ||
|
||||
!"lunarsky-objectstore".equals(Json.string(document.get("service"), "service")))
|
||||
throw new ProtocolException("Unsupported capability manifest");
|
||||
String version = Json.string(document.get("serviceVersion"), "serviceVersion");
|
||||
String mode = Json.string(document.get("storageMode"), "storageMode");
|
||||
Map<String, Capabilities.Support> operations = new HashMap<>();
|
||||
for (Object value : Json.array(document.get("operations"), "operations")) {
|
||||
String name = Json.string(value, "operation name");
|
||||
if (operations.put(name, Capabilities.Support.SUPPORTED) != null)
|
||||
throw new ProtocolException("Duplicate capability operation");
|
||||
}
|
||||
Map<String, Object> rawLimits = Json.object(document.get("limits"), "limits");
|
||||
Map<String, Long> limits = new HashMap<>();
|
||||
for (var entry : rawLimits.entrySet()) {
|
||||
long value = Json.integer(entry.getValue(), "limit " + entry.getKey());
|
||||
if (value < 0) throw new ProtocolException("Negative capability limit");
|
||||
limits.put(entry.getKey(), value);
|
||||
}
|
||||
for (String required : List.of("maxObjectBytes", "maxTotalBytes", "maxParts"))
|
||||
if (!limits.containsKey(required)) throw new ProtocolException("Missing capability limit " + required);
|
||||
return new Capabilities(Capabilities.ServiceKind.OBJECTSTORE, operations, limits, version, mode, true);
|
||||
}
|
||||
|
||||
private boolean probeHealth() throws IOException {
|
||||
URI uri = URI.create(origin() + "/health");
|
||||
HttpRequest request = HttpRequest.newBuilder(uri).timeout(timeout).GET().build();
|
||||
HttpResponse<InputStream> response = execute(request);
|
||||
try (InputStream body = response.body()) {
|
||||
if (response.statusCode() != 200) return false;
|
||||
String value = new String(readLimited(body, 1024), StandardCharsets.UTF_8);
|
||||
return value.matches("(?s)\\s*\\{\\s*\"status\"\\s*:\\s*\"ok\"\\s*,\\s*\"service\"\\s*:\\s*\"lunarsky-objectstore\"\\s*}\\s*");
|
||||
}
|
||||
}
|
||||
|
||||
private Element xml(String method, String path, Map<String, String> query) throws IOException {
|
||||
HttpResponse<InputStream> response = send(method, path, query, Map.of(),
|
||||
HttpRequest.BodyPublishers.noBody(), EMPTY_HASH);
|
||||
return responseXml(response, null);
|
||||
}
|
||||
|
||||
private static Element responseXml(HttpResponse<InputStream> response, String expectedRoot) throws IOException {
|
||||
try (InputStream body = response.body()) {
|
||||
if (!successful(response)) throw failure(response);
|
||||
Element root = Xml.parse(readLimited(body, MAX_XML)).getDocumentElement();
|
||||
if ("Error".equals(root.getLocalName())) {
|
||||
String code = Xml.text(root, "Code");
|
||||
throw new ObjectStorageException(response.statusCode(), code,
|
||||
response.headers().firstValue("x-amz-request-id").orElse(null));
|
||||
}
|
||||
if (expectedRoot != null) requireRoot(root, expectedRoot);
|
||||
return root;
|
||||
}
|
||||
}
|
||||
|
||||
private HttpResponse<InputStream> send(String method, String path, Map<String, String> query,
|
||||
Map<String, String> extra, HttpRequest.BodyPublisher body,
|
||||
String payloadHash) throws IOException {
|
||||
String queryString = SigV4.query(query);
|
||||
URI uri = URI.create(origin() + path + (queryString.isEmpty() ? "" : "?" + queryString));
|
||||
Instant now = clock.instant();
|
||||
Map<String, String> headers = new HashMap<>(extra);
|
||||
headers.put("host", uri.getRawAuthority().toLowerCase(Locale.ROOT));
|
||||
headers.put("x-amz-date", SigV4.timestamp(now));
|
||||
headers.put("x-amz-content-sha256", payloadHash);
|
||||
String authorization = SigV4.authorization(method, uri, headers, payloadHash, now,
|
||||
region, accessKey, secretKey);
|
||||
HttpRequest.Builder request = HttpRequest.newBuilder(uri).timeout(timeout);
|
||||
headers.forEach((name, value) -> {
|
||||
if (!"host".equals(name)) request.header(name, value);
|
||||
});
|
||||
request.header("Authorization", authorization);
|
||||
return execute(request.method(method, body).build());
|
||||
}
|
||||
|
||||
private HttpResponse<InputStream> execute(HttpRequest request) throws IOException {
|
||||
try { return http.send(request, HttpResponse.BodyHandlers.ofInputStream()); }
|
||||
catch (InterruptedException e) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IOException("Storage request interrupted", e);
|
||||
} catch (IOException e) { throw new TransportException(e); }
|
||||
}
|
||||
|
||||
private static boolean successful(HttpResponse<?> response) {
|
||||
return response.statusCode() >= 200 && response.statusCode() < 300;
|
||||
}
|
||||
|
||||
private static ObjectStorageException failure(HttpResponse<InputStream> response) throws IOException {
|
||||
String code = null;
|
||||
String requestId = response.headers().firstValue("x-amz-request-id").orElse(null);
|
||||
try {
|
||||
byte[] bytes = readLimited(response.body(), MAX_ERROR);
|
||||
if (bytes.length > 0) {
|
||||
Element root = Xml.parse(bytes).getDocumentElement();
|
||||
if ("Error".equals(root.getLocalName())) {
|
||||
code = Xml.text(root, "Code");
|
||||
if (requestId == null) requestId = Xml.text(root, "RequestId");
|
||||
}
|
||||
}
|
||||
} catch (IOException ignored) {
|
||||
}
|
||||
return new ObjectStorageException(response.statusCode(), code, requestId);
|
||||
}
|
||||
|
||||
private static byte[] readLimited(InputStream stream, int limit) throws IOException {
|
||||
byte[] bytes = stream.readNBytes(limit + 1);
|
||||
if (bytes.length > limit) throw new ProtocolException("Storage response exceeds size limit");
|
||||
return bytes;
|
||||
}
|
||||
|
||||
private static long length(HttpResponse<?> response) {
|
||||
return response.headers().firstValueAsLong("content-length").orElse(-1);
|
||||
}
|
||||
|
||||
private static void requireRoot(Element root, String name) throws IOException {
|
||||
if (!name.equals(root.getLocalName())) throw new ProtocolException("Unexpected storage XML response");
|
||||
}
|
||||
|
||||
private String origin() {
|
||||
return endpoint.getScheme() + "://" + endpoint.getRawAuthority();
|
||||
}
|
||||
|
||||
private static String bucketPath(String bucket) {
|
||||
validateBucket(bucket);
|
||||
return "/" + SigV4.encode(bucket, false);
|
||||
}
|
||||
|
||||
private static String objectPath(String bucket, String key) {
|
||||
if (key == null || key.isEmpty() || key.indexOf('\0') >= 0 ||
|
||||
key.getBytes(StandardCharsets.UTF_8).length > 1024)
|
||||
throw new IllegalArgumentException("object key must be 1..1024 UTF-8 bytes without NUL");
|
||||
return bucketPath(bucket) + "/" + SigV4.encode(key, true);
|
||||
}
|
||||
|
||||
private static void validateBucket(String bucket) {
|
||||
if (bucket == null || !bucket.matches("[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]"))
|
||||
throw new IllegalArgumentException("bucket must be a 3..63 character DNS-style name");
|
||||
}
|
||||
|
||||
/** Closes the underlying HTTP client. Do not use this instance afterward. */
|
||||
@Override public void close() { http.close(); }
|
||||
|
||||
/** An open response stream and selected object headers. Close this result after reading. */
|
||||
public record ObjectData(InputStream body, long length, String contentType, String etag)
|
||||
implements AutoCloseable {
|
||||
@Override public void close() throws IOException { body.close(); }
|
||||
}
|
||||
|
||||
/** Headers returned by a HEAD request. A negative length means it was not supplied. */
|
||||
public record ObjectMetadata(long length, String contentType, String etag) { }
|
||||
|
||||
/** A listed object. */
|
||||
public record ObjectEntry(String key, long size, String etag) { }
|
||||
|
||||
/** One listing page. A null next token means no next page was reported. */
|
||||
public record ObjectPage(List<ObjectEntry> objects, String nextContinuationToken) {
|
||||
public ObjectPage { objects = List.copyOf(objects); }
|
||||
}
|
||||
|
||||
/** An unfinished multipart upload. Persist all three fields if resuming in a later process. */
|
||||
public record MultipartUpload(String bucket, String key, String uploadId) {
|
||||
public MultipartUpload {
|
||||
validateBucket(bucket);
|
||||
if (key == null || key.isEmpty()) throw new IllegalArgumentException("key is required");
|
||||
if (uploadId == null || uploadId.isBlank()) throw new IllegalArgumentException("upload ID is required");
|
||||
}
|
||||
}
|
||||
|
||||
/** A successfully uploaded part, including the service-provided ETag. */
|
||||
public record Part(int number, String etag, long size) {
|
||||
public Part {
|
||||
checkPartNumber(number);
|
||||
if (etag == null || etag.isBlank()) throw new IllegalArgumentException("ETag is required");
|
||||
if (size < 0) throw new IllegalArgumentException("part size must not be negative");
|
||||
}
|
||||
}
|
||||
|
||||
/** A page of uploaded parts. Pass nextPartNumberMarker to the next call when truncated. */
|
||||
public record PartPage(List<Part> parts, boolean truncated, int nextPartNumberMarker) {
|
||||
public PartPage { parts = List.copyOf(parts); }
|
||||
}
|
||||
|
||||
/** A page of unfinished uploads. Pass both next markers to the next call when truncated. */
|
||||
public record MultipartUploadPage(List<MultipartUpload> uploads, boolean truncated,
|
||||
String nextKeyMarker, String nextUploadIdMarker) {
|
||||
public MultipartUploadPage { uploads = List.copyOf(uploads); }
|
||||
}
|
||||
}
|
||||
+70
@@ -0,0 +1,70 @@
|
||||
package cloud.lunarsky.objectstore.client;
|
||||
|
||||
import java.net.URI;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.util.Objects;
|
||||
|
||||
/** Configures a path-style, Signature V4 S3 client without third-party dependencies. */
|
||||
public final class ObjectStorageClientBuilder {
|
||||
private URI endpoint;
|
||||
private String region = "us-east-1";
|
||||
private String accessKey;
|
||||
private String secretKey;
|
||||
private Duration timeout = Duration.ofSeconds(30);
|
||||
private boolean allowInsecureHttp;
|
||||
|
||||
/** Sets the service root, for example {@code https://storage.example.com}. */
|
||||
public ObjectStorageClientBuilder endpoint(URI value) {
|
||||
endpoint = Objects.requireNonNull(value, "endpoint");
|
||||
return this;
|
||||
}
|
||||
|
||||
/** Sets the Signature V4 region. The default is {@code us-east-1}. */
|
||||
public ObjectStorageClientBuilder region(String value) {
|
||||
region = requireText(value, "region");
|
||||
return this;
|
||||
}
|
||||
|
||||
/** Sets credentials used only in request signatures. Keep them out of logs. */
|
||||
public ObjectStorageClientBuilder credentials(String access, String secret) {
|
||||
accessKey = requireText(access, "access key");
|
||||
secretKey = requireText(secret, "secret key");
|
||||
return this;
|
||||
}
|
||||
|
||||
/** Sets the per-request timeout. */
|
||||
public ObjectStorageClientBuilder timeout(Duration value) {
|
||||
if (Objects.requireNonNull(value, "timeout").isNegative() || value.isZero())
|
||||
throw new IllegalArgumentException("timeout must be positive");
|
||||
timeout = value;
|
||||
return this;
|
||||
}
|
||||
|
||||
/** Allows plain HTTP for a trusted local test endpoint. HTTPS is required by default. */
|
||||
public ObjectStorageClientBuilder allowInsecureHttp() {
|
||||
allowInsecureHttp = true;
|
||||
return this;
|
||||
}
|
||||
|
||||
/** Builds an independent client. */
|
||||
public ObjectStorageClient build() {
|
||||
if (endpoint == null) throw new IllegalStateException("endpoint is required");
|
||||
if (accessKey == null || secretKey == null) throw new IllegalStateException("credentials are required");
|
||||
if (!region.matches("[a-z0-9-]+")) throw new IllegalArgumentException("invalid region");
|
||||
if (!accessKey.matches("[A-Za-z0-9_+=./@-]+")) throw new IllegalArgumentException("invalid access key");
|
||||
String scheme = endpoint.getScheme();
|
||||
if (!"https".equalsIgnoreCase(scheme) && !(allowInsecureHttp && "http".equalsIgnoreCase(scheme)))
|
||||
throw new IllegalArgumentException("HTTPS endpoint required unless insecure HTTP is explicitly allowed");
|
||||
if (endpoint.getHost() == null || endpoint.getUserInfo() != null || endpoint.getRawQuery() != null ||
|
||||
endpoint.getRawFragment() != null || !(endpoint.getRawPath() == null || endpoint.getRawPath().isEmpty() ||
|
||||
"/".equals(endpoint.getRawPath())))
|
||||
throw new IllegalArgumentException("endpoint must be an origin without path, query, fragment, or user info");
|
||||
return new ObjectStorageClient(endpoint, region, accessKey, secretKey, timeout, Clock.systemUTC());
|
||||
}
|
||||
|
||||
private static String requireText(String value, String name) {
|
||||
if (value == null || value.isBlank()) throw new IllegalArgumentException(name + " is required");
|
||||
return value;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
package cloud.lunarsky.objectstore.client;
|
||||
|
||||
import java.io.IOException;
|
||||
|
||||
/** An S3 error response with stable fields for callers to inspect. */
|
||||
public final class ObjectStorageException extends IOException {
|
||||
private final int statusCode;
|
||||
private final String errorCode;
|
||||
private final String requestId;
|
||||
|
||||
ObjectStorageException(int statusCode, String errorCode, String requestId) {
|
||||
super("Storage request failed: HTTP " + statusCode + (errorCode == null ? "" : " (" + errorCode + ")"));
|
||||
this.statusCode = statusCode;
|
||||
this.errorCode = errorCode;
|
||||
this.requestId = requestId;
|
||||
}
|
||||
|
||||
/** Returns the HTTP status. */
|
||||
public int statusCode() { return statusCode; }
|
||||
|
||||
/** Returns the S3 error code, or {@code null} if the server supplied none. */
|
||||
public String errorCode() { return errorCode; }
|
||||
|
||||
/** Returns the request ID, or {@code null}. */
|
||||
public String requestId() { return requestId; }
|
||||
|
||||
/** Returns whether retry might help. A failed write may already have reached the server. */
|
||||
public boolean retryable() { return statusCode == 429 || statusCode == 500 || statusCode == 502 ||
|
||||
statusCode == 503 || statusCode == 504; }
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
package cloud.lunarsky.objectstore.client;
|
||||
|
||||
import java.io.IOException;
|
||||
|
||||
/** A successful HTTP response that does not match the expected storage protocol. */
|
||||
public final class ProtocolException extends IOException {
|
||||
ProtocolException(String message) { super(message); }
|
||||
ProtocolException(String message, Throwable cause) { super(message, cause); }
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package cloud.lunarsky.objectstore.client;
|
||||
|
||||
import java.net.URI;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.security.NoSuchAlgorithmException;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.time.format.DateTimeFormatter;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HexFormat;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.TreeMap;
|
||||
import javax.crypto.Mac;
|
||||
import javax.crypto.spec.SecretKeySpec;
|
||||
|
||||
final class SigV4 {
|
||||
private static final DateTimeFormatter DATE = DateTimeFormatter.ofPattern("yyyyMMdd", Locale.ROOT)
|
||||
.withZone(ZoneOffset.UTC);
|
||||
private static final DateTimeFormatter TIME = DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'", Locale.ROOT)
|
||||
.withZone(ZoneOffset.UTC);
|
||||
|
||||
private SigV4() { }
|
||||
|
||||
static String hash(byte[] bytes) {
|
||||
try { return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(bytes)); }
|
||||
catch (NoSuchAlgorithmException e) { throw new IllegalStateException(e); }
|
||||
}
|
||||
|
||||
static String encode(String text, boolean keepSlash) {
|
||||
StringBuilder result = new StringBuilder();
|
||||
for (byte value : text.getBytes(StandardCharsets.UTF_8)) {
|
||||
int b = value & 0xff;
|
||||
if (b >= 'A' && b <= 'Z' || b >= 'a' && b <= 'z' || b >= '0' && b <= '9' ||
|
||||
b == '-' || b == '_' || b == '.' || b == '~' || keepSlash && b == '/') result.append((char) b);
|
||||
else result.append('%').append(Character.toUpperCase(Character.forDigit(b >>> 4, 16)))
|
||||
.append(Character.toUpperCase(Character.forDigit(b & 15, 16)));
|
||||
}
|
||||
return result.toString();
|
||||
}
|
||||
|
||||
static String query(Map<String, String> values) {
|
||||
List<String> entries = new ArrayList<>();
|
||||
values.forEach((key, value) -> entries.add(encode(key, false) + "=" + encode(value, false)));
|
||||
entries.sort(String::compareTo);
|
||||
return String.join("&", entries);
|
||||
}
|
||||
|
||||
static String authorization(String method, URI uri, Map<String, String> headers, String payloadHash,
|
||||
Instant now, String region, String accessKey, String secretKey) {
|
||||
TreeMap<String, String> signed = new TreeMap<>();
|
||||
headers.forEach((name, value) -> signed.put(name.toLowerCase(Locale.ROOT),
|
||||
value.trim().replaceAll("\\s+", " ")));
|
||||
String names = String.join(";", signed.keySet());
|
||||
StringBuilder canonicalHeaders = new StringBuilder();
|
||||
signed.forEach((name, value) -> canonicalHeaders.append(name).append(':').append(value).append('\n'));
|
||||
String canonical = method + '\n' + uri.getRawPath() + '\n' +
|
||||
(uri.getRawQuery() == null ? "" : uri.getRawQuery()) + '\n' + canonicalHeaders + '\n' +
|
||||
names + '\n' + payloadHash;
|
||||
String day = DATE.format(now);
|
||||
String scope = day + '/' + region + "/s3/aws4_request";
|
||||
String toSign = "AWS4-HMAC-SHA256\n" + TIME.format(now) + '\n' + scope + '\n' +
|
||||
hash(canonical.getBytes(StandardCharsets.UTF_8));
|
||||
byte[] key = hmac(("AWS4" + secretKey).getBytes(StandardCharsets.UTF_8), day);
|
||||
key = hmac(key, region);
|
||||
key = hmac(key, "s3");
|
||||
key = hmac(key, "aws4_request");
|
||||
return "AWS4-HMAC-SHA256 Credential=" + accessKey + '/' + scope + ",SignedHeaders=" + names +
|
||||
",Signature=" + HexFormat.of().formatHex(hmac(key, toSign));
|
||||
}
|
||||
|
||||
private static byte[] hmac(byte[] key, String value) {
|
||||
try {
|
||||
Mac mac = Mac.getInstance("HmacSHA256");
|
||||
mac.init(new SecretKeySpec(key, "HmacSHA256"));
|
||||
return mac.doFinal(value.getBytes(StandardCharsets.UTF_8));
|
||||
} catch (Exception e) { throw new IllegalStateException("HMAC-SHA256 unavailable", e); }
|
||||
}
|
||||
|
||||
static String timestamp(Instant now) { return TIME.format(now); }
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
package cloud.lunarsky.objectstore.client;
|
||||
|
||||
import java.io.IOException;
|
||||
|
||||
/** A connection or I/O failure. A write may have completed before this was observed. */
|
||||
public final class TransportException extends IOException {
|
||||
TransportException(IOException cause) { super("Storage transport failed", cause); }
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
package cloud.lunarsky.objectstore.client;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.IOException;
|
||||
import javax.xml.XMLConstants;
|
||||
import javax.xml.parsers.DocumentBuilderFactory;
|
||||
import javax.xml.parsers.ParserConfigurationException;
|
||||
import org.w3c.dom.Document;
|
||||
import org.w3c.dom.Element;
|
||||
import org.w3c.dom.Node;
|
||||
import org.w3c.dom.NodeList;
|
||||
import org.xml.sax.SAXException;
|
||||
|
||||
final class Xml {
|
||||
private Xml() { }
|
||||
|
||||
static Document parse(byte[] bytes) throws IOException {
|
||||
try {
|
||||
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
|
||||
factory.setNamespaceAware(true);
|
||||
factory.setFeature("http://apache.org/xml/features/disallow-doctype-decl", true);
|
||||
factory.setFeature("http://xml.org/sax/features/external-general-entities", false);
|
||||
factory.setFeature("http://xml.org/sax/features/external-parameter-entities", false);
|
||||
factory.setFeature(XMLConstants.FEATURE_SECURE_PROCESSING, true);
|
||||
factory.setXIncludeAware(false);
|
||||
factory.setExpandEntityReferences(false);
|
||||
factory.setAttribute(XMLConstants.ACCESS_EXTERNAL_DTD, "");
|
||||
factory.setAttribute(XMLConstants.ACCESS_EXTERNAL_SCHEMA, "");
|
||||
return factory.newDocumentBuilder().parse(new ByteArrayInputStream(bytes));
|
||||
} catch (ParserConfigurationException | SAXException | IllegalArgumentException e) {
|
||||
throw new ProtocolException("Malformed or unsafe XML response", e);
|
||||
}
|
||||
}
|
||||
|
||||
static Element child(Element element, String name) {
|
||||
for (Node node = element.getFirstChild(); node != null; node = node.getNextSibling())
|
||||
if (node instanceof Element found && name.equals(found.getLocalName())) return found;
|
||||
return null;
|
||||
}
|
||||
|
||||
static String text(Element element, String name) {
|
||||
Element found = child(element, name);
|
||||
return found == null ? null : found.getTextContent();
|
||||
}
|
||||
|
||||
static NodeList descendants(Element element, String name) {
|
||||
return element.getElementsByTagNameNS("*", name);
|
||||
}
|
||||
|
||||
static String escape(String text) {
|
||||
return text.replace("&", "&").replace("<", "<").replace(">", ">")
|
||||
.replace("\"", """).replace("'", "'");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,190 @@
|
||||
package cloud.lunarsky.objectstore.client;
|
||||
|
||||
import com.sun.net.httpserver.HttpExchange;
|
||||
import com.sun.net.httpserver.HttpServer;
|
||||
import java.io.IOException;
|
||||
import java.net.InetSocketAddress;
|
||||
import java.net.URI;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
|
||||
public final class ClientTest {
|
||||
private static final String S3_NS = "http://s3.amazonaws.com/doc/2006-03-01/";
|
||||
|
||||
public static void main(String[] args) throws Exception {
|
||||
signingVector();
|
||||
protocol();
|
||||
System.out.println("Client tests passed");
|
||||
}
|
||||
|
||||
private static void signingVector() {
|
||||
URI uri = URI.create("https://examplebucket.s3.amazonaws.com/test.txt");
|
||||
String empty = SigV4.hash(new byte[0]);
|
||||
String actual = SigV4.authorization("GET", uri, Map.of(
|
||||
"host", "examplebucket.s3.amazonaws.com", "range", "bytes=0-9",
|
||||
"x-amz-content-sha256", empty, "x-amz-date", "20130524T000000Z"), empty,
|
||||
Instant.parse("2013-05-24T00:00:00Z"), "us-east-1", "AKIAIOSFODNN7EXAMPLE",
|
||||
"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY");
|
||||
check(actual.endsWith("Signature=f0e8bdb87c964420e857bd35b5d6ed310bd44f0170aba48dd91039c6036bdb41"),
|
||||
"AWS Signature V4 vector");
|
||||
check("a=%20&z=%2F".equals(SigV4.query(Map.of("z", "/", "a", " "))), "query encoding");
|
||||
}
|
||||
|
||||
private static void protocol() throws Exception {
|
||||
AtomicReference<String> observedHash = new AtomicReference<>();
|
||||
AtomicReference<String> observedAcl = new AtomicReference<>();
|
||||
AtomicReference<String> manifest = new AtomicReference<>();
|
||||
AtomicInteger manifestStatus = new AtomicInteger(200);
|
||||
AtomicBoolean health = new AtomicBoolean(true);
|
||||
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
|
||||
server.createContext("/", exchange -> {
|
||||
try { handle(exchange, observedHash, observedAcl, manifest, manifestStatus, health); }
|
||||
finally { exchange.close(); }
|
||||
});
|
||||
server.start();
|
||||
try (ObjectStorageClient client = new ObjectStorageClientBuilder()
|
||||
.endpoint(URI.create("http://127.0.0.1:" + server.getAddress().getPort()))
|
||||
.allowInsecureHttp().credentials("test-key", "test-secret").build()) {
|
||||
client.putObject("mybucket", "hello world.txt", "hello".getBytes(StandardCharsets.UTF_8), "text/plain");
|
||||
check(SigV4.hash("hello".getBytes(StandardCharsets.UTF_8)).equals(observedHash.get()), "payload hash");
|
||||
try (ObjectStorageClient.ObjectData data = client.getObject("mybucket", "hello world.txt")) {
|
||||
check("hello".equals(new String(data.body().readAllBytes(), StandardCharsets.UTF_8)), "GET body");
|
||||
}
|
||||
check("text/plain".equals(client.headObject("mybucket", "hello world.txt").contentType()),
|
||||
"HEAD content type");
|
||||
var page = client.listObjects("mybucket", "hello", null, 10);
|
||||
check(page.objects().size() == 1 && "hello world.txt".equals(page.objects().getFirst().key()),
|
||||
"list object key");
|
||||
check("next-token".equals(page.nextContinuationToken()), "list continuation");
|
||||
AclPolicy policy = client.getBucketAcl("mybucket");
|
||||
check(policy.grants().size() == 1 && policy.grants().getFirst().permission() ==
|
||||
AclPolicy.Permission.FULL_CONTROL, "GET ACL");
|
||||
client.putObjectAcl("mybucket", "hello world.txt", new AclPolicy("owner-id", List.of(
|
||||
new AclPolicy.Grant(AclPolicy.GranteeType.GROUP,
|
||||
"http://acs.amazonaws.com/groups/global/AllUsers", AclPolicy.Permission.READ))));
|
||||
check(observedAcl.get().contains("<Permission>READ</Permission>"), "PUT ACL");
|
||||
check(client.getCapabilities().service() == Capabilities.ServiceKind.OBJECTSTORE, "service discovery");
|
||||
check(client.getCapabilities().support("versioning") == Capabilities.Support.UNKNOWN,
|
||||
"unknown support remains unknown");
|
||||
manifest.set("{\"schemaVersion\":1,\"service\":\"lunarsky-objectstore\"," +
|
||||
"\"serviceVersion\":\"0.0.5\",\"storageMode\":\"disk\"," +
|
||||
"\"operations\":[\"PutObject\",\"ListParts\"]," +
|
||||
"\"limits\":{\"maxObjectBytes\":1024,\"maxTotalBytes\":4096,\"maxParts\":10000}}");
|
||||
var capabilities = client.getCapabilities();
|
||||
check(capabilities.service() == Capabilities.ServiceKind.OBJECTSTORE &&
|
||||
capabilities.support("ListParts") == Capabilities.Support.SUPPORTED &&
|
||||
capabilities.support("PutBucketAcl") == Capabilities.Support.UNSUPPORTED &&
|
||||
capabilities.limits().get("maxParts") == 10000 &&
|
||||
"0.0.5".equals(capabilities.serviceVersion()), "signed manifest");
|
||||
manifest.set("{\"schemaVersion\":1,\"schemaVersion\":1}");
|
||||
try {
|
||||
client.getCapabilities();
|
||||
throw new AssertionError("Expected malformed manifest");
|
||||
} catch (ProtocolException expected) { }
|
||||
health.set(false);
|
||||
check(client.getCapabilities().service() == Capabilities.ServiceKind.UNKNOWN_S3,
|
||||
"generic S3 with unrelated matching path");
|
||||
manifestStatus.set(503);
|
||||
check(client.getCapabilities().support("CopyObject") == Capabilities.Support.UNKNOWN,
|
||||
"ambiguous manifest failure stays unknown");
|
||||
manifestStatus.set(200);
|
||||
manifest.set(null);
|
||||
check(client.getCapabilities().service() == Capabilities.ServiceKind.UNKNOWN_S3, "generic S3 fallback");
|
||||
check(client.getCapabilities().support("PutObject") == Capabilities.Support.SUPPORTED &&
|
||||
client.getCapabilities().support("CopyObject") == Capabilities.Support.UNKNOWN,
|
||||
"generic S3 observed support");
|
||||
capabilities = client.probeReadOnlyCapabilities("mybucket", "hello world.txt");
|
||||
check(capabilities.support("ListMultipartUploads") == Capabilities.Support.UNKNOWN &&
|
||||
capabilities.support("ListObjectsV2") == Capabilities.Support.SUPPORTED,
|
||||
"denied generic probe remains unknown");
|
||||
client.deleteObject("mybucket", "hello world.txt");
|
||||
try {
|
||||
client.getObject("mybucket", "denied");
|
||||
throw new AssertionError("Expected AccessDenied");
|
||||
} catch (ObjectStorageException error) {
|
||||
check(error.statusCode() == 403 && "AccessDenied".equals(error.errorCode()) &&
|
||||
"request-123".equals(error.requestId()), "typed S3 error");
|
||||
}
|
||||
} finally { server.stop(0); }
|
||||
}
|
||||
|
||||
private static void handle(HttpExchange exchange, AtomicReference<String> hash,
|
||||
AtomicReference<String> acl, AtomicReference<String> manifest,
|
||||
AtomicInteger manifestStatus, AtomicBoolean health) throws IOException {
|
||||
String path = exchange.getRequestURI().getRawPath();
|
||||
String query = exchange.getRequestURI().getRawQuery();
|
||||
if ("/health".equals(path)) {
|
||||
respond(exchange, health.get() ? 200 : 404,
|
||||
health.get() ? "{\"status\":\"ok\",\"service\":\"lunarsky-objectstore\"}" : "");
|
||||
return;
|
||||
}
|
||||
check(exchange.getRequestHeaders().getFirst("Authorization") != null, "signed request");
|
||||
if ("/_objectstore/capabilities".equals(path)) {
|
||||
String value = manifest.get();
|
||||
respond(exchange, value == null ? 404 : manifestStatus.get(), value == null ? "" : value);
|
||||
return;
|
||||
}
|
||||
if ("/mybucket".equals(path) && query != null && query.contains("uploads=")) {
|
||||
respond(exchange, 403, "<Error><Code>AccessDenied</Code></Error>");
|
||||
return;
|
||||
}
|
||||
if ("/mybucket".equals(path) && query != null && query.contains("list-type=2")) {
|
||||
respond(exchange, 200, "<ListBucketResult xmlns=\"" + S3_NS + "\"><Contents><Key>hello%20world.txt</Key>" +
|
||||
"<Size>5</Size><ETag>\"abc\"</ETag></Contents><NextContinuationToken>next-token" +
|
||||
"</NextContinuationToken></ListBucketResult>");
|
||||
return;
|
||||
}
|
||||
if (query != null && query.equals("acl=")) {
|
||||
if ("GET".equals(exchange.getRequestMethod())) {
|
||||
respond(exchange, 200, "<AccessControlPolicy xmlns=\"" + S3_NS + "\" " +
|
||||
"xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"><Owner><ID>owner-id</ID></Owner>" +
|
||||
"<AccessControlList><Grant><Grantee xsi:type=\"CanonicalUser\"><ID>owner-id</ID>" +
|
||||
"</Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>");
|
||||
} else {
|
||||
acl.set(new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8));
|
||||
check(exchange.getRequestHeaders().getFirst("Content-MD5") != null, "ACL Content-MD5");
|
||||
respond(exchange, 200, "");
|
||||
}
|
||||
return;
|
||||
}
|
||||
if ("/mybucket/denied".equals(path)) {
|
||||
exchange.getResponseHeaders().add("x-amz-request-id", "request-123");
|
||||
respond(exchange, 403, "<Error><Code>AccessDenied</Code></Error>");
|
||||
return;
|
||||
}
|
||||
check("/mybucket/hello%20world.txt".equals(path), "path-style key encoding");
|
||||
switch (exchange.getRequestMethod()) {
|
||||
case "PUT" -> {
|
||||
byte[] body = exchange.getRequestBody().readAllBytes();
|
||||
hash.set(exchange.getRequestHeaders().getFirst("x-amz-content-sha256"));
|
||||
check(SigV4.hash(body).equals(hash.get()), "uploaded body hash");
|
||||
respond(exchange, 200, "");
|
||||
}
|
||||
case "GET" -> respond(exchange, 200, "hello");
|
||||
case "HEAD" -> {
|
||||
exchange.getResponseHeaders().add("Content-Type", "text/plain");
|
||||
exchange.sendResponseHeaders(200, -1);
|
||||
}
|
||||
case "DELETE" -> respond(exchange, 204, "");
|
||||
default -> throw new AssertionError("Unexpected method");
|
||||
}
|
||||
}
|
||||
|
||||
private static void respond(HttpExchange exchange, int status, String content) throws IOException {
|
||||
byte[] bytes = content.getBytes(StandardCharsets.UTF_8);
|
||||
if (status == 204 || bytes.length == 0) exchange.sendResponseHeaders(status, -1);
|
||||
else {
|
||||
exchange.sendResponseHeaders(status, bytes.length);
|
||||
exchange.getResponseBody().write(bytes);
|
||||
}
|
||||
}
|
||||
|
||||
private static void check(boolean condition, String description) {
|
||||
if (!condition) throw new AssertionError(description);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
package cloud.lunarsky.objectstore.client;
|
||||
|
||||
import com.sun.net.httpserver.HttpExchange;
|
||||
import com.sun.net.httpserver.HttpServer;
|
||||
import java.io.IOException;
|
||||
import java.net.InetSocketAddress;
|
||||
import java.net.URI;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
|
||||
public final class MultipartClientTest {
|
||||
public static void main(String[] args) throws Exception {
|
||||
AtomicReference<String> completion = new AtomicReference<>();
|
||||
AtomicBoolean completionError = new AtomicBoolean();
|
||||
List<Integer> sizes = new ArrayList<>();
|
||||
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
|
||||
server.createContext("/", exchange -> {
|
||||
try { handle(exchange, completion, completionError, sizes); }
|
||||
finally { exchange.close(); }
|
||||
});
|
||||
server.start();
|
||||
try (ObjectStorageClient client = new ObjectStorageClientBuilder()
|
||||
.endpoint(URI.create("http://127.0.0.1:" + server.getAddress().getPort()))
|
||||
.allowInsecureHttp().credentials("test-key", "test-secret").build()) {
|
||||
var upload = client.createMultipartUpload("mybucket", "large file.bin", "application/octet-stream");
|
||||
check("u-1".equals(upload.uploadId()), "initiation ID");
|
||||
var first = client.uploadPart(upload, 1, "first".getBytes(StandardCharsets.UTF_8));
|
||||
var second = client.uploadPart(upload, 2, "second".getBytes(StandardCharsets.UTF_8));
|
||||
check(first.number() == 1 && "\"etag-1\"".equals(first.etag()), "part result");
|
||||
var page = client.listParts(upload, 0, 1);
|
||||
check(page.truncated() && page.nextPartNumberMarker() == 1 && page.parts().size() == 1,
|
||||
"first part page");
|
||||
page = client.listParts(upload, page.nextPartNumberMarker(), 1);
|
||||
check(!page.truncated() && page.parts().getFirst().number() == 2, "second part page");
|
||||
var uploads = client.listMultipartUploads("mybucket", "large", null, null, 10);
|
||||
check(uploads.uploads().size() == 1 && "u-1".equals(uploads.uploads().getFirst().uploadId()),
|
||||
"unfinished upload listing");
|
||||
String etag = client.completeMultipartUpload(upload, List.of(second, first));
|
||||
check("\"final-etag\"".equals(etag), "completion ETag");
|
||||
check(completion.get().indexOf("<PartNumber>1</PartNumber>") <
|
||||
completion.get().indexOf("<PartNumber>2</PartNumber>"), "completion part order");
|
||||
completionError.set(true);
|
||||
try {
|
||||
client.completeMultipartUpload(upload, List.of(first));
|
||||
throw new AssertionError("Expected embedded S3 error");
|
||||
} catch (ObjectStorageException error) {
|
||||
check("InvalidPart".equals(error.errorCode()), "HTTP 200 completion error");
|
||||
}
|
||||
completionError.set(false);
|
||||
Path file = Files.createTempFile("objectstore-client-multipart", ".bin");
|
||||
try {
|
||||
byte[] content = new byte[5 * 1024 * 1024 + 3];
|
||||
content[content.length - 1] = 42;
|
||||
Files.write(file, content);
|
||||
var fileParts = client.uploadFileParts(upload, file, 5 * 1024 * 1024);
|
||||
check(fileParts.size() == 2 && fileParts.getLast().size() == 3, "file part slicing");
|
||||
check(sizes.contains(5 * 1024 * 1024) && sizes.contains(3), "part request sizes");
|
||||
} finally { Files.deleteIfExists(file); }
|
||||
client.abortMultipartUpload(upload);
|
||||
System.out.println("Multipart client tests passed");
|
||||
} finally { server.stop(0); }
|
||||
}
|
||||
|
||||
private static void handle(HttpExchange exchange, AtomicReference<String> completion,
|
||||
AtomicBoolean completionError, List<Integer> sizes) throws IOException {
|
||||
check(exchange.getRequestHeaders().getFirst("Authorization") != null, "signed multipart request");
|
||||
String path = exchange.getRequestURI().getRawPath();
|
||||
String query = exchange.getRequestURI().getRawQuery();
|
||||
String method = exchange.getRequestMethod();
|
||||
if ("/mybucket".equals(path) && "GET".equals(method) && query.contains("uploads=")) {
|
||||
reply(exchange, 200, "<ListMultipartUploadsResult><Upload><Key>large file.bin</Key>" +
|
||||
"<UploadId>u-1</UploadId></Upload><IsTruncated>false</IsTruncated></ListMultipartUploadsResult>");
|
||||
return;
|
||||
}
|
||||
check("/mybucket/large%20file.bin".equals(path), "multipart path");
|
||||
if ("POST".equals(method) && "uploads=".equals(query)) {
|
||||
reply(exchange, 200, "<InitiateMultipartUploadResult><UploadId>u-1" +
|
||||
"</UploadId></InitiateMultipartUploadResult>");
|
||||
} else if ("PUT".equals(method) && query.contains("partNumber=")) {
|
||||
int number = query.contains("partNumber=1") ? 1 : 2;
|
||||
byte[] body = exchange.getRequestBody().readAllBytes();
|
||||
check(SigV4.hash(body).equals(exchange.getRequestHeaders().getFirst("x-amz-content-sha256")),
|
||||
"part hash");
|
||||
sizes.add(body.length);
|
||||
exchange.getResponseHeaders().set("ETag", "\"etag-" + number + "\"");
|
||||
reply(exchange, 200, "");
|
||||
} else if ("GET".equals(method) && query.contains("uploadId=")) {
|
||||
boolean first = query.contains("part-number-marker=0");
|
||||
reply(exchange, 200, "<ListPartsResult><Part><PartNumber>" + (first ? 1 : 2) +
|
||||
"</PartNumber><ETag>\"etag-" + (first ? 1 : 2) + "\"</ETag><Size>" +
|
||||
(first ? 5 : 6) + "</Size></Part><IsTruncated>" + first + "</IsTruncated>" +
|
||||
"<NextPartNumberMarker>" + (first ? 1 : 2) + "</NextPartNumberMarker></ListPartsResult>");
|
||||
} else if ("POST".equals(method) && query.contains("uploadId=")) {
|
||||
completion.set(new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8));
|
||||
reply(exchange, 200, completionError.get() ? "<Error><Code>InvalidPart</Code></Error>" :
|
||||
"<CompleteMultipartUploadResult><ETag>\"final-etag\"</ETag></CompleteMultipartUploadResult>");
|
||||
} else if ("DELETE".equals(method) && query.contains("uploadId=")) {
|
||||
reply(exchange, 204, "");
|
||||
} else throw new AssertionError("Unexpected multipart operation: " + method + " " + query);
|
||||
}
|
||||
|
||||
private static void reply(HttpExchange exchange, int status, String content) throws IOException {
|
||||
byte[] bytes = content.getBytes(StandardCharsets.UTF_8);
|
||||
if (bytes.length == 0) exchange.sendResponseHeaders(status, -1);
|
||||
else {
|
||||
exchange.sendResponseHeaders(status, bytes.length);
|
||||
exchange.getResponseBody().write(bytes);
|
||||
}
|
||||
}
|
||||
|
||||
private static void check(boolean condition, String description) {
|
||||
if (!condition) throw new AssertionError(description);
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user