feat(store): implement pluggable cloud storage for HStore

[vaijosh]

Purpose

• Closes: [Feature] Implement Cloud Storage Support for HugeGraph #3079

This PR introduces a cloud storage architecture for HugeGraph Store (HStore). In cloud-native environments, storage nodes are ephemeral, making reliance on local disk storage a single point of failure. This implementation decouples the storage layer from local disk dependencies, allowing SST files to be offloaded to durable, scalable cloud storage providers.

Key Changes

Pluggable Architecture

• Created the CloudStorageProvider SPI interface to enable extensible storage backends.
• Added CloudStorageProviderFactory for seamless provider discovery and lifecycle management.

S3 Provider Implementation

• Introduced the hg-store-cloud-s3 module, leveraging AWS SDK v2 for production-ready S3/S3 compatible storage interaction.

Lifecycle & Event Integration

• Integrated CloudStorageEventListener with RocksDBFactory. This hooks into critical SST lifecycle events (onTableFileCreated, onTableFileDeleted) to ensure synchronized state between local RocksDB and cloud storage.
• Implemented startup hydration (onDBCreated) to backfill pre-existing files and read-miss on-demand hydration to ensure data availability.

Infrastructure & Testing

• Added a comprehensive local development environment via docker/cloud-storage/ using MinIO.
• Included an integration test script (test-graph-queries-and-sst.sh) to verify end-to-end data durability and query consistency.

Implementation Highlights

• SPI Integration: Standardized service discovery via META-INF/services.
• RocksDB Hooking: Registered a singleton RocksdbEventListener within RocksDBFactory to intercept file operations without modifying core RocksDB logic.
• Read-Miss Protection: Added a guard window mechanism in onReadMiss to prevent redundant hydration requests during high-concurrency read scenarios.

Verifying These Changes

• Unit Tests: Coverage for config parsing (CloudStorageConfigTest), factory registration, and event listener logic.
• Integration Tests: Verified using the new docker-compose setup with MinIO.
• Configuration: Verified application.yml bindings for credentials, bucket management, and sync intervals.

Impact

• Dependencies: Added AWS SDK v2; updated LICENSE and NOTICE accordingly.
• Configurations: Added cloud.storage namespace to application.yml (disabled by default).
• Public API: No breaking changes to existing public-facing client APIs.

Documentation

• Architecture and configuration details are documented in hugegraph-store/docs/pluggable-cloud-storage-architecture.md.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 107 lines in your changes missing coverage. Please review.
✅ Project coverage is 1.53%. Comparing base (b9710a7) to head (b7fcb28).

Files with missing lines	Patch %	Lines
...graph/store/cloud/CloudStorageProviderFactory.java	0.00%	64 Missing ⚠️
...ache/hugegraph/store/cloud/CloudStorageConfig.java	0.00%	15 Missing ⚠️
...he/hugegraph/store/cloud/CloudStorageProvider.java	0.00%	13 Missing ⚠️
.../hugegraph/store/business/BusinessHandlerImpl.java	0.00%	13 Missing ⚠️
...store/cloud/CloudStorageNonRetryableException.java	0.00%	2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (b9710a7) and HEAD (b7fcb28). Click for more details.

HEAD has 2 uploads less than BASE

Flag	BASE (b9710a7)	HEAD (b7fcb28)
	3	1

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #3081       +/-   ##
============================================
- Coverage     39.19%   1.53%   -37.66%     
+ Complexity      264      21      -243     
============================================
  Files           770     752       -18     
  Lines         65779   63366     -2413     
  Branches       8726    8288      -438     
============================================
- Hits          25779     975    -24804     
- Misses        37247   62307    +25060     
+ Partials       2753      84     -2669     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[VGalaxies]

Review summary

• Blocking: yes
• Summary: The change cannot currently guarantee recoverable RocksDB state, may silently disable cloud storage, and contains S3 retry and deployment defects.
• Evidence:
  • Static analysis of git diff origin/master...HEAD
  • shell syntax and POM XML checks passed
  • git diff --check reported whitespace-only errors

[VGalaxies]

🚨 Review summary

Important

The cloud recovery path has blocking consistency, configuration, deletion-safety, release-metadata, and verification gaps.

📊 Risk dashboard

Signal	Result
🚦 Review gate	Blocked
🔎 Actionable findings	9 (8 High, 1 Medium)
🧪 Verification coverage	5 checks
⚠️ Confirmed / limited checks	2 / 1

🔬 Coverage details

⚠️ S3 failure semantics — Confirmed

Trace: S3CloudStorageProvider upload, download, delete, and exception-classification paths, CloudStorageEventListener purge and hydration flows, AWS SDK 2.33.8 retry-code and file-transformer behavior

Conclusion: Confirmed ignored per-object deletion errors, retryable error-code misclassification, and non-atomic download recovery risks; lower-impact lifecycle leads were dropped.

⚠️ Release and E2E integrity — Confirmed

Trace: install-dist release LICENSE and known-dependencies inventory, cloud-storage Docker artifact build and recovery workflow, Maven provider packaging and test activation

Conclusion: Confirmed the release dependency inventory mismatch and E2E false-green paths caused by suppressed build and wipe failures.

🟡 Focused unit suites — Limited

Trace: hg-store-common cloud configuration, provider, and factory tests, hg-store-cloud-s3 exception-classification tests, hg-store-node listener, retry, metrics, tracker, configuration, and callback tests, hg-store-core BusinessHandlerImplTest

Conclusion: Surefire reports recorded 184 tests with 0 failures, 0 errors, and 0 skipped. The suites do not cover concurrent metadata publication, comma-separated data roots, or partial S3 batch deletion.

✅ Maven reactor — Clear

Trace: Full 44-project reactor with the cloud-s3 profile, hugegraph-store module ordering and dependency graph

Conclusion: Maven validate completed successfully for all 44 reactor projects; the suspected duplicate cloud-s3 module activation did not reproduce.

✅ Shell syntax — Clear

Trace: pd-entrypoint.sh, store-entrypoint.sh, test-graph-queries-and-sst.sh

Conclusion: bash -n returned successfully for all three changed shell entrypoints and workflows.

Warning

Verification limits

• No live Docker and MinIO recovery drill was run, so cluster recovery and purge behavior were not exercised end to end.
• The test environment used Java 17; the legacy JaCoCo agent emitted unsupported class-version instrumentation errors, so coverage instrumentation was not verified even though test assertions passed.
• S3SingleLargeFileE2ETest was intentionally not run because its default path generates a 20 GiB file.
• The independent recovery-consistency run timed out after 900 seconds and produced no usable result.

---

_{🤖 Codex review · GPT-5.6 Sol · effort: xhigh}

[vaijosh]

🚨 Review summary

Important

The cloud recovery path has blocking consistency, configuration, deletion-safety, release-metadata, and verification gaps.

📊 Risk dashboard

Signal Result
🚦 Review gate Blocked
🔎 Actionable findings 9 (8 High, 1 Medium)
🧪 Verification coverage 5 checks
⚠️ Confirmed / limited checks 2 / 1

🔬 Coverage details

⚠️ S3 failure semantics — Confirmed
Trace: S3CloudStorageProvider upload, download, delete, and exception-classification paths, CloudStorageEventListener purge and hydration flows, AWS SDK 2.33.8 retry-code and file-transformer behavior

Conclusion: Confirmed ignored per-object deletion errors, retryable error-code misclassification, and non-atomic download recovery risks; lower-impact lifecycle leads were dropped.

⚠️ Release and E2E integrity — Confirmed
Trace: install-dist release LICENSE and known-dependencies inventory, cloud-storage Docker artifact build and recovery workflow, Maven provider packaging and test activation

Conclusion: Confirmed the release dependency inventory mismatch and E2E false-green paths caused by suppressed build and wipe failures.

🟡 Focused unit suites — Limited
Trace: hg-store-common cloud configuration, provider, and factory tests, hg-store-cloud-s3 exception-classification tests, hg-store-node listener, retry, metrics, tracker, configuration, and callback tests, hg-store-core BusinessHandlerImplTest

Conclusion: Surefire reports recorded 184 tests with 0 failures, 0 errors, and 0 skipped. The suites do not cover concurrent metadata publication, comma-separated data roots, or partial S3 batch deletion.

✅ Maven reactor — Clear
Trace: Full 44-project reactor with the cloud-s3 profile, hugegraph-store module ordering and dependency graph

Conclusion: Maven validate completed successfully for all 44 reactor projects; the suspected duplicate cloud-s3 module activation did not reproduce.

✅ Shell syntax — Clear
Trace: pd-entrypoint.sh, store-entrypoint.sh, test-graph-queries-and-sst.sh

Conclusion: bash -n returned successfully for all three changed shell entrypoints and workflows.

Warning

Verification limits

• No live Docker and MinIO recovery drill was run, so cluster recovery and purge behavior were not exercised end to end.
• The test environment used Java 17; the legacy JaCoCo agent emitted unsupported class-version instrumentation errors, so coverage instrumentation was not verified even though test assertions passed.
• S3SingleLargeFileE2ETest was intentionally not run because its default path generates a 20 GiB file.
• The independent recovery-consistency run timed out after 900 seconds and produced no usable result.

🤖 Codex review · GPT-5.6 Sol · effort: xhigh

Hi @VGalaxies.
Thanks for reviewing the changes. I have addressed all the review comments. Can you please review it again?

[vaijosh]

@VGalaxies
I am not sure why code coverage check is failing.
We have
The Diff has 94.79% coverage (agains target 42.32%) [codecov/patch](https://github.com/apache/hugegraph/pull/3081/checks?check_run_id=89150435397)Successful in 1s — 94.79% of diff hit (target 42.32%)

Not sure why the coverage is suddenly dropped at project level to 30.60% (-11.72%) compared to e960cc5 codecov/projectFailing after 1s — 30.60% (-11.72%) compared to e960cc5

[Copilot]

Pull request overview

Implements a pluggable cloud storage layer for HugeGraph Store (HStore), integrating with RocksDB lifecycle events to offload/hydrate SSTs via an SPI-based provider model (with an initial S3 provider), plus local Docker/MinIO tooling and expanded test coverage.

Changes:

• Introduces the cloud storage SPI (CloudStorageProvider, CloudStorageProviderFactory) and config model, plus S3 provider module (hg-store-cloud-s3).
• Hooks cloud lifecycle into RocksDB open/truncate and read-miss hydration paths, and adds metrics + tracking utilities for durability/sync observability.
• Adds local dev/integration environment under docker/cloud-storage/ and updates release docs (LICENSE/NOTICE/known-dependencies) for new third-party deps.

Reviewed changes

Copilot reviewed 56 out of 58 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
pom.xml	Excludes cloud-storage runtime artifacts/generated dirs from checks.
install-dist/scripts/dependency/regenerate_known_dependencies.sh	Adds helper script to regenerate known runtime dependencies list.
install-dist/scripts/dependency/known-dependencies.txt	Updates runtime dependency inventory for new cloud/S3 deps.
install-dist/release-docs/NOTICE	Adds AWS SDK v2 notice block.
install-dist/release-docs/licenses/LICENSE-reactive-streams-1.0.4.txt	Adds MIT-0 license text for reactive-streams.
install-dist/release-docs/LICENSE	Updates third-party license inventory (AWS SDK v2, Netty, etc.).
hugegraph-store/pom.xml	Adds hg-store-cloud-s3 module and a build profile stanza.
hugegraph-store/hg-store-rocksdb/src/test/java/org/apache/hugegraph/rocksdb/access/RocksDBSessionTest.java	Test updates/imports related to new RocksDB behaviors.
hugegraph-store/hg-store-rocksdb/src/test/java/org/apache/hugegraph/rocksdb/access/RocksDBFactoryTest.java	Adds tests for metadata checkpoint flushing + hydration/open ordering + retryability.
hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/SessionOperatorImpl.java	Adds read-miss hook to trigger cloud hydration and retry reads.
hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/RocksDBSession.java	Adds pre-open hydration hook, listener registration, truncate notifications, metadata checkpoint capture.
hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/metrics/CloudStorageMetricsTest.java	Adds unit tests for cloud storage metrics registration/lifecycle.
hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/CloudSyncTrackerTest.java	Adds unit tests for SST confirmation tracking bitmap logic.
hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/CloudRecoveryIntegrationTest.java	Adds end-to-end recovery + JNI boundary failure integration tests with in-memory provider.
hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/AppConfigCloudStorageTest.java	Adds Spring binding + stable cloud-scope resolution tests.
hugegraph-store/hg-store-node/src/main/resources/application.yml	Adds default cloud.storage config (disabled by default) for node.
hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/FailedUploadTask.java	Adds DLQ task model for failed uploads with epoch safety.
hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudSyncTracker.java	Adds confirmed-SST tracking + epoch-based stale callback protection.
hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudStorageMetricsConst.java	Adds metric name/tag constants for cloud storage monitoring.
hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudStorageMetrics.java	Adds Micrometer metric registration and per-DB meter cleanup helpers.
hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/AppConfig.java	Wires cloud storage initialization/shutdown, metrics binding, stable scope persistence.
hugegraph-store/hg-store-node/pom.xml	Adds cloud S3 module + RoaringBitmap dependency.
hugegraph-store/hg-store-dist/src/assembly/static/conf/application.yml	Documents and adds full cloud.storage configuration block.
hugegraph-store/hg-store-core/src/test/java/org/apache/hugegraph/store/business/BusinessHandlerImplTest.java	Adds/merges tests (including truncate callback expectations).
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/business/BusinessHandlerImpl.java	Fires truncate callbacks around key-range delete.
hugegraph-store/hg-store-common/src/test/resources/META-INF/services/org.apache.hugegraph.store.cloud.CloudStorageProvider	Adds test SPI registration for discovery/duplicate-name testing.
hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/TestDuplicateNamedProvider.java	Adds test-only provider for SPI duplicate-name behavior.
hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/CloudStorageProviderTest.java	Adds tests for provider default method behaviors (list/deletePrefix).
hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/CloudStorageProviderFactoryTest.java	Adds tests for factory init/disable/swap behavior and SPI discovery paths.
hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/CloudStorageNonRetryableExceptionTest.java	Adds tests for non-retryable exception type.
hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageProviderFactory.java	Introduces SPI discovery + provider lifecycle management.
hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageProvider.java	Introduces provider SPI interface with default bulk-delete helpers.
hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageNonRetryableException.java	Adds marker exception for non-retryable cloud failures.
hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageConfig.java	Adds cloud storage configuration POJO and common knobs.
hugegraph-store/hg-store-cloud-s3/src/test/java/org/apache/hugegraph/store/cloud/s3/S3CloudStorageProviderTest.java	Adds S3 provider unit tests (exception classification, pagination safety, lifecycle).
hugegraph-store/hg-store-cloud-s3/src/main/resources/META-INF/services/org.apache.hugegraph.store.cloud.CloudStorageProvider	Registers S3 provider via SPI.
hugegraph-store/hg-store-cloud-s3/src/main/java/org/apache/hugegraph/store/cloud/s3/S3CloudStorageConfig.java	Adds S3 provider config POJO and keys/defaults.
hugegraph-store/hg-store-cloud-s3/pom.xml	Adds S3 provider module deps (AWS SDK v2) and test configuration.
docker/README.md	Documents selecting an alternate Store Dockerfile via env var.
docker/docker-compose.dev.yml	Makes Store Dockerfile configurable via HG_STORE_DOCKERFILE.
docker/cloud-storage/images/store.Dockerfile	Adds cloud-storage local Store image build for MinIO/dev.
docker/cloud-storage/images/pd.Dockerfile	Adds cloud-storage local PD image build.
docker/cloud-storage/entrypoints/store-entrypoint.sh	Adds Store entrypoint wiring Spring JSON + cloud env vars.
docker/cloud-storage/entrypoints/pd-entrypoint.sh	Adds PD entrypoint wiring Spring JSON env vars.
docker/cloud-storage/docker-compose.yml	Adds MinIO + PD + 3 Store nodes + Server local integration stack.
docker/cloud-storage/.gitignore	Ignores local artifacts/logs/data under cloud-storage dev setup.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.