feat: Add task result chunking support (issue #3071)

[cyforkk]

Summary

This PR adds support for chunking large task results to avoid memory overflow.

Changes

• Added TASK_RESULT_CHUNK_SIZE config option to CoreOptions.java (default 1MB)
• Added chunkKey() and isChunkedProperty() helper methods to P class
• Implemented chunked write in asArray() method
• Added RESULT_CHUNK_COUNT constant

Usage

When the compressed result exceeds task.result_chunk_size, it will be split into
multiple properties (~task_result_0, ~task_result_1, ...).

Set task.result_chunk_size=0 to disable chunking (preserves original behavior).

Related

Part of issue #3071 (Chunk 1-4)

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR aims to add configurable chunking for large Gremlin task results in hugegraph-core, splitting compressed task result blobs into smaller pieces to avoid exceeding per-property storage limits.

Changes:

• Introduces task.result_chunk_size (CoreOptions.TASK_RESULT_CHUNK_SIZE) with a 1MB default.
• Updates HugeTask.asArray() to optionally split compressed ~task_result into chunk properties and write a chunk-count marker.
• Adds helper constants/methods in HugeTask.P for chunk key naming and identification.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/task/HugeTask.java	Adds chunked result property writing and chunk-related helpers/constants.
hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/config/CoreOptions.java	Adds the task.result_chunk_size configuration option.

Comments suppressed due to low confidence (2)

hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/task/HugeTask.java:590

• checkPropertySize(chunk.length, P.chunkKey(i)) uses the default limit BytesBuffer.MAX_PROPERTIES (65,535) for non-RESULT keys, so with the default 1MB chunk size this will always throw LimitExceedException and chunking can never succeed. Validate against BytesBuffer.BYTES_LEN_MAX (or rely on the config range) instead of checkPropertySize() here.

                    byte[] chunk = new byte[end - start];
                    System.arraycopy(bytes, start, chunk, 0, end - start);
                    checkPropertySize(chunk.length, P.chunkKey(i));
                    list.add(P.chunkKey(i));
                    list.add(chunk);

hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/task/HugeTask.java:594

• Writing chunked results as ~task_result_0, ~task_result_1, … plus ~task_result_chunk_count will break both persistence and loading: (1) TaskTransaction.initProperties() only creates the ~task_result property key for the task vertex label, so these new keys are not in schema; (2) HugeTask.property() only handles P.RESULT and throws AssertionError for unknown keys, so fromVertex(..., withResult=true) will fail when it encounters the chunk keys/count. This needs a schema/model change (e.g., store chunks under a single multi-valued property or add explicit support for these keys in schema + deserialization).

                    list.add(P.chunkKey(i));
                    list.add(chunk);
                }
                // Write chunk count
                list.add(P.RESULT_CHUNK_COUNT);
                list.add(String.valueOf(numChunks));

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[imbajin]

Blocking: yes. Summary: The chunking implementation does not compile and is not wired into the task schema, read path, or distributed result-storage path. Evidence: exact-head CI compilation errors and static traces through HugeTask, TaskTransaction, StandardHugeGraph, and TaskAndResultScheduler.

[imbajin]

‼️ This is not the shared persisted-result path. TaskAndResultScheduler.save() stores task metadata through asArrayWithoutResult() and writes the result through unchanged HugeTaskResult.asArray(), so HStore/distributed deployments still persist one BLOB; HugeTask.set() also enforces the total result limit before this branch runs. Please center chunking in the actual result-storage abstraction and cover both local and distributed persistence, including old-format compatibility.