Skip to content

feat: add write builder configuration#367

Open
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:write-builder-config
Open

feat: add write builder configuration#367
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:write-builder-config

Conversation

@QuakeWang

Copy link
Copy Markdown
Contributor

Purpose

Part of #232.

The current write path configures some writer behavior after creating TableWrite, while TableCommit is created independently from the same WriteBuilder. This makes builder-level write configuration incomplete and prevents writers and committers from sharing an explicit commit user.

Brief change log

  • Add WriteBuilder::with_commit_user so writers and committers created from the same builder share the same commit user.
  • Validate custom commit users as safe file name segments, because the value is persisted in snapshot metadata, used for duplicate commit detection, and embedded in postpone-bucket data file name prefixes.
  • Add WriteBuilder::with_overwrite and route DataFusion overwrite writers through the builder instead of mutating TableWrite after creation.
  • Add focused tests for commit user sharing, invalid commit user rejection, and overwrite-aware writer creation.

Tests

  • cargo test -p paimon table::write_builder
  • cargo test -p paimon table::table_write
  • cargo test -p paimon table::table_commit
  • cargo test -p paimon-datafusion table::tests::test_insert_overwrite

API and Format

Adds public WriteBuilder APIs: commit_user, with_commit_user, and with_overwrite.

No storage format change.

Documentation

Public API doc comments document the commit_user semantics and validation requirements.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant