feat(python): add user header and origin timestamp support

[jiengup]

Add user headers to SendMessage and ReceiveMessage, expose origin timestamp, and fix Docker test infrastructure.

Which issue does this PR address?

Closes #3601 #3612

Rationale

The Python SDK could send and receive message payloads, but it did not expose the typed user headers already carried by the underlying Rust IggyMessage. This left Python behind the Rust, Node, Go, Java, and C# SDKs for message metadata.

What changed?

Python messages can attach user headers and read them back from received messages. SendMessage accepts user_headers and an optional custom id, while ReceiveMessage exposes user_headers() and origin_timestamp().

According to the discussion below, unlike the original proposal in #3601 (which exposed only a plain dict[str, str | bytes | bool | int | float]), the binding now supports the full Rust header type surface through two exposed classes, plus a plain-scalar convenience layer:

• HeaderKey / HeaderValue — typed complex enums covering every HeaderKind: Raw, String, Bool, Int8/16/32/64/128, UnsignedInt8/16/32/64/128, Float32, Float64.
• UserHeaders — the mapping returned by ReceiveMessage.user_headers(). It is a real dict subclass (dict[HeaderKey, HeaderValue]), so all mapping operations work, and it adds a chainable to_scalar_dict() for the convenient scalar form: message.user_headers().to_scalar_dict().

Sending (plain → Rust)

Each key/value pair is converted independently, so typed and plain forms can be freely mixed in one dict (e.g. {HeaderKey.String("a"): 7, "b": HeaderValue.Bool(True), "c": "plain"}):

• Plain scalars map by best-effort, lossless conversion for both keys and values (keys are no longer restricted to str):
  • str → String, bytes → Raw, bool → Bool
  • int → the smallest header kind that holds it exactly: non-negative → Uint8/16/32/64/128, negative → Int8/16/32/64/128; values beyond the 128-bit range raise ValueError.
  • float → Float32 when the value is exactly representable as f32, otherwise Float64 (always lossless).
• Explicit typed values are range-checked: integer variants are enforced by the binding at construction (OverflowError for out-of-range, e.g. HeaderValue.Int8(300)), and an explicit Float32 whose value overflows f32 (→ inf) is rejected with ValueError.

Receiving (Rust → plain)

Every HeaderKind maps losslessly onto a Python scalar (128-bit ints fit Python's arbitrary-precision int; f32 → f64 is exact), so:

• user_headers() returns typed UserHeaders (dict[HeaderKey, HeaderValue]), or None when the message carries no user headers.
• UserHeaders.to_scalar_dict() returns dict[str | bytes | bool | int | float, str | bytes | bool | int | float] — a direct, lossless conversion (no logging); it only raises ValueError on a genuine decode error of a stored field.

Header decode errors from known-but-invalid or unknown semantic kinds surface as ValueError. SendMessage(id=...) (mapped to u128) is part of the same binding update.

Minor Fix

The Python test compose setup starts the server with fresh default root credentials, binds HTTP/TCP/QUIC addresses explicitly, and uses iggy ping for the healthcheck instead of the HTTP stats endpoint.

TODO: make pre-commit and other scripts compatible with other shells (e.g. zsh); commands like mapfile don't work in some shells.

Discussion Notes

Choices from the issue discussion, updated to the final implementation:

• Python supports both a plain dict API for the common case and explicit HeaderKey/HeaderValue wrapper classes; the two can be mixed per entry.
• Integers are stored in the narrowest lossless kind (unsigned for non-negative, signed for negative); only values beyond ±128-bit raise ValueError.
• Floats select Float32/Float64 by exact representability instead of always Float64.
• Non-string keys are now supported (any plain scalar or HeaderKey), instead of being rejected.
• ReceiveMessage.user_headers() returns None for no headers.
• origin_timestamp() is exposed on received messages.

API Usage

Sending — plain scalars (common case)

from apache_iggy import SendMessage as Message

# Keys and values are plain Python scalars; each is converted losslessly.
Message(
    "order-created",
    user_headers={
        "content-type": "application/json",  # -> String
        "trace-blob": b"\x00\x01",           # -> Raw
        "retryable": False,                  # -> Bool
        "attempt": 3,                        # -> UnsignedInt8 (smallest fit)
        "created-at-ms": 1_700_000_000_000,  # -> UnsignedInt64
        "score": 1.25,                       # -> Float32 (exactly representable)
    },
    id=42,  # optional custom message id (u128)
)

Sending — explicit typed kinds

from apache_iggy import HeaderKey, HeaderValue
from apache_iggy import SendMessage as Message

Message(
    "order-created",
    user_headers={
        HeaderKey.String("schema-version"): HeaderValue.UnsignedInt16(1),
        HeaderKey.UnsignedInt32(7): HeaderValue.String("order-id"),  # non-string key
    },
)

Sending — mixed typed + plain in one dict

Message(
    "order-created",
    user_headers={
        HeaderKey.String("typed"): HeaderValue.Int32(-5),
        "plain-key": HeaderValue.Bool(True),
        HeaderKey.String("plain-val"): "hello",
        "both-plain": 9,
    },
)

Receiving

message = polled_messages[0]

message.origin_timestamp()          # int

headers = message.user_headers()    # UserHeaders | None
if headers is not None:
    # Typed access (full wire-type fidelity):
    value = headers.get(HeaderKey.String("schema-version"))
    if isinstance(value, HeaderValue.UnsignedInt16):
        print(value.value)          # 1

    # Convenient plain form (lossless):
    plain = headers.to_scalar_dict()      # dict[str|bytes|bool|int|float, str|bytes|bool|int|float]
    print(plain["schema-version"])  # 1

Validation / errors

Message("p", user_headers={"k": 2**128})            # ValueError: 128-bit range
Message("p", user_headers={"k": HeaderValue.Int8(300)})   # OverflowError
Message("p", user_headers={"k": HeaderValue.Float32(1e40)})  # ValueError: 32-bit float
Message("p", user_headers={object(): "v"})          # ValueError: keys must be str/bytes/bool/int/float/HeaderKey

Local Execution

• Passed: cargo fmt --check --manifest-path foreign/python/Cargo.toml
• Passed: cargo check --manifest-path foreign/python/Cargo.toml
• Passed: cargo test --manifest-path foreign/python/Cargo.toml
• Passed: uv run --extra dev ruff check tests/test_message_operations.py tests/test_consumer_group.py
• Passed: .venv/bin/python -m pytest tests/test_message_operations.py::TestMessageOperations::test_invalid_user_headers_are_rejected -q
• Successfully run: example/message-headers/typed-headers/consumer.py, example/message-headers/typed-headers/producer.py, example/message-headers/plain-headers/consumer.py, example/message-headers/plain-headers/producer.py

AI Usage

Codex was used to inspect the existing Python, Rust, Node, and Go SDK behavior, implement the Python binding changes, add tests. All the modification was reviewed carefully by the human.

[hubcio]

one additional request from my side: please update python examples to include a code that would show how to use iggy message user headers. there is one for rust already.

[codecov]

Codecov Report

❌ Patch coverage is 81.31868% with 102 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.50%. Comparing base (d9635b6) to head (6d9c1ff).
⚠️ Report is 14 commits behind head on master.

Files with missing lines	Patch %	Lines
foreign/python/src/user_headers.rs	79.76%	102 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3613      +/-   ##
============================================
- Coverage     74.53%   74.50%   -0.04%     
  Complexity      969      969              
============================================
  Files          1306     1307       +1     
  Lines        150028   150564     +536     
  Branches     125462   125536      +74     
============================================
+ Hits         111829   112182     +353     
- Misses        34700    34790      +90     
- Partials       3499     3592      +93     

Components	Coverage Δ
Rust Core	74.83% <ø> (-0.01%)	⬇️
Java SDK	62.71% <ø> (ø)
C# SDK	71.13% <ø> (-1.14%)	⬇️
Python SDK	88.44% <81.31%> (-3.83%)	⬇️
PHP SDK	84.52% <ø> (ø)
Node SDK	92.24% <ø> (ø)
Go SDK	43.08% <ø> (ø)

Files with missing lines	Coverage Δ
foreign/python/src/lib.rs	100.00% <100.00%> (ø)
foreign/python/src/receive_message.rs	88.37% <100.00%> (+3.99%)	⬆️
foreign/python/src/send_message.rs	97.95% <100.00%> (+1.18%)	⬆️
foreign/python/src/user_headers.rs	79.76% <79.76%> (ø)

... and 44 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jiengup]

one additional request from my side: please update python examples to include a code that would show how to use iggy message user headers. there is one for rust already.

That will be included when the PR is ready.

[slbotbm]

@hubcio @jiengup

I was thinking that maybe we should change the signature of the headers from dict[str, str | bytes | bool | int | float] | None to dict[str, str | bytes | bool | int | float] | dict[HeaderKey, HeaderValue] | None, where HeaderKey and HeaderValue are python classes mirroring rust and expose subclasses like HeaderKey.String(value) and HeaderValue.UnsignedInt128(value), etc..

This would allow us to have two advantages: users that do not have much experience/do not require explicit control can use dict[str, str | bytes | bool | int | float], while users that require precise control on the types of headers can use dict[HeaderKey, HeaderValue]. Also, by providing dict[HeaderKey, HeaderValue] as an option, we allow AI Agents to make less mistakes as compared to dict[str, str | bytes | bool | int | float], which is a little vague from the agents' point of view.

What do you guys think?

[jiengup]

change the signature of the headers from dict[str, str | bytes | bool | int | float] | None to dict[str, str | bytes | bool | int | float] | dict[HeaderKey, HeaderValue] | None

I think it's a good idea and well aligned with the first point worth discussing in my PR. I'll push a new patch soon.

[hubcio]

@slbotbm for me this sounds fine, however one thing comes to my mind: will it have any impact on performance?

[jiengup]

@slbotbm for me this sounds fine, however one thing comes to my mind: will it have any impact on performance?

Actually, I think additional typed HeaderKey support will not introduce more overhead compared to the current implementation.

The only overhead we introduce is the iteration of user_header and the transformation of Rust and Python types, which need to be considered if we have a large number of user header pairs in some user cases.

[hubcio]

alright, then don't worry about it now.

[jiengup]

@hubcio @slbotbm

Hi, I just reflected the change and merged all the commits into a final one. The PR message is also updated.

Instead of a plain-only dict[str, ...] with int → Int64 / float → Float64 and string-only keys, the binding now exposes the full typed HeaderKey/HeaderValue surface with smallest-fit kind selection, supports non-string keys, and allows typed and plain pairs to be freely mixed in one dict.

Please have a look at it when you are free.

[slbotbm]

first round of review

[slbotbm]

/author

[jiengup]

/ready

[slbotbm]

Some more comments:

• You have set the type of user headers in some places to [Any, Any], which makes any type checker incapable of catching mistakes. Would be better to explicitly declare the types.

[jiengup]

/ready

[slbotbm]

Right now, since UserHeaders is a
publicly constructible and mutable dict, this is possible:

UserHeaders({object(): object()}).to_scalar_dict()

The resulting dictionary can contain arbitrary objects despite the documented
and generated scalar-only return type.

Validate entries during construction and mutation, or reject unsupported
objects inside to_scalar_dict(). Add tests for unsupported keys and values
inserted through both the constructor and normal dictionary mutation.

[slbotbm]

This accepts plain float("inf") and stores it as Float32, because narrowing infinity to f32 still compares equal after widening. In contrast, checked_float32 reject the equivalent typed HeaderValue.Float32(float("inf")).

This also means a Float32 infinity or NaN received from a Rust client can be
represented by Python but cannot be sent again. Float64 and plain NaN follow
different paths again.

Let's reject non-finite headers everywhere in plain, Float32, and Float64 paths.

Add tests for positive and negative infinity and NaN across plain, typed, and
receive-then-resend paths.

[slbotbm]

The linked issue notes that structurally malformed wire headers are logged and
returned as None, making them indistinguishable from absent headers.

Document the None behavior and distinguish it from known semantic decode
errors, which become ValueError.