Expose TLS handshake bytes from Incoming

[aiguy110]

Summary

Adds Incoming::handshake_bytes() (and a forwarding method on the high-level quinn::Incoming wrapper), which decrypts the Initial packet using the publicly-derivable Initial keys, walks the QUIC frames, and returns the concatenated CRYPTO frame contents in offset order.

Per RFC 9001 §4.1.3, TLS handshake messages travel inside CRYPTO frames with no intervening TLS record layer, so the returned bytes start directly with the TLS handshake message header (0x01 for ClientHello, then a 3-byte length, then the body). Callers can feed this to a TLS parser to inspect extensions such as SNI or ALPN before deciding whether to accept(), retry(), refuse(), or ignore() the connection.

Motivation

Incoming (from #2076) gave users the ability to make accept/refuse/retry/ignore decisions before any QUIC response goes on the wire — a very nice primitive for hardening. The decision can today be informed by remote address, address-validation status, original DCID, and local IP. What's missing is any ability to look at the ClientHello itself.

Concrete use case I bumped into: port-knocking style authentication on a UDP service. The goal is for the port to appear closed to scanners that don't carry a pre-shared token (an HMAC, say, embedded in a custom ALPN entry). Without ClientHello inspection, the closest one can do today is gate at the rustls layer via a custom ResolvesServerCert — but by that point a TLS alert has already left the box, so the service is observable. With handshake_bytes() + Incoming::ignore(), the port becomes truly silent for invalid clients (WireGuard-grade posture), while still being a normal QUIC service for valid ones. I've verified this end-to-end with packet capture: zero response packets to clients failing the check.

Other plausible uses: SNI-keyed rate limiting before TLS handshake, ClientHello-fingerprint denylists, GREASE/ECH-aware admission, eBPF-style port-knocking with cleaner state handling than out-of-band knock daemons.

API shape

impl Incoming {
    pub fn handshake_bytes(&self) -> Result<Vec<u8>, HandshakeBytesError>;
}

pub enum HandshakeBytesError {
    Decrypt,            // AEAD authentication failed
    MalformedFrames,    // frame parsing failed
    NonContiguous,      // CRYPTO frames don't run from offset 0
}

Re-exported through quinn-proto::Incoming and quinn::Incoming.

Design choices / alternatives considered

• Why return bytes instead of a parsed ClientHello? Keeps quinn-proto firmly in QUIC framing and out of the TLS layer. Callers already have rustls (or webpki-types, or whatever) available for the TLS-side parsing. Avoids growing quinn-proto's API surface with a TLS data model.
• Why a method on Incoming rather than exposing internal frame parsing? Iter/Frame/Crypto are currently pub(crate); making them public is a much larger commitment. A single inspection method is the smallest user-facing surface for the same capability.
• Why Vec<u8> and not Bytes / &[u8]? The CRYPTO frames need to be concatenated in offset order, so the result is inherently owned. Happy to switch to Bytes if that's preferred.
• Non-CRYPTO frames (PADDING, PING, ACK, CONNECTION_CLOSE) are silently skipped — they're legal in Initials per RFC 9000 §17.2.2.
• Non-contiguous CRYPTO frames return an explicit error rather than being silently reassembled with gaps. Real ClientHellos in Initials are typically a single frame at offset 0; a gap suggests either a fragmented handshake (rare for ClientHello) or something malformed.
• Cost. The method clones the (encrypted) payload buffer and does one AEAD decrypt + a frame walk. Cheap enough that callers can invoke it unconditionally; not so cheap that it should be called many times — the doc could note "cache the result if you need it more than once," happy to add that.

What's not in this PR

• No tests. Most happy to add one — an integration test that captures an Initial via the existing simulated-IO test harness and asserts the returned bytes start with 0x01 and parse as a ClientHello would be straightforward. Held off pending design feedback on the API shape.
• No changes to accept()/ignore()/etc. The method is purely an additional read-only inspection point.

Compatibility

Additive only. No existing methods change behaviour or signatures.

---

Happy to convert this to an issue / discussion if you'd prefer to align on the API shape before reviewing code. Also happy to iterate on naming (handshake_bytes vs client_hello_bytes vs initial_crypto etc.), error type granularity, or return type.

[Ralith]

This is a really cool idea! The use case makes perfect sense to me, and is something I'm happy to support. I particularly appreciate the design to yield unprocessed plaintext bytes; that's good layering.

One alternative strategy might be to have the crypto layer parse them into something that's then passed back with type erasure, like we've done for Connection::peer_identity. This could be an improvement over requiring callers to all reimplement TLS decoding, if there's a stronger type still broad enough not to exclude reasonable use cases, which isn't immediately obvious to me. I'm inclined to accept the raw bytes as a starting point, and we can always iterate later.

I do have some reservations about the frame processing strategy:

• We should handle out-of-order frames. We already have the tools in connection::Assembler, so it's strange to reject easily intelligible packets.
• I'm concerned that, for example, post-quantum handshakes might make multi-packet ClientHellos routine. I'd prefer to expose a RecvStream or something very similar here, so callers can decide for themselves how much it's useful to read. Later packets are currently queued per connection in quinn_proto::Endpoint::incoming_buffers.

[djc]

In my mind this is (mostly) a poor man's version of

• Access Client Hello before accepting connection #2024

I'd prefer to get that done holistically, including making sure that callers don't have to commit to a ServerConfig up front but can only select one on accept().