From 40623d2ae09dd6dc849d44cddee1af68c7215ca4 Mon Sep 17 00:00:00 2001 From: "Evgeny @ SimpleX Chat" <259188159+evgeny-simplex@users.noreply.github.com> Date: Sat, 11 Jul 2026 16:39:15 +0000 Subject: [PATCH 1/6] rfc: agent support for PRC pattern without duplex connection --- .../2026-07-11-service-rpc-implementation.md | 229 ++++++++++++++++++ rfcs/2026-07-11-service-rpc.md | 175 +++++++++++++ 2 files changed, 404 insertions(+) create mode 100644 plans/2026-07-11-service-rpc-implementation.md create mode 100644 rfcs/2026-07-11-service-rpc.md diff --git a/plans/2026-07-11-service-rpc-implementation.md b/plans/2026-07-11-service-rpc-implementation.md new file mode 100644 index 000000000..e7fa0fd75 --- /dev/null +++ b/plans/2026-07-11-service-rpc-implementation.md @@ -0,0 +1,229 @@ +# Service RPC implementation plan + +RFC: [../rfcs/2026-07-11-service-rpc.md](../rfcs/2026-07-11-service-rpc.md) + +Types and instances below must encode exactly the ABNF in the RFC. Constructor and event names are provisional. + +## Versions + +- `VersionSMPA` (agent protocol): new version for `AgentRequest`/`AgentResponse` envelopes and the service key bundle in link data. +- `VersionSMPC` (SMP client): new version for the hybrid public header. +- `VersionSMP` (SMP protocol): new version for the `SSND` command. + +## Address type - `Simplex.Messaging.Agent.Protocol` + +```haskell +data ContactConnType = CCTContact | CCTChannel | CCTGroup | CCTRelay | CCTService + +ctTypeChar CCTService = 'S' -- 's' in links: link encoding lowercases, parsing uppercases + +ctTypeP 'S' = pure CCTService +``` + +## Service key bundle - `Simplex.Messaging.Agent.Protocol` + +```haskell +data ServiceKeyBundle = ServiceKeyBundle + { keyId :: ByteString, + reqDhKey :: C.PublicKeyX25519, + reqKemKey :: KEMPublicKey + } + +instance Encoding ServiceKeyBundle where + smpEncode ServiceKeyBundle {keyId, reqDhKey, reqKemKey} = smpEncode (keyId, reqDhKey, reqKemKey) + smpP = do + (keyId, reqDhKey, reqKemKey) <- smpP + pure ServiceKeyBundle {keyId, reqDhKey, reqKemKey} +``` + +`UserContactData` gains a field, appended to the encoding (earlier versions skip it, its absence parses as `Nothing`): + +```haskell +data UserContactData = UserContactData + { direct :: Bool, + owners :: [OwnerAuth], + relays :: [ConnShortLink 'CMContact], + userData :: UserLinkData, + serviceKeys :: Maybe ServiceKeyBundle + } + +instance Encoding UserContactData where + smpEncode UserContactData {direct, owners, relays, userData, serviceKeys} = + B.concat [smpEncode direct, smpEncodeList owners, smpEncodeList relays, smpEncode userData, smpEncode serviceKeys] + smpP = do + direct <- smpP + owners <- smpListP + relays <- smpListP + userData <- smpP + serviceKeys <- fromMaybe Nothing <$> optional smpP -- absent in data from earlier versions + _ <- A.takeByteString -- ignoring tail for forward compatibility + pure UserContactData {direct, owners, relays, userData, serviceKeys} +``` + +The service address record stores current and previous private key bundles with rotation timestamps; expired bundles are deleted. + +## SSND command - `Simplex.Messaging.Protocol`, `Simplex.Messaging.Server` + +```haskell +-- Protocol.hs +SSND :: SndPublicAuthKey -> MsgFlags -> MsgBody -> Command Sender + +-- CommandTag, encoding "SSND" +SSND_ :: CommandTag Sender + +-- encodeProtocol +SSND k flags msg -> e (SSND_, ' ', k, ' ', flags, ' ', Tail msg) + +-- protocolP, gated by VersionSMP +SSND_ -> SSND <$> (smpP <* A.space) <*> (smpP <* A.space) <*> (unTail <$> smpP) +``` + +`checkCredentials`: as `SKEY` - authorization required, sender entity ID. Server verification: `vc SSender (SSND k _ _) = verifySecure k` - authorized by the key it sets. + +Server processing: `checkMode QMMessaging`, then `secureQueue_` (existing, idempotent), then deliver with deduplication. The queue store keeps the hash of the unacknowledged message delivered by `SSND`; a repeated `SSND` with an equal body hash responds `OK` without storing; the hash is dropped when the message is acknowledged. + +## Hybrid public header - `Simplex.Messaging.Protocol` + +```haskell +data PubHeader + = PubHeader + { phVersion :: VersionSMPC, + phE2ePubKey :: Maybe C.PublicKeyX25519 + } + | PubHeaderHybrid + { phVersion :: VersionSMPC, + phKeyId :: Maybe ByteString, -- present in requests, absent in replies + phDhKey :: C.PublicKeyX25519, + phKemCt :: KEMCiphertext + } + +instance Encoding PubHeader where + smpEncode = \case + PubHeader v k_ -> smpEncode (v, k_) -- Maybe encodes as '0' / '1' key, as today + PubHeaderHybrid v kId_ k ct -> smpEncode (v, '2', kId_, k, ct) + smpP = do + v <- smpP + A.anyChar >>= \case + '0' -> pure $ PubHeader v Nothing + '1' -> PubHeader v . Just <$> smpP + '2' -> PubHeaderHybrid v <$> smpP <*> smpP <*> smpP + _ -> fail "bad PubHeader" +``` + +Secret derivation and encryption (`Simplex.Messaging.Crypto`, used from `Simplex.Messaging.Agent.Client`): + +```haskell +hybridSecret :: C.DhSecretX25519 -> KEMSharedKey -> C.SbKey +hybridSecret dh (KEMSharedKey k) = C.unsafeSbKey $ C.hkdf "" (C.dhBytes' dh <> BA.convert k) "SimpleXSMPQueue" 32 +``` + +The body is encrypted with `C.sbEncrypt` (secret_box) instead of `C.cbEncrypt`, padded to the same lengths. Sending variant of `agentCbEncryptOnce`: generate ephemeral X25519 pair, encapsulate to the published KEM key, encode `PubHeaderHybrid`. Receiving: select private keys by `phKeyId` (requests) or use the reply queue keys from the request (replies); unknown key ID produces the error that makes the client re-fetch link data. + +## Agent envelopes - `Simplex.Messaging.Agent.Protocol` + +```haskell +data AgentMsgEnvelope + = ... -- existing constructors + | AgentRequest + { agentVersion :: VersionSMPA, + replyQueue :: RequestReplyQueue, + requestBody :: ByteString + } + | AgentResponse + { agentVersion :: VersionSMPA, + signature :: C.Signature 'C.Ed25519, -- root key signature of signedReply + signedReply :: ByteString -- encoded SignedReply, parsed after signature verification + } + +data RequestReplyQueue = RequestReplyQueue + { smpClientVersion :: VersionSMPC, + smpServer :: SMPServer, + senderId :: SMP.SenderId, + dhPublicKey :: C.PublicKeyX25519, -- fresh per request + kemPublicKey :: KEMPublicKey, -- fresh per request + sndAuthKey :: SndPublicAuthKey -- to secure the reply queue + } + +data SignedReply = SignedReply + { requestHash :: ByteString, -- SHA3-256 of the request message body + prevMsgHash :: ByteString, -- SHA3-256 of the previous reply envelope, empty in the first reply + final :: Bool, + replyBody :: ByteString + } +``` + +```haskell +instance Encoding AgentMsgEnvelope where + smpEncode = \case + ... -- existing constructors + AgentRequest {agentVersion, replyQueue, requestBody} -> + smpEncode (agentVersion, 'Q', replyQueue, Tail requestBody) + AgentResponse {agentVersion, signature, signedReply} -> + smpEncode (agentVersion, 'P', signature, Tail signedReply) + smpP = do + agentVersion <- smpP + smpP >>= \case + ... -- existing constructors + 'Q' -> do + (replyQueue, Tail requestBody) <- smpP + pure AgentRequest {agentVersion, replyQueue, requestBody} + 'P' -> do + (signature, Tail signedReply) <- smpP + pure AgentResponse {agentVersion, signature, signedReply} + +instance Encoding RequestReplyQueue where + smpEncode RequestReplyQueue {smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey, sndAuthKey} = + smpEncode (smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey, sndAuthKey) + smpP = do + (smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey, sndAuthKey) <- smpP + pure RequestReplyQueue {smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey, sndAuthKey} + +instance Encoding SignedReply where + smpEncode SignedReply {requestHash, prevMsgHash, final, replyBody} = + smpEncode (requestHash, prevMsgHash, final, Tail replyBody) + smpP = do + (requestHash, prevMsgHash, final, Tail replyBody) <- smpP + pure SignedReply {requestHash, prevMsgHash, final, replyBody} +``` + +Signing and verification follow the link data pattern (`Simplex.Messaging.Crypto.ShortLink`): + +```haskell +-- service +mkAgentResponse :: C.PrivateKeyEd25519 -> VersionSMPA -> SignedReply -> AgentMsgEnvelope +mkAgentResponse rootPrivKey agentVersion reply = + let signedReply = smpEncode reply + in AgentResponse {agentVersion, signature = C.sign' rootPrivKey signedReply, signedReply} + +-- client +verifyAgentResponse :: C.PublicKeyEd25519 -> AgentMsgEnvelope -> Either AgentErrorType SignedReply +verifyAgentResponse rootKey AgentResponse {signature, signedReply} + | C.verify' rootKey signature signedReply = parse smpP (AGENT A_MESSAGE) signedReply + | otherwise = Left ... -- verification error +``` + +## Agent processing - `Simplex.Messaging.Agent`, `Simplex.Messaging.Agent.Client` + +Client side: + +- `sendServiceRequest`: read link data (cached or `LGET`, proxied per config), create the reply queue (`NEW`, `QRMessaging`, subscribe), generate fresh X25519 + KEM keys and sender auth key, build and store the request message, send (proxied per config), return request ID. +- Retries re-send the stored byte-identical message. Deadline, cancellation and `final` delete the reply queue and the request record. +- Reply queue messages in `processSMPTransmissions`: decrypt with the stored hybrid secret (first reply establishes it from the hybrid header), verify with `verifyAgentResponse` and the hash chain, deliver as events; failures are acknowledged and dropped. +- New events: reply received (request ID, final flag, body), request failed (request ID, error). + +Service side: + +- Address queue messages with `AgentRequest`: compute request hash, look up the dedup store; on hit re-send stored replies (or the expiry error); on miss deliver a request event to the service application. +- Send replies: secure the reply queue and send the first reply with `SSND`, subsequent replies with `SEND`; store sent replies under the request hash until the window expires. + +Database: + +- Client: in-flight requests - request ID, stored request message, reply queue, hybrid secret, deadline, last reply hash. +- Service: dedup store - request hash, sent replies, expiry. + +## Phases + +1. SMP protocol: `SSND`, hybrid public header, version bumps, server dedup marker. +2. Agent: address type, key bundle, envelopes, client API and reply processing. +3. Agent: service-side dedup store and request events. +4. Adoption: `SSND` in the fast connection handshake; hybrid scheme for invitations and confirmations. diff --git a/rfcs/2026-07-11-service-rpc.md b/rfcs/2026-07-11-service-rpc.md new file mode 100644 index 000000000..b29e132c0 --- /dev/null +++ b/rfcs/2026-07-11-service-rpc.md @@ -0,0 +1,175 @@ +# One-off requests to service addresses + +Implementation plan: [../plans/2026-07-11-service-rpc-implementation.md](../plans/2026-07-11-service-rpc-implementation.md) + +## Problem + +Client applications need to interact with services, for example: badge issuance, directory requests, telemetry submissions, blockchain reads and writes, LLM calls. The only communication primitive available today is a duplex connection, so each of these interactions requires the full connection procedure - creating queues, key agreement, double ratchet initialization - and leaves persistent state on both sides: queues, ratchet state, connection records, message history. + +This is the wrong primitive for most service interactions: + +1. Cost. To send the first request to a not yet connected service, the client and the service exchange multiple commands across two servers. For "search the directory" all of it is overhead. The setup cost also creates an incentive to keep connections open, and a service with N users who used it once permanently holds N sets of queues and ratchet states. + +2. Privacy. A connection is a stable pairwise pseudonym. If a service were to use a duplex connection, it could link all requests made over it into a profile: search history in the directory, blockchain operations linked even when different on-chain keys are used, telemetry that becomes longitudinal tracking. The client also accumulates history that can be recovered from the device. Where continuity is needed, it can be provided in the application protocol (e.g., a token included in requests), without a transport-level identity. + +3. Encryption. Messages sent to contact addresses outside an established connection (invitations) have a single layer of X25519 encryption. This was acceptable while such messages contained only a connection link and profile (though it is planned to be improved); it is not acceptable for service requests. + +In-app service addresses should be stored as names resolving to links via the existing addressing layer (server host in the link authority, current link data retrieved with LGET), so that service links can be changed without redeploying the apps. Name resolution is already supported, and out of scope. + +## Security objectives + +1. Requests from the same client must not be linkable to each other by the service or by servers, and no state that outlives the exchange is created on either side. +2. Post-quantum resistant e2e encryption of requests and replies; decryptability of recorded requests must be bounded by key rotation, without changing the address. +3. Reply authenticity must be verifiable against the link; substitution, replay, dropping or reordering of replies by servers must be detectable. +4. A repeated or replayed request must not be executed twice. + +## Solution + +A service address is a contact address subtype. The client sends a request in a single message to the address queue and receives replies in a reply queue it creates for this request: + +1. Resolve the service name to a short link; retrieve and cache link data (`LGET`, via proxy when IP protection is needed). +2. Create a reply queue on a client-chosen server (`NEW`, messaging mode, sender can secure, subscribed at creation). +3. Send the request to the service address queue (unauthenticated `SEND`, via proxy when IP protection is needed). The request contains the reply queue address with fresh keys, and is encrypted to rotatable service keys published in link data, using hybrid X25519 + KEM encryption at the queue layer. +4. The service secures the reply queue and sends the first reply with one combined command. Replies are encrypted to the fresh keys from the request; each reply is signed with the root key over the reply and the request hash, and includes a flag whether more replies follow. +5. The client receives replies until the final one or a deadline, then deletes the reply queue (`DEL`). + +To the first request the client sends 3 commands (2 with cached link data). Streamed replies (LLM output) and delayed replies (credential issued after payment) are ordinary queue messages, distinguished only by the flag whether more replies follow. + +How this meets the objectives: + +1. Unlinkability: fresh keys and a fresh reply queue per request; the sender's IP address and session are protected by existing private routing; the reply queue is deleted after the exchange; nothing is shared between two requests. +2. Encryption: hybrid X25519 + sntrup761 secret at the queue layer for the request and all replies; keys are published in mutable link data, rotated with `LSET`, old keys deleted after a short window. The double ratchet is not used: it would create per-request state before the service decides to reply, and its properties serve long-lived sessions, not one-off exchanges. +3. Authenticity: every reply is signed with the root key committed in the immutable link data, over the reply and the request hash; the hash chain across replies detects dropped and reordered replies. +4. Replay protection: a retry or replay is the byte-identical request; the service agent stores sent replies for a limited window and re-sends them without executing the request again. + +## Design + +Syntax below uses [ABNF][1] with [case-sensitive strings extension][2]. `shortString`, `largeString` and `x509encoded` are defined in the [SMP protocol](../protocol/simplex-messaging.md); `smpServer` and `agentVersion` - in the [agent protocol](../protocol/agent-protocol.md). All hashes below are SHA3-256. + +### Service address + +A new contact address type - char `s` in short links, next to existing `a`/`c`/`g`/`r`: + +```abnf +contactType =/ %s"s" ; service address +``` + +Link data has the same structure as contact addresses: immutable fixed data (root key, connection request data with the queue address) committed by the link hash, and mutable user data signed by the root key. Fixed data and contact user data encodings ignore trailing bytes, so the fields below are added without a new link format. + +Agents dispatch on the address type and the envelope type: a normal address rejects requests, a service address rejects invitations and confirmations. + +### Service key bundle + +Mutable user data of a service address includes a key bundle, appended to the existing contact user data encoding: + +```abnf +userContactData = direct ownersList relaysList userLinkData [serviceKeys] +serviceKeys = %s"0" / (%s"1" serviceKeyBundle) ; skipped by earlier versions +serviceKeyBundle = keyId reqDhKey reqKemKey +keyId = shortString ; referenced by requests +reqDhKey = length x509encoded ; X25519, used instead of the fixed-data queue key +reqKemKey = largeString ; sntrup761 encapsulation key, 1158 bytes +``` + +The service periodically rotates the bundle with `LSET` and keeps previous private keys for a window covering client link data caching and queue message retention. A request encrypted to an unknown key ID fails with an error; the client then re-fetches link data and retries. Deleting expired keys is what bounds decryptability of recorded requests, so the window must be short and fixed. + +Link data is encrypted with a symmetric key derived from the link, which is not weakened by quantum computers, so publishing the bundle in link data does not undermine the scheme. + +Sizes: KEM public key is 1158 bytes, user data is padded to 13784 bytes - the bundle fits together with application data. + +### Queue-layer PQ encryption + +The single-shot queue encryption (today: crypto_box with a secret from the queue DH key and a sender ephemeral key, ephemeral key in the message public header) is extended to a hybrid scheme in a new SMP client version. The public header gains a third variant: + +```abnf +smpPubHeaderHybrid = smpClientVersion %s"2" optKeyId senderPublicDhKey kemCiphertext +optKeyId = %s"0" / (%s"1" keyId) ; present in requests, absent in replies +senderPublicDhKey = length x509encoded ; fresh X25519 key +kemCiphertext = largeString ; sntrup761 ciphertext, 1039 bytes +``` + +The secret is derived from both shared secrets, and the body is encrypted with NaCl secret_box, padded as today: + +``` +secret = HKDF(dh(recipient key, sender ephemeral key) || KEM shared secret) +``` + +The header is visible to the destination server, as invitation headers are today (with proxied sending it is not visible to the proxy). + +The same scheme applies to replies, using the keys from the request: the first reply includes the hybrid header, the hybrid secret is computed once per reply queue and used for subsequent replies with the empty header - the same one-secret-per-queue model as existing sender queues. Invitations and confirmations can adopt the same scheme independently of this design, removing the single-layer X25519 exposure of the profile. + +### Request + +A new agent envelope (alongside `agentConfirmation`, `agentMsgEnvelope`, `agentInvitation`, `agentRatchetKey`): + +```abnf +agentRequest = agentVersion %s"Q" replyQueue requestBody +replyQueue = smpClientVersion smpServer senderId dhPublicKey kemPublicKey sndAuthPublicKey +smpClientVersion = 2*2 OCTET +senderId = shortString ; sender ID of the reply queue +dhPublicKey = length x509encoded ; X25519, fresh per request +kemPublicKey = largeString ; sntrup761 encapsulation key, fresh per request +sndAuthPublicKey = length x509encoded ; key to secure the reply queue +requestBody = *OCTET ; remaining bytes, application-defined +``` + +The envelope type indicates to the receiving agent that no connection is created and replies are sent to the provided queue. The reply queue keys never appear outside the encrypted request. Requests must fit in one message; larger payloads are passed as an XFTP file description in the request body. + +A retry is the byte-identical stored request message, so all correlation uses one value: the request hash - the hash of the message body, the same bytes as sent by the client and as received by the service after the server encryption layer is removed. + +Request delivery failures follow existing SEND semantics: on QUOTA (the address queue is full) the client retries with backoff within the request deadline. + +### Replies + +The service secures the reply queue with the sender key from the request and sends the first reply in one combined SMP command (see below). Replies use a new agent envelope, following the signature-first structure of link data: + +```abnf +agentResponse = agentVersion %s"P" signature signedReply +signature = length 64*64 OCTET ; root key signature of signedReply +signedReply = requestHash prevMsgHash final replyBody +requestHash = shortString ; hash of the request message body +prevMsgHash = shortString ; hash of the previous agentResponse, empty in the first reply +final = %s"T" / %s"F" ; F - more replies follow +replyBody = *OCTET ; remaining bytes, application-defined +``` + +The client verifies each reply: the signature against the root key from link data, the request hash against the sent request, the hash chain against the previous reply. Forging a reply requires the root key even if the reply queue keys are compromised; a signed reply cannot be replayed for a different request; the hash chain detects replies dropped or reordered by the reply queue server. + +The client deletes the reply queue on the final flag, on the request deadline (application-defined per request), or on cancellation. Deleting the queue also cancels the stream: subsequent SEND commands from the service fail with AUTH. Router queue and message expiry limit the lifetime of abandoned queues. Stream length between client acknowledgements is bounded by queue capacity. + +Delayed replies (e.g., a credential issued after payment) are stored in the reply queue within message retention time and received when the client subscribes again; notification credentials can be added to the reply queue with existing commands. Messages in the reply queue that fail decryption or verification are acknowledged and dropped. + +### Combined SKEY+SEND command + +A new SMP command combining `SKEY` and `SEND` in one transmission: + +```abnf +secureSend = %s"SSND " senderAuthPublicKey SP msgFlags SP smpEncMessage +senderAuthPublicKey = length x509encoded +``` + +The command is authorized with the key it sets, as `SKEY`, and only accepted on messaging-mode queues. The server responds `OK` or `ERR`. It is idempotent in both parts: + +- key part: as `SKEY` today - repeated command with the same key succeeds, different key fails with AUTH +- send part: the server stores a hash of the message body until the message is acknowledged; within that window a repeated command with the same body is not delivered again and is reported as delivered + +A retry arriving after the message was acknowledged is delivered as a duplicate and suppressed by the receiving agent (by message hash), so the server-side marker covers the common case and the agent covers the rest. + +It is used by the service for the first reply, and by the joining party in the fast connection handshake (currently `SKEY` then `SEND` confirmation), removing one command and round trip from both flows. + +### Idempotency + +Handled uniformly by the agent: it stores the request hash and sent replies for a window declared in link data (with a default), and re-sends stored replies for a repeated request without invoking the service application. This gives exactly-once execution over at-least-once delivery for all services; services with idempotent semantics of their own lose nothing. + +The guarantee is bounded by the window: the client must not retry after the request deadline, and the deadline must not exceed the declared window. Stored replies may be evicted under storage pressure; a retry whose replies were evicted receives an error prompting a new request. Within the window a request is never executed twice. + +### Out of scope + +- Continuity: sessions across requests are an application concern (tokens in request and reply bodies). +- Service-initiated messages: there is no standing channel; use a connection where push is needed. +- Abuse protection beyond existing queue quotas: services can require application-level credentials (e.g., a badge) in request bodies; rate limiting options are a separate discussion. +- Scaling request reception: a single address queue bounds service throughput; distributing reception across multiple queues or relays (the existing `relays` field in contact link data) is a separate discussion. +- Name resolution: existing addressing layer. + +[1]: https://tools.ietf.org/html/rfc5234 +[2]: https://tools.ietf.org/html/rfc7405 From 97e7e9b7c5fc5ec0d217d713072da710ad7a6fb8 Mon Sep 17 00:00:00 2001 From: "Evgeny @ SimpleX Chat" <259188159+evgeny-simplex@users.noreply.github.com> Date: Sun, 12 Jul 2026 15:09:01 +0000 Subject: [PATCH 2/6] update rfc, plan --- .../2026-07-11-service-rpc-implementation.md | 270 ++++++++++++++---- rfcs/2026-07-11-service-rpc.md | 107 +++---- 2 files changed, 265 insertions(+), 112 deletions(-) diff --git a/plans/2026-07-11-service-rpc-implementation.md b/plans/2026-07-11-service-rpc-implementation.md index e7fa0fd75..68b9bff67 100644 --- a/plans/2026-07-11-service-rpc-implementation.md +++ b/plans/2026-07-11-service-rpc-implementation.md @@ -2,7 +2,7 @@ RFC: [../rfcs/2026-07-11-service-rpc.md](../rfcs/2026-07-11-service-rpc.md) -Types and instances below must encode exactly the ABNF in the RFC. Constructor and event names are provisional. +Types and instances below must encode exactly the ABNF in the RFC. Constructor, event, table and function names are provisional. ## Versions @@ -20,7 +20,9 @@ ctTypeChar CCTService = 'S' -- 's' in links: link encoding lowercases, parsing u ctTypeP 'S' = pure CCTService ``` -## Service key bundle - `Simplex.Messaging.Agent.Protocol` +The contact type is currently fixed to `CCTContact` when the agent reconstructs a contact link (`Agent.hs`, `cslContact`). It becomes a stored property of the address (column `link_contact_type` below), read when the link is reconstructed and when an address queue message is dispatched. + +## Service key bundle in link data - `Simplex.Messaging.Agent.Protocol` ```haskell data ServiceKeyBundle = ServiceKeyBundle @@ -36,7 +38,7 @@ instance Encoding ServiceKeyBundle where pure ServiceKeyBundle {keyId, reqDhKey, reqKemKey} ``` -`UserContactData` gains a field, appended to the encoding (earlier versions skip it, its absence parses as `Nothing`): +`UserContactData` gains a field, appended to the encoding, absent in data written by earlier versions. The agent sets it when it signs and updates link data; the application supplies only its own data. There is no retention value in link data - retention is service configuration (see Idempotency). ```haskell data UserContactData = UserContactData @@ -60,8 +62,6 @@ instance Encoding UserContactData where pure UserContactData {direct, owners, relays, userData, serviceKeys} ``` -The service address record stores current and previous private key bundles with rotation timestamps; expired bundles are deleted. - ## SSND command - `Simplex.Messaging.Protocol`, `Simplex.Messaging.Server` ```haskell @@ -74,13 +74,15 @@ SSND_ :: CommandTag Sender -- encodeProtocol SSND k flags msg -> e (SSND_, ' ', k, ' ', flags, ' ', Tail msg) --- protocolP, gated by VersionSMP +-- protocolP, in a new VersionSMP SSND_ -> SSND <$> (smpP <* A.space) <*> (smpP <* A.space) <*> (unTail <$> smpP) ``` `checkCredentials`: as `SKEY` - authorization required, sender entity ID. Server verification: `vc SSender (SSND k _ _) = verifySecure k` - authorized by the key it sets. -Server processing: `checkMode QMMessaging`, then `secureQueue_` (existing, idempotent), then deliver with deduplication. The queue store keeps the hash of the unacknowledged message delivered by `SSND`; a repeated `SSND` with an equal body hash responds `OK` without storing; the hash is dropped when the message is acknowledged. +Server processing: `checkMode QMMessaging`, then `secureQueue_` (existing, idempotent), then deliver with deduplication. All queue store backends (STM, journal, PostgreSQL) keep the hash of the message delivered by `SSND` until it is acknowledged; a repeated `SSND` with an equal message hash responds `OK` without delivering it again; the hash is removed when the message is acknowledged. Server stats gain an `SSND` counter. + +Proxying: `proxySMPCommand` is polymorphic in the sender command, so `SSND` forwards through `PFWD`/`RFWD` without changes. ## Hybrid public header - `Simplex.Messaging.Protocol` @@ -117,7 +119,9 @@ hybridSecret :: C.DhSecretX25519 -> KEMSharedKey -> C.SbKey hybridSecret dh (KEMSharedKey k) = C.unsafeSbKey $ C.hkdf "" (C.dhBytes' dh <> BA.convert k) "SimpleXSMPQueue" 32 ``` -The body is encrypted with `C.sbEncrypt` (secret_box) instead of `C.cbEncrypt`, padded to the same lengths. Sending variant of `agentCbEncryptOnce`: generate ephemeral X25519 pair, encapsulate to the published KEM key, encode `PubHeaderHybrid`. Receiving: select private keys by `phKeyId` (requests) or use the reply queue keys from the request (replies); unknown key ID produces the error that makes the client re-fetch link data. +The body is encrypted with `C.sbEncrypt` (secret_box) rather than `C.cbEncrypt`, padded to the same lengths. The sending side generates an ephemeral X25519 pair, encapsulates to the recipient KEM key, and writes `PubHeaderHybrid`. The receiving side selects private keys by `phKeyId` for a request, or uses the reply queue keys for a reply, and computes the same secret. A request with an unknown key ID is discarded and counted. + +Size constants: the hybrid header adds about 1.2KB (KEM ciphertext 1039 bytes plus the ephemeral key). Define the request and reply body length constants from the existing padded body lengths at implementation. ## Agent envelopes - `Simplex.Messaging.Agent.Protocol` @@ -127,103 +131,249 @@ data AgentMsgEnvelope | AgentRequest { agentVersion :: VersionSMPA, replyQueue :: RequestReplyQueue, - requestBody :: ByteString + requestPayload :: ByteString -- opaque, hashed } | AgentResponse { agentVersion :: VersionSMPA, - signature :: C.Signature 'C.Ed25519, -- root key signature of signedReply - signedReply :: ByteString -- encoded SignedReply, parsed after signature verification + signature :: C.Signature 'C.Ed25519, -- root key signature of signedResponse + signedResponse :: ByteString -- encoded SignedResponse, parsed after the signature is verified } data RequestReplyQueue = RequestReplyQueue { smpClientVersion :: VersionSMPC, smpServer :: SMPServer, - senderId :: SMP.SenderId, - dhPublicKey :: C.PublicKeyX25519, -- fresh per request - kemPublicKey :: KEMPublicKey, -- fresh per request - sndAuthKey :: SndPublicAuthKey -- to secure the reply queue + senderId :: SMP.SenderId, -- sender ID of the reply queue + dhPublicKey :: C.PublicKeyX25519, -- reply queue X25519 key (its e2e key) + kemPublicKey :: KEMPublicKey -- reply queue sntrup761 key } -data SignedReply = SignedReply - { requestHash :: ByteString, -- SHA3-256 of the request message body - prevMsgHash :: ByteString, -- SHA3-256 of the previous reply envelope, empty in the first reply - final :: Bool, - replyBody :: ByteString +data SignedResponse = SignedResponse + { requestHash :: ByteString, -- SHA3-256 of requestPayload + prevMsgHash :: ByteString, -- SHA3-256 of the previous AgentResponse, empty in the first + more :: Bool, -- True if more reply messages follow + responses :: NonEmpty ByteString -- opaque application responses } ``` +The request does not include a key to secure the reply queue: the service generates its own sender key and secures the queue with `SSND`. `requestPayload` is the only hashed part; the reply queue fields are not hashed. + ```haskell instance Encoding AgentMsgEnvelope where smpEncode = \case ... -- existing constructors - AgentRequest {agentVersion, replyQueue, requestBody} -> - smpEncode (agentVersion, 'Q', replyQueue, Tail requestBody) - AgentResponse {agentVersion, signature, signedReply} -> - smpEncode (agentVersion, 'P', signature, Tail signedReply) + AgentRequest {agentVersion, replyQueue, requestPayload} -> + smpEncode (agentVersion, 'Q', replyQueue, Tail requestPayload) + AgentResponse {agentVersion, signature, signedResponse} -> + smpEncode (agentVersion, 'P', signature, Tail signedResponse) smpP = do agentVersion <- smpP smpP >>= \case ... -- existing constructors 'Q' -> do - (replyQueue, Tail requestBody) <- smpP - pure AgentRequest {agentVersion, replyQueue, requestBody} + (replyQueue, Tail requestPayload) <- smpP + pure AgentRequest {agentVersion, replyQueue, requestPayload} 'P' -> do - (signature, Tail signedReply) <- smpP - pure AgentResponse {agentVersion, signature, signedReply} + (signature, Tail signedResponse) <- smpP + pure AgentResponse {agentVersion, signature, signedResponse} instance Encoding RequestReplyQueue where - smpEncode RequestReplyQueue {smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey, sndAuthKey} = - smpEncode (smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey, sndAuthKey) + smpEncode RequestReplyQueue {smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey} = + smpEncode (smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey) smpP = do - (smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey, sndAuthKey) <- smpP - pure RequestReplyQueue {smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey, sndAuthKey} + (smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey) <- smpP + pure RequestReplyQueue {smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey} -instance Encoding SignedReply where - smpEncode SignedReply {requestHash, prevMsgHash, final, replyBody} = - smpEncode (requestHash, prevMsgHash, final, Tail replyBody) +instance Encoding SignedResponse where + smpEncode SignedResponse {requestHash, prevMsgHash, more, responses} = + smpEncode (requestHash, prevMsgHash, more) <> smpEncodeList (map Large $ L.toList responses) smpP = do - (requestHash, prevMsgHash, final, Tail replyBody) <- smpP - pure SignedReply {requestHash, prevMsgHash, final, replyBody} + (requestHash, prevMsgHash, more) <- smpP + rs <- map unLarge <$> smpListP + responses <- maybe (fail "empty responses") pure $ L.nonEmpty rs + pure SignedResponse {requestHash, prevMsgHash, more, responses} ``` Signing and verification follow the link data pattern (`Simplex.Messaging.Crypto.ShortLink`): ```haskell --- service -mkAgentResponse :: C.PrivateKeyEd25519 -> VersionSMPA -> SignedReply -> AgentMsgEnvelope -mkAgentResponse rootPrivKey agentVersion reply = - let signedReply = smpEncode reply - in AgentResponse {agentVersion, signature = C.sign' rootPrivKey signedReply, signedReply} - --- client -verifyAgentResponse :: C.PublicKeyEd25519 -> AgentMsgEnvelope -> Either AgentErrorType SignedReply -verifyAgentResponse rootKey AgentResponse {signature, signedReply} - | C.verify' rootKey signature signedReply = parse smpP (AGENT A_MESSAGE) signedReply +-- service: signing key is linkPrivSigKey from the address ShortLinkCreds +mkAgentResponse :: C.PrivateKeyEd25519 -> VersionSMPA -> SignedResponse -> AgentMsgEnvelope +mkAgentResponse rootPrivKey agentVersion resp = + let signedResponse = smpEncode resp + in AgentResponse {agentVersion, signature = C.sign' rootPrivKey signedResponse, signedResponse} + +-- client: rootKey from the fixed link data +verifyAgentResponse :: C.PublicKeyEd25519 -> AgentMsgEnvelope -> Either AgentErrorType SignedResponse +verifyAgentResponse rootKey AgentResponse {signature, signedResponse} + | C.verify' rootKey signature signedResponse = parse smpP (AGENT A_MESSAGE) signedResponse | otherwise = Left ... -- verification error ``` +## Reply queue - no new connection type + +The reply queue on the client is a receive connection (`RcvConnection`), extended to carry the KEM key and the computed hybrid secret. There is no new connection type and no flag on the connection: a client service request row (below) references the reply queue connection, and its presence identifies the connection as an RPC reply queue for dispatch and cleanup. + +The reply queue reuses the existing receive queue fields. `e2ePrivKey` is its X25519 key, its public half is the queue address DH key sent in the request. Two nullable columns are added to `rcv_queues`: + +- `reply_kem_priv_key` - the reply queue KEM private key. +- `reply_secret` - the hybrid secret, empty until the first reply, then set from the first reply's header, as the per-queue DH secret is set by `setRcvQueueConfirmedE2E` on the first message today. Later replies are decrypted with it by the standard receive path. + +The service does not create a connection or queue for the reply queue. It sends replies directly to the reply queue address, as `sendInvitation` sends to a queue with no connection. + +## Database schema - `Agent/Store/SQLite/Migrations`, `Agent/Store/Postgres/Migrations` + +One migration (`M20260712_service_rpc`), SQLite shown, PostgreSQL mirrors it; column types follow existing schema conventions. + +```sql +-- address contact type (contact address queues), and reply queue keys (reply queues). +-- NULL link_contact_type means 'A' (contact) for existing rows. +ALTER TABLE rcv_queues ADD COLUMN link_contact_type TEXT; +ALTER TABLE rcv_queues ADD COLUMN reply_kem_priv_key BLOB; +ALTER TABLE rcv_queues ADD COLUMN reply_secret BLOB; + +-- service side: current and retired bundle private keys. +CREATE TABLE service_address_keys( + service_address_key_id INTEGER PRIMARY KEY AUTOINCREMENT, + conn_id BLOB NOT NULL REFERENCES connections ON DELETE CASCADE, + key_id BLOB NOT NULL, + dh_priv_key BLOB NOT NULL, + kem_priv_key BLOB NOT NULL, + created_at TEXT NOT NULL, + retired_at TEXT -- set on rotation, row deleted when the key retention period ends +); +CREATE UNIQUE INDEX idx_service_address_keys ON service_address_keys(conn_id, key_id); + +-- client side: one pending request per reply queue connection. +CREATE TABLE snd_service_requests( + snd_service_request_id INTEGER PRIMARY KEY AUTOINCREMENT, + conn_id BLOB NOT NULL REFERENCES connections ON DELETE CASCADE, -- reply queue connection + request_hash BLOB NOT NULL, + root_key BLOB NOT NULL, -- to verify replies + last_reply_hash BLOB, -- updated as reply messages arrive + deadline TEXT NOT NULL, + created_at TEXT NOT NULL +); + +-- service side: one record per distinct request hash on an address. +CREATE TABLE rcv_service_requests( + rcv_service_request_id INTEGER PRIMARY KEY AUTOINCREMENT, + conn_id BLOB NOT NULL REFERENCES connections ON DELETE CASCADE, -- address connection + request_hash BLOB NOT NULL, + ended INTEGER NOT NULL DEFAULT 0, -- a reply message with more = False was sent + expires_at TEXT NOT NULL, -- created + retention + created_at TEXT NOT NULL +); +CREATE UNIQUE INDEX idx_rcv_service_requests ON rcv_service_requests(conn_id, request_hash); + +-- service side: ordered signed reply messages for a request; the signed envelope +-- is independent of the reply queue, so it is stored once and encrypted per queue on send. +CREATE TABLE rcv_service_replies( + rcv_service_reply_id INTEGER PRIMARY KEY AUTOINCREMENT, + rcv_service_request_id INTEGER NOT NULL REFERENCES rcv_service_requests ON DELETE CASCADE, + reply_seq INTEGER NOT NULL, + more INTEGER NOT NULL, + signed_reply BLOB NOT NULL -- encoded AgentResponse +); +CREATE UNIQUE INDEX idx_rcv_service_replies ON rcv_service_replies(rcv_service_request_id, reply_seq); + +-- service side: reply queues subscribed under a request (the first, and any repeat). +CREATE TABLE rcv_service_reply_queues( + rcv_service_reply_queue_id INTEGER PRIMARY KEY AUTOINCREMENT, + rcv_service_request_id INTEGER NOT NULL REFERENCES rcv_service_requests ON DELETE CASCADE, + host TEXT NOT NULL, + port TEXT NOT NULL, + snd_id BLOB NOT NULL, + server_key_hash BLOB, + reply_secret BLOB NOT NULL, -- hybrid secret to this reply queue + secured INTEGER NOT NULL DEFAULT 0, -- reply queue secured with SSND + last_sent_seq INTEGER NOT NULL DEFAULT 0 +); +``` + +## Agent API - `Simplex.Messaging.Agent` + +Service side: + +```haskell +-- Creates a contact connection with CCTService link type, generates the root key +-- and the first key bundle, composes and signs link data. +createServiceAddress :: AgentClient -> UserId -> UserLinkData -> AE (ConnId, ConnShortLink 'CMContact) + +-- Re-signs and updates user data with LSET, keeping the agent-owned key bundle. +updateServiceAddressData :: AgentClient -> ConnId -> UserLinkData -> AE () + +-- Generates a new bundle, updates link data, retires the previous keys until the retention period ends. +-- Also runs on a schedule (config) while the address is subscribed. +rotateServiceAddressKeys :: AgentClient -> ConnId -> AE () + +-- Sends one reply message with a list of responses; more = False ends the exchange. +-- The first message to each reply queue secures it with SSND; later messages use SEND. +sendServiceReply :: AgentClient -> ServiceRequestRef -> Bool -> NonEmpty ByteString -> AE () +``` + +Address deletion: existing `deleteConnection`. + +Client side (name resolution to a link is an existing API; the link must have `CCTService` type and a key bundle in link data): + +```haskell +-- Retrieves link data, creates the reply queue, sends the request, and waits for the first +-- reply message up to the deadline. The callback receives later reply messages while the process runs. +sendServiceRequest :: + AgentClient -> UserId -> ConnShortLink 'CMContact -> UTCTime -> + ByteString -> (ServiceReply -> IO ()) -> AE ServiceReply + +-- Deletes the reply queue and the request row, and drops the callback. +cancelServiceRequest :: AgentClient -> ConnId -> AE () + +data ServiceReply = ServiceReply {responses :: NonEmpty ByteString, more :: Bool} +``` + +The first reply message returns from `sendServiceRequest`; later messages are delivered through the callback. Both the waiting call and the callback are held in an in-memory map in `AgentClient`, keyed by the reply queue connection, and filled by the receive path. They do not survive a restart. + +Service side events (`AEvent`, entity is the address connection): + +```haskell +SREQ :: ServiceRequestRef -> ByteString -> AEvent AEConn -- request received, payload for the bot +``` + +`ServiceRequestRef` identifies the request record; the bot passes it to `sendServiceReply`. + ## Agent processing - `Simplex.Messaging.Agent`, `Simplex.Messaging.Agent.Client` Client side: -- `sendServiceRequest`: read link data (cached or `LGET`, proxied per config), create the reply queue (`NEW`, `QRMessaging`, subscribe), generate fresh X25519 + KEM keys and sender auth key, build and store the request message, send (proxied per config), return request ID. -- Retries re-send the stored byte-identical message. Deadline, cancellation and `final` delete the reply queue and the request record. -- Reply queue messages in `processSMPTransmissions`: decrypt with the stored hybrid secret (first reply establishes it from the hybrid header), verify with `verifyAgentResponse` and the hash chain, deliver as events; failures are acknowledged and dropped. -- New events: reply received (request ID, final flag, body), request failed (request ID, error). +- `sendServiceRequest`: retrieve link data (`LGET`, proxied per config) for every request - no caching; create the reply queue (`NEW`, messaging mode, subscribed) with an added KEM key; write the `snd_service_requests` row; send the request (proxied per config); wait on the in-memory sink for the first reply until the deadline. +- Reply processing in `processSMPTransmissions`: a message on a receive queue that has a `snd_service_requests` row is a reply. Decrypt (the first message sets `reply_secret` from its header), verify with `verifyAgentResponse`, check the request hash and the previous-message hash, update `last_reply_hash`, deliver to the waiting call or the callback. A message that fails verification is acknowledged and discarded. On `more = False`, or on the deadline, or on `cancelServiceRequest`, delete the reply queue (`DEL`) and the row. +- `cleanupManager` gains one step: delete `snd_service_requests` past the deadline and mark their reply queue connections deleted; the existing deleted-connections step sends `DEL` and removes them. After a restart every row is stale, so this step removes all reply queues left behind. Service side: -- Address queue messages with `AgentRequest`: compute request hash, look up the dedup store; on hit re-send stored replies (or the expiry error); on miss deliver a request event to the service application. -- Send replies: secure the reply queue and send the first reply with `SSND`, subsequent replies with `SEND`; store sent replies under the request hash until the window expires. +- Address queue message dispatch reads `link_contact_type`: a service address accepts only `AgentRequest` and rejects invitations and confirmations; a non-service address rejects `AgentRequest`. +- On `AgentRequest`: select bundle private keys by `keyId`, decrypt, compute the request hash. Look up `rcv_service_requests` by (address conn, hash): + - new: insert the request, insert a `rcv_service_reply_queues` row for the reply queue in the request, deliver `SREQ` to the bot. + - existing: insert a `rcv_service_reply_queues` row for the reply queue in this request, and send it every `rcv_service_replies` message already stored, in order. +- `sendServiceReply`: append a `rcv_service_replies` message (sign with the address root key), then send it to every reply queue of the request - `SSND` for a queue not yet secured, `SEND` after - encrypting the stored envelope to each queue's `reply_secret`; set `ended` when `more = False`. +- `cleanupManager` gains one step: delete `rcv_service_requests` past `expires_at` (cascading to replies and reply queues) and `service_address_keys` past the key retention period. + +Configuration (`AgentConfig`): default request deadline, request retention period (1 to 24 hours), key rotation interval. + +Errors: API failures reuse `AgentErrorType` (for example `CMD PROHIBITED` for a link that is not a service address). A new constructor is added only if the chat library needs to distinguish service request failures - decided at implementation. + +## Correlation and chat + +A reply is connected to its request by the reply queue: one request has one reply queue, and every message in that queue is a reply to that request. The request hash is used only in the reply signature, to bind a reply to the request content. The application ID is content inside `requestPayload`, used only by the application to make two requests equal or different; the agent does not read it. + +Both ends are chat bots on the chat library. The chat library serializes a service command into `requestPayload` and deserializes the responses; the agent transports them and correlates by reply queue. The chat framework's `chatServiceCalls` correlation by `(AgentConnId, SharedMsgId)` is not used - the agent correlates by reply queue and the call returns the first reply directly. -Database: +## Tests -- Client: in-flight requests - request ID, stored request message, reply queue, hybrid secret, deadline, last reply hash. -- Service: dedup store - request hash, sent replies, expiry. +- Encoding roundtrips: envelopes, `ServiceKeyBundle` in `UserContactData`, `PubHeader` (all three variants), `SSND`, `SignedResponse` with one and several responses. +- Server: `SSND` on messaging and contact queues, repeated `SSND` before and after ACK, a different key, via proxy. +- Agent end-to-end: a request with one reply message; several responses in one message; responses in several messages delivered to the callback; a repeat request while pending coalesces onto the same operation; a repeat request after completion receives the stored messages without a second execution; key rotation with an old key still accepted while kept; a request whose key was deleted fails at the deadline; deadline; cancellation; a restart deletes reply queues; envelope rejection on both address types. ## Phases -1. SMP protocol: `SSND`, hybrid public header, version bumps, server dedup marker. -2. Agent: address type, key bundle, envelopes, client API and reply processing. -3. Agent: service-side dedup store and request events. -4. Adoption: `SSND` in the fast connection handshake; hybrid scheme for invitations and confirmations. +1. SMP protocol: `SSND`, hybrid public header, version bumps, server dedup hash, server tests. +2. Agent: address type, key bundle in link data, envelopes, reply queue columns, schema migration, `createServiceAddress`, `sendServiceRequest`, reply processing, cleanup. +3. Agent: service-side request store, `sendServiceReply`, coalescing and repeats, key rotation, end-to-end tests. +4. Adoption: `SSND` in the fast connection handshake; the hybrid scheme for invitations and confirmations. diff --git a/rfcs/2026-07-11-service-rpc.md b/rfcs/2026-07-11-service-rpc.md index b29e132c0..1eb7a42df 100644 --- a/rfcs/2026-07-11-service-rpc.md +++ b/rfcs/2026-07-11-service-rpc.md @@ -21,26 +21,26 @@ In-app service addresses should be stored as names resolving to links via the ex 1. Requests from the same client must not be linkable to each other by the service or by servers, and no state that outlives the exchange is created on either side. 2. Post-quantum resistant e2e encryption of requests and replies; decryptability of recorded requests must be bounded by key rotation, without changing the address. 3. Reply authenticity must be verifiable against the link; substitution, replay, dropping or reordering of replies by servers must be detectable. -4. A repeated or replayed request must not be executed twice. +4. A repeated request for the same operation must not be executed twice. ## Solution -A service address is a contact address subtype. The client sends a request in a single message to the address queue and receives replies in a reply queue it creates for this request: +A service address is a contact address subtype. Both the client and the service are chat bots using the chat library over the agent. The chat library serializes a request into opaque bytes and calls the agent; the agent transports it and returns replies; the chat library deserializes them. The correlation of a reply with its request is the reply queue: each request has its own reply queue, and every message in that queue is a reply to that request. -1. Resolve the service name to a short link; retrieve and cache link data (`LGET`, via proxy when IP protection is needed). -2. Create a reply queue on a client-chosen server (`NEW`, messaging mode, sender can secure, subscribed at creation). -3. Send the request to the service address queue (unauthenticated `SEND`, via proxy when IP protection is needed). The request contains the reply queue address with fresh keys, and is encrypted to rotatable service keys published in link data, using hybrid X25519 + KEM encryption at the queue layer. -4. The service secures the reply queue and sends the first reply with one combined command. Replies are encrypted to the fresh keys from the request; each reply is signed with the root key over the reply and the request hash, and includes a flag whether more replies follow. -5. The client receives replies until the final one or a deadline, then deletes the reply queue (`DEL`). +The exchange: -To the first request the client sends 3 commands (2 with cached link data). Streamed replies (LLM output) and delayed replies (credential issued after payment) are ordinary queue messages, distinguished only by the flag whether more replies follow. +1. Resolve the service name to a short link and retrieve link data (`LGET`, via proxy when IP protection is needed). Link data holds the root key for verifying replies and the service key bundle for encrypting the request. +2. Create a reply queue (`NEW`, messaging mode, subscribed at creation). The reply queue is a receive queue with an added KEM key; the service encrypts replies to the reply queue keys. +3. Send the request to the service address queue once (unauthenticated `SEND`, via proxy when IP protection is needed). The request includes the reply queue address and its KEM key, and the application payload. It is encrypted to the service key bundle with hybrid X25519 + KEM encryption at the queue layer. There is no transport retry; a reply is the success signal, and a hard error fails the call. +4. The service decrypts the request, delivers the payload to the service application, secures the reply queue, and sends replies. Each reply message includes the request hash, the previous message hash, a flag whether more messages follow, and a non-empty list of application responses; it is signed with the root key. +5. The client verifies each reply message and delivers its responses to the application. The first reply message returns from the call; later reply messages are delivered through a callback the application registered. The client deletes the reply queue on the final message, on the request deadline, or when the application cancels the request. How this meets the objectives: 1. Unlinkability: fresh keys and a fresh reply queue per request; the sender's IP address and session are protected by existing private routing; the reply queue is deleted after the exchange; nothing is shared between two requests. -2. Encryption: hybrid X25519 + sntrup761 secret at the queue layer for the request and all replies; keys are published in mutable link data, rotated with `LSET`, old keys deleted after a short window. The double ratchet is not used: it would create per-request state before the service decides to reply, and its properties serve long-lived sessions, not one-off exchanges. -3. Authenticity: every reply is signed with the root key committed in the immutable link data, over the reply and the request hash; the hash chain across replies detects dropped and reordered replies. -4. Replay protection: a retry or replay is the byte-identical request; the service agent stores sent replies for a limited window and re-sends them without executing the request again. +2. Encryption: hybrid X25519 + sntrup761 secret at the queue layer for the request and all replies. Request keys are published in mutable link data and rotated with `LSET`; the double ratchet is not used, because it would create per-request state before the service decides to reply and its properties serve long-lived sessions. +3. Authenticity: every reply message is signed with the root key committed in the immutable link data, over the request hash and the message; the previous-message hash in each message detects dropped and reordered messages. +4. Single execution: the service identifies a request by the hash of its payload and, within a fixed retention period, repeats the stored replies for a repeated request without running the operation again. ## Design @@ -54,7 +54,7 @@ A new contact address type - char `s` in short links, next to existing `a`/`c`/` contactType =/ %s"s" ; service address ``` -Link data has the same structure as contact addresses: immutable fixed data (root key, connection request data with the queue address) committed by the link hash, and mutable user data signed by the root key. Fixed data and contact user data encodings ignore trailing bytes, so the fields below are added without a new link format. +Link data has the same structure as contact addresses: immutable fixed data (root key, connection request data with the queue address) committed by the link hash, and mutable user data signed by the root key. Fixed data and contact user data encodings ignore trailing bytes, so the field below is added without a new link format. Agents dispatch on the address type and the envelope type: a normal address rejects requests, a service address rejects invitations and confirmations. @@ -64,80 +64,80 @@ Mutable user data of a service address includes a key bundle, appended to the ex ```abnf userContactData = direct ownersList relaysList userLinkData [serviceKeys] -serviceKeys = %s"0" / (%s"1" serviceKeyBundle) ; skipped by earlier versions +serviceKeys = %s"0" / (%s"1" serviceKeyBundle) ; absent in data from earlier versions serviceKeyBundle = keyId reqDhKey reqKemKey -keyId = shortString ; referenced by requests +keyId = shortString ; identifies the bundle, included in requests reqDhKey = length x509encoded ; X25519, used instead of the fixed-data queue key reqKemKey = largeString ; sntrup761 encapsulation key, 1158 bytes ``` -The service periodically rotates the bundle with `LSET` and keeps previous private keys for a window covering client link data caching and queue message retention. A request encrypted to an unknown key ID fails with an error; the client then re-fetches link data and retries. Deleting expired keys is what bounds decryptability of recorded requests, so the window must be short and fixed. +The service rotates the bundle with `LSET` and keeps the previous private keys long enough to decrypt requests already in the address queue - the queue message retention. The client retrieves link data for every request, so the published keys are current when the request is sent; a request becomes undecryptable only if it stays in the address queue longer than its key is kept, and such a request is not answered and fails at the client's deadline. Deleting old keys bounds the time recorded requests can be decrypted, so the retention is short and fixed. Nothing about retries or retention is published in link data. -Link data is encrypted with a symmetric key derived from the link, which is not weakened by quantum computers, so publishing the bundle in link data does not undermine the scheme. +Link data is encrypted with a symmetric key derived from the link, which is not weakened by quantum computers, so publishing the bundle in link data does not weaken the scheme. -Sizes: KEM public key is 1158 bytes, user data is padded to 13784 bytes - the bundle fits together with application data. +Sizes: the KEM public key is 1158 bytes and user data is padded to 13784 bytes, so the bundle fits together with application data. ### Queue-layer PQ encryption -The single-shot queue encryption (today: crypto_box with a secret from the queue DH key and a sender ephemeral key, ephemeral key in the message public header) is extended to a hybrid scheme in a new SMP client version. The public header gains a third variant: +The single-shot queue encryption (today: crypto_box with a secret from the queue DH key and a sender ephemeral key, the ephemeral key in the message public header) is extended to a hybrid scheme in a new SMP client version. The public header gains a third variant: ```abnf smpPubHeaderHybrid = smpClientVersion %s"2" optKeyId senderPublicDhKey kemCiphertext -optKeyId = %s"0" / (%s"1" keyId) ; present in requests, absent in replies -senderPublicDhKey = length x509encoded ; fresh X25519 key +optKeyId = %s"0" / (%s"1" keyId) ; present in requests to select the bundle, absent in replies +senderPublicDhKey = length x509encoded ; sender ephemeral X25519 key kemCiphertext = largeString ; sntrup761 ciphertext, 1039 bytes ``` -The secret is derived from both shared secrets, and the body is encrypted with NaCl secret_box, padded as today: +The secret combines both shared secrets, and the body is encrypted with NaCl secret_box, padded as today: ``` secret = HKDF(dh(recipient key, sender ephemeral key) || KEM shared secret) ``` -The header is visible to the destination server, as invitation headers are today (with proxied sending it is not visible to the proxy). +The header is readable by the destination server, as invitation headers are today; with proxied sending it is not readable by the proxy. -The same scheme applies to replies, using the keys from the request: the first reply includes the hybrid header, the hybrid secret is computed once per reply queue and used for subsequent replies with the empty header - the same one-secret-per-queue model as existing sender queues. Invitations and confirmations can adopt the same scheme independently of this design, removing the single-layer X25519 exposure of the profile. +For a request, the recipient keys are the service bundle keys selected by `keyId`. For a reply, the recipient keys are the reply queue keys included in the request. The first reply message includes the hybrid header; the client computes the secret once and stores it in the reply queue record, and later reply messages use it with the plain header. This is the one-secret-per-queue model of existing queues, and the secret is stored the same way the per-queue DH secret is stored on receiving the first message. Invitations and confirmations can adopt the same scheme independently of this design. ### Request A new agent envelope (alongside `agentConfirmation`, `agentMsgEnvelope`, `agentInvitation`, `agentRatchetKey`): ```abnf -agentRequest = agentVersion %s"Q" replyQueue requestBody -replyQueue = smpClientVersion smpServer senderId dhPublicKey kemPublicKey sndAuthPublicKey +agentRequest = agentVersion %s"Q" replyQueue requestPayload +replyQueue = smpClientVersion smpServer senderId dhPublicKey kemPublicKey smpClientVersion = 2*2 OCTET senderId = shortString ; sender ID of the reply queue -dhPublicKey = length x509encoded ; X25519, fresh per request -kemPublicKey = largeString ; sntrup761 encapsulation key, fresh per request -sndAuthPublicKey = length x509encoded ; key to secure the reply queue -requestBody = *OCTET ; remaining bytes, application-defined +dhPublicKey = length x509encoded ; reply queue X25519 key +kemPublicKey = largeString ; reply queue sntrup761 encapsulation key +requestPayload = *OCTET ; remaining bytes, opaque application payload ``` -The envelope type indicates to the receiving agent that no connection is created and replies are sent to the provided queue. The reply queue keys never appear outside the encrypted request. Requests must fit in one message; larger payloads are passed as an XFTP file description in the request body. +The envelope type indicates to the receiving agent that no connection is created and replies are sent to the reply queue. The request does not include a key to secure the reply queue: the service generates its own sender key and secures the queue with it, as a sender does in the fast handshake. The reply queue address and keys are not part of the hashed payload. -A retry is the byte-identical stored request message, so all correlation uses one value: the request hash - the hash of the message body, the same bytes as sent by the client and as received by the service after the server encryption layer is removed. +The request hash is the SHA3-256 of `requestPayload` - the same bytes the client sends and the service reads after the server encryption layer is removed. The application sets a request ID inside the payload; two requests are the same operation when their payloads, and therefore their hashes, are equal. A request must fit in one message; a larger payload is sent as an XFTP file description in the payload. -Request delivery failures follow existing SEND semantics: on QUOTA (the address queue is full) the client retries with backoff within the request deadline. +Request delivery follows existing SEND semantics: a hard error (AUTH, QUOTA) fails the call, and the application decides whether to send a new request. ### Replies -The service secures the reply queue with the sender key from the request and sends the first reply in one combined SMP command (see below). Replies use a new agent envelope, following the signature-first structure of link data: +The service secures the reply queue and sends the first reply message with one combined SMP command (see below). A reply message uses a new agent envelope, following the signature-first structure of link data: ```abnf -agentResponse = agentVersion %s"P" signature signedReply -signature = length 64*64 OCTET ; root key signature of signedReply -signedReply = requestHash prevMsgHash final replyBody -requestHash = shortString ; hash of the request message body -prevMsgHash = shortString ; hash of the previous agentResponse, empty in the first reply -final = %s"T" / %s"F" ; F - more replies follow -replyBody = *OCTET ; remaining bytes, application-defined +agentResponse = agentVersion %s"P" signature signedResponse +signature = length 64*64 OCTET ; root key signature of signedResponse +signedResponse = requestHash prevMsgHash more responses +requestHash = shortString ; hash of the request payload +prevMsgHash = shortString ; hash of the previous agentResponse message, empty in the first +more = %s"T" / %s"F" ; T - more reply messages follow +responses = length 1*responseItem ; non-empty list of application responses +responseItem = largeString ; opaque application response ``` -The client verifies each reply: the signature against the root key from link data, the request hash against the sent request, the hash chain against the previous reply. Forging a reply requires the root key even if the reply queue keys are compromised; a signed reply cannot be replayed for a different request; the hash chain detects replies dropped or reordered by the reply queue server. +A reply message includes a list of responses, so responses known together are sent in one message rather than several; responses that become known over time are sent in separate messages. The signature covers the whole message. -The client deletes the reply queue on the final flag, on the request deadline (application-defined per request), or on cancellation. Deleting the queue also cancels the stream: subsequent SEND commands from the service fail with AUTH. Router queue and message expiry limit the lifetime of abandoned queues. Stream length between client acknowledgements is bounded by queue capacity. +The client verifies each reply message: the signature against the root key from link data, the request hash against the sent request, the previous-message hash against the previous message. A message that fails verification is acknowledged and discarded. Forging a reply message requires the root key even if the reply queue keys are known; a signed message cannot be reused for a different request; the previous-message hash detects messages dropped or reordered by the reply queue server. -Delayed replies (e.g., a credential issued after payment) are stored in the reply queue within message retention time and received when the client subscribes again; notification credentials can be added to the reply queue with existing commands. Messages in the reply queue that fail decryption or verification are acknowledged and dropped. +The first reply message returns from the request call. Later reply messages are delivered through the callback the application registered with the request. The exchange ends on a message with `more` set to false. The client deletes the reply queue on that message, on the request deadline, or when the application cancels the request. Deleting the queue stops further replies: later SEND commands from the service fail with AUTH. Server queue and message expiry limit the lifetime of a reply queue left after a client restart. ### Combined SKEY+SEND command @@ -150,24 +150,27 @@ senderAuthPublicKey = length x509encoded The command is authorized with the key it sets, as `SKEY`, and only accepted on messaging-mode queues. The server responds `OK` or `ERR`. It is idempotent in both parts: -- key part: as `SKEY` today - repeated command with the same key succeeds, different key fails with AUTH -- send part: the server stores a hash of the message body until the message is acknowledged; within that window a repeated command with the same body is not delivered again and is reported as delivered +- key part: as `SKEY` today - a repeated command with the same key succeeds, a different key fails with AUTH. +- send part: the server keeps the hash of the message until it is acknowledged, and a repeated command with the same message within that period is reported as delivered without delivering it again. -A retry arriving after the message was acknowledged is delivered as a duplicate and suppressed by the receiving agent (by message hash), so the server-side marker covers the common case and the agent covers the rest. +A repeat arriving after the message was acknowledged is delivered as a duplicate and discarded by the receiving agent by message hash, so the server-side hash covers the common case and the agent covers the rest. -It is used by the service for the first reply, and by the joining party in the fast connection handshake (currently `SKEY` then `SEND` confirmation), removing one command and round trip from both flows. +The service uses this command for the first reply message. The joining party uses it in the fast connection handshake, where it replaces `SKEY` followed by the `SEND` confirmation, removing one command and one round trip. ### Idempotency -Handled uniformly by the agent: it stores the request hash and sent replies for a window declared in link data (with a default), and re-sends stored replies for a repeated request without invoking the service application. This gives exactly-once execution over at-least-once delivery for all services; services with idempotent semantics of their own lose nothing. +The service agent identifies a request by its hash and keeps, for a fixed retention period it chooses (in the 1 to 24 hour range, in service configuration, not in link data), a record of the ordered reply messages it has sent and the reply queues subscribed under that hash. A repeat request with the same hash does not reach the service application: -The guarantee is bounded by the window: the client must not retry after the request deadline, and the deadline must not exceed the declared window. Stored replies may be evicted under storage pressure; a retry whose replies were evicted receives an error prompting a new request. Within the window a request is never executed twice. +- while the first request is being answered, the repeat is added to the record and receives the reply messages already sent, and each later message is sent to every reply queue in the record. +- after the operation completed, the repeat receives the whole stored sequence of reply messages, in order. + +This gives single execution over at-least-once delivery. The retention period bounds it: after the record is deleted, a request with the same hash is a new operation and runs again. The application does not rely on the transport for recovery across a restart; it keeps its own state and, after a restart, sends whatever request fits what it knows, which is often a different request (for example, "is this payment still pending" rather than a repeat of "start this payment"). ### Out of scope -- Continuity: sessions across requests are an application concern (tokens in request and reply bodies). -- Service-initiated messages: there is no standing channel; use a connection where push is needed. -- Abuse protection beyond existing queue quotas: services can require application-level credentials (e.g., a badge) in request bodies; rate limiting options are a separate discussion. +- Recovery across restart: the transport keeps no exchange across a client restart. The application persists its own state and sends a new request when it needs to. +- Service-initiated messages: there is no standing channel; use a connection where the service must reach the client without a request. +- Abuse protection beyond existing queue quotas: services can require application-level credentials (e.g., a badge) in the request payload; rate limiting is a separate discussion. - Scaling request reception: a single address queue bounds service throughput; distributing reception across multiple queues or relays (the existing `relays` field in contact link data) is a separate discussion. - Name resolution: existing addressing layer. From 5633bbfb139e5c1e0733b8b3f7756c8ae4f50001 Mon Sep 17 00:00:00 2001 From: Evgeny Date: Sun, 12 Jul 2026 16:16:58 +0100 Subject: [PATCH 3/6] corrections Co-authored-by: Evgeny --- rfcs/2026-07-11-service-rpc.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/rfcs/2026-07-11-service-rpc.md b/rfcs/2026-07-11-service-rpc.md index 1eb7a42df..074658e4f 100644 --- a/rfcs/2026-07-11-service-rpc.md +++ b/rfcs/2026-07-11-service-rpc.md @@ -12,13 +12,13 @@ This is the wrong primitive for most service interactions: 2. Privacy. A connection is a stable pairwise pseudonym. If a service were to use a duplex connection, it could link all requests made over it into a profile: search history in the directory, blockchain operations linked even when different on-chain keys are used, telemetry that becomes longitudinal tracking. The client also accumulates history that can be recovered from the device. Where continuity is needed, it can be provided in the application protocol (e.g., a token included in requests), without a transport-level identity. -3. Encryption. Messages sent to contact addresses outside an established connection (invitations) have a single layer of X25519 encryption. This was acceptable while such messages contained only a connection link and profile (though it is planned to be improved); it is not acceptable for service requests. +3. Encryption. Messages sent to contact addresses outside an established connection (invitations) have a single layer of X25519 encryption. This was acceptable while such messages contained only a connection link and profile (though it is planned to be improved); it is probably less acceptable for service requests. In-app service addresses should be stored as names resolving to links via the existing addressing layer (server host in the link authority, current link data retrieved with LGET), so that service links can be changed without redeploying the apps. Name resolution is already supported, and out of scope. ## Security objectives -1. Requests from the same client must not be linkable to each other by the service or by servers, and no state that outlives the exchange is created on either side. +1. Requests from the same client must not be linkable to each other by the service or by servers, and no long term state is created on either side in the transport layer. 2. Post-quantum resistant e2e encryption of requests and replies; decryptability of recorded requests must be bounded by key rotation, without changing the address. 3. Reply authenticity must be verifiable against the link; substitution, replay, dropping or reordering of replies by servers must be detectable. 4. A repeated request for the same operation must not be executed twice. @@ -171,7 +171,7 @@ This gives single execution over at-least-once delivery. The retention period bo - Recovery across restart: the transport keeps no exchange across a client restart. The application persists its own state and sends a new request when it needs to. - Service-initiated messages: there is no standing channel; use a connection where the service must reach the client without a request. - Abuse protection beyond existing queue quotas: services can require application-level credentials (e.g., a badge) in the request payload; rate limiting is a separate discussion. -- Scaling request reception: a single address queue bounds service throughput; distributing reception across multiple queues or relays (the existing `relays` field in contact link data) is a separate discussion. +- Scaling request reception: a single address queue bounds service throughput; distributing reception across multiple queues or relays (the existing `relays` field in contact link data) is a separate question, but it would fit well with name resolving to multiple addresses, both for redundancy, reliability and higher throughput. - Name resolution: existing addressing layer. [1]: https://tools.ietf.org/html/rfc5234 From d7226635319dd5fc0527a2378486bd84953c940a Mon Sep 17 00:00:00 2001 From: "Evgeny @ SimpleX Chat" <259188159+evgeny-simplex@users.noreply.github.com> Date: Mon, 13 Jul 2026 10:55:00 +0000 Subject: [PATCH 4/6] split rfcs, add plan --- .../2026-07-11-service-rpc-implementation.md | 380 +++++------------- plans/2026-07-12-address-dr-implementation.md | 195 +++++++++ rfcs/2026-07-11-service-rpc.md | 152 ++----- rfcs/2026-07-12-address-pqdr-keys.md | 95 +++++ rfcs/2026-07-12-queue-pq-encryption.md | 42 ++ rfcs/2026-07-12-smp-secure-send.md | 55 +++ 6 files changed, 534 insertions(+), 385 deletions(-) create mode 100644 plans/2026-07-12-address-dr-implementation.md create mode 100644 rfcs/2026-07-12-address-pqdr-keys.md create mode 100644 rfcs/2026-07-12-queue-pq-encryption.md create mode 100644 rfcs/2026-07-12-smp-secure-send.md diff --git a/plans/2026-07-11-service-rpc-implementation.md b/plans/2026-07-11-service-rpc-implementation.md index 68b9bff67..d3529f141 100644 --- a/plans/2026-07-11-service-rpc-implementation.md +++ b/plans/2026-07-11-service-rpc-implementation.md @@ -2,378 +2,204 @@ RFC: [../rfcs/2026-07-11-service-rpc.md](../rfcs/2026-07-11-service-rpc.md) -Types and instances below must encode exactly the ABNF in the RFC. Constructor, event, table and function names are provisional. +Depends on: [2026-07-12-address-dr-implementation.md](2026-07-12-address-dr-implementation.md). RPC establishes the double ratchet from the service address exactly as the address-DR plan does; this plan is the RPC layer on top of it. Steps named R2'/O2'/O3' below are from that plan. Constructor, event, table and function names are provisional. -## Versions - -- `VersionSMPA` (agent protocol): new version for `AgentRequest`/`AgentResponse` envelopes and the service key bundle in link data. -- `VersionSMPC` (SMP client): new version for the hybrid public header. -- `VersionSMP` (SMP protocol): new version for the `SSND` command. - -## Address type - `Simplex.Messaging.Agent.Protocol` - -```haskell -data ContactConnType = CCTContact | CCTChannel | CCTGroup | CCTRelay | CCTService - -ctTypeChar CCTService = 'S' -- 's' in links: link encoding lowercases, parsing uppercases - -ctTypeP 'S' = pure CCTService -``` +RPC is a one-directional short-lived DR exchange: the client establishes the ratchet from the address (address-DR requester), sends one request, receives one or more responses on its reply queue, and both sides tear the ratchet down after the final response. Unlike a connection, the service does not send a reply queue back and no persistent connection remains. -The contact type is currently fixed to `CCTContact` when the agent reconstructs a contact link (`Agent.hs`, `cslContact`). It becomes a stored property of the address (column `link_contact_type` below), read when the link is reconstructed and when an address queue message is dispatched. +## Versions -## Service key bundle in link data - `Simplex.Messaging.Agent.Protocol` +- `VersionSMPA` (agent protocol): the new `AgentServiceRequest`/`AgentServiceResponse` messages and the service address subtype. RPC requires the address-DR agent version, whose ratchet establishment it reuses. -```haskell -data ServiceKeyBundle = ServiceKeyBundle - { keyId :: ByteString, - reqDhKey :: C.PublicKeyX25519, - reqKemKey :: KEMPublicKey - } - -instance Encoding ServiceKeyBundle where - smpEncode ServiceKeyBundle {keyId, reqDhKey, reqKemKey} = smpEncode (keyId, reqDhKey, reqKemKey) - smpP = do - (keyId, reqDhKey, reqKemKey) <- smpP - pure ServiceKeyBundle {keyId, reqDhKey, reqKemKey} -``` +`SSND` (SMP protocol) and the hybrid queue header (SMP client) are not used here - they are separate RFCs (combined secure-send; queue-layer PQ). The ratchet provides post-quantum encryption, and the reply queue is secured with `SKEY` as in the address-DR owner path (O3'). -`UserContactData` gains a field, appended to the encoding, absent in data written by earlier versions. The agent sets it when it signs and updates link data; the application supplies only its own data. There is no retention value in link data - retention is service configuration (see Idempotency). +## Service address -```haskell -data UserContactData = UserContactData - { direct :: Bool, - owners :: [OwnerAuth], - relays :: [ConnShortLink 'CMContact], - userData :: UserLinkData, - serviceKeys :: Maybe ServiceKeyBundle - } - -instance Encoding UserContactData where - smpEncode UserContactData {direct, owners, relays, userData, serviceKeys} = - B.concat [smpEncode direct, smpEncodeList owners, smpEncodeList relays, smpEncode userData, smpEncode serviceKeys] - smpP = do - direct <- smpP - owners <- smpListP - relays <- smpListP - userData <- smpP - serviceKeys <- fromMaybe Nothing <$> optional smpP -- absent in data from earlier versions - _ <- A.takeByteString -- ignoring tail for forward compatibility - pure UserContactData {direct, owners, relays, userData, serviceKeys} -``` +A service address is a DR-advertising contact address (address-DR plan: `linkRatchetKey` in fixed data, `RatchetKeys` in mutable data) with a service subtype. There is no separate `ServiceKeyBundle` - requests are encrypted by establishing the ratchet against the advertised keys. -## SSND command - `Simplex.Messaging.Protocol`, `Simplex.Messaging.Server` +The subtype distinguishes RPC from a normal connection and drives the owner's handling. Reuse the `ContactConnType` char slot: ```haskell --- Protocol.hs -SSND :: SndPublicAuthKey -> MsgFlags -> MsgBody -> Command Sender - --- CommandTag, encoding "SSND" -SSND_ :: CommandTag Sender - --- encodeProtocol -SSND k flags msg -> e (SSND_, ' ', k, ' ', flags, ' ', Tail msg) - --- protocolP, in a new VersionSMP -SSND_ -> SSND <$> (smpP <* A.space) <*> (smpP <* A.space) <*> (unTail <$> smpP) +data ContactConnType = CCTContact | CCTChannel | CCTGroup | CCTRelay | CCTService +ctTypeChar CCTService = 'S' -- 's' in links +ctTypeP 'S' = pure CCTService ``` -`checkCredentials`: as `SKEY` - authorization required, sender entity ID. Server verification: `vc SSender (SSND k _ _) = verifySecure k` - authorized by the key it sets. +Stored on the address receive queue as `link_contact_type` (NULL means the current contact type). A service address rejects `AgentInvitation` and connection `AgentConfirmation`; a non-service address rejects service requests. -Server processing: `checkMode QMMessaging`, then `secureQueue_` (existing, idempotent), then deliver with deduplication. All queue store backends (STM, journal, PostgreSQL) keep the hash of the message delivered by `SSND` until it is acknowledged; a repeated `SSND` with an equal message hash responds `OK` without delivering it again; the hash is removed when the message is acknowledged. Server stats gain an `SSND` counter. +## RPC messages -Proxying: `proxySMPCommand` is polymorphic in the sender command, so `SSND` forwards through `PFWD`/`RFWD` without changes. - -## Hybrid public header - `Simplex.Messaging.Protocol` - -```haskell -data PubHeader - = PubHeader - { phVersion :: VersionSMPC, - phE2ePubKey :: Maybe C.PublicKeyX25519 - } - | PubHeaderHybrid - { phVersion :: VersionSMPC, - phKeyId :: Maybe ByteString, -- present in requests, absent in replies - phDhKey :: C.PublicKeyX25519, - phKemCt :: KEMCiphertext - } - -instance Encoding PubHeader where - smpEncode = \case - PubHeader v k_ -> smpEncode (v, k_) -- Maybe encodes as '0' / '1' key, as today - PubHeaderHybrid v kId_ k ct -> smpEncode (v, '2', kId_, k, ct) - smpP = do - v <- smpP - A.anyChar >>= \case - '0' -> pure $ PubHeader v Nothing - '1' -> PubHeader v . Just <$> smpP - '2' -> PubHeaderHybrid v <$> smpP <*> smpP <*> smpP - _ -> fail "bad PubHeader" -``` - -Secret derivation and encryption (`Simplex.Messaging.Crypto`, used from `Simplex.Messaging.Agent.Client`): +RPC adds two `AgentMessage` variants (Protocol.hs:883-889, parsed by `parseMessage`), parallel to `AgentConnInfoReply`, not new top-level envelopes. They are the decrypted content of the existing per-queue-e2e envelopes: the request is the `encConnInfo` of the address-DR `AgentConfirmation`; each response is the `encConnInfo` of an `AgentConfirmation` (first message to the reply queue) or the `encAgentMessage` of an `AgentMsgEnvelope` (later messages), all double-ratchet-encrypted. Their tags `Q`/`P` are the RFC's `agentRequest`/`agentResponse`. ```haskell -hybridSecret :: C.DhSecretX25519 -> KEMSharedKey -> C.SbKey -hybridSecret dh (KEMSharedKey k) = C.unsafeSbKey $ C.hkdf "" (C.dhBytes' dh <> BA.convert k) "SimpleXSMPQueue" 32 +data AgentMessage + = ... -- AgentConnInfo 'I', AgentConnInfoReply 'D', AgentRatchetInfo 'R', AgentMessage 'M' + | AgentServiceRequest (NonEmpty SMPQueueInfo) MsgBody -- 'Q': reply queue(s) + opaque request payload + | AgentServiceResponse Bool (NonEmpty MsgBody) -- 'P': final flag + one or more response payloads + +-- encoding (extends AgentMessage Encoding) +AgentServiceRequest qs body -> smpEncode ('Q', qs, Tail body) +AgentServiceResponse final bodies -> smpEncode ('P', final) <> smpEncodeList (map Large (L.toList bodies)) ``` -The body is encrypted with `C.sbEncrypt` (secret_box) rather than `C.cbEncrypt`, padded to the same lengths. The sending side generates an ephemeral X25519 pair, encapsulates to the recipient KEM key, and writes `PubHeaderHybrid`. The receiving side selects private keys by `phKeyId` for a request, or uses the reply queue keys for a reply, and computes the same secret. A request with an unknown key ID is discarded and counted. +One response message carries one or more response payloads, so responses known together are one message; responses over time are separate messages, each with `final = False` until the last. There is no signature and no previous-message hash: the ratchet, established against the `linkRatchetKey` committed by the link hash (address-DR authentication), authenticates each message and its message numbering detects dropping and reordering. The request payload is opaque application bytes (the chat command); the reply queue fields are outside it. -Size constants: the hybrid header adds about 1.2KB (KEM ciphertext 1039 bytes plus the ephemeral key). Define the request and reply body length constants from the existing padded body lengths at implementation. +The request hash used for idempotency is `SHA3-256` of the decrypted request payload (the `MsgBody` in `AgentServiceRequest`). -## Agent envelopes - `Simplex.Messaging.Agent.Protocol` +The `AgentMsgEnvelope` receive path (`agentClientMsg`, Agent.hs:3289) today expects only `AgentMessage APrivHeader aMessage`; it is extended to accept `AgentServiceResponse` for the stream after the first response. -```haskell -data AgentMsgEnvelope - = ... -- existing constructors - | AgentRequest - { agentVersion :: VersionSMPA, - replyQueue :: RequestReplyQueue, - requestPayload :: ByteString -- opaque, hashed - } - | AgentResponse - { agentVersion :: VersionSMPA, - signature :: C.Signature 'C.Ed25519, -- root key signature of signedResponse - signedResponse :: ByteString -- encoded SignedResponse, parsed after the signature is verified - } - -data RequestReplyQueue = RequestReplyQueue - { smpClientVersion :: VersionSMPC, - smpServer :: SMPServer, - senderId :: SMP.SenderId, -- sender ID of the reply queue - dhPublicKey :: C.PublicKeyX25519, -- reply queue X25519 key (its e2e key) - kemPublicKey :: KEMPublicKey -- reply queue sntrup761 key - } - -data SignedResponse = SignedResponse - { requestHash :: ByteString, -- SHA3-256 of requestPayload - prevMsgHash :: ByteString, -- SHA3-256 of the previous AgentResponse, empty in the first - more :: Bool, -- True if more reply messages follow - responses :: NonEmpty ByteString -- opaque application responses - } -``` - -The request does not include a key to secure the reply queue: the service generates its own sender key and secures the queue with `SSND`. `requestPayload` is the only hashed part; the reply queue fields are not hashed. +## Ratchet establishment - reuse of the address-DR flow -```haskell -instance Encoding AgentMsgEnvelope where - smpEncode = \case - ... -- existing constructors - AgentRequest {agentVersion, replyQueue, requestPayload} -> - smpEncode (agentVersion, 'Q', replyQueue, Tail requestPayload) - AgentResponse {agentVersion, signature, signedResponse} -> - smpEncode (agentVersion, 'P', signature, Tail signedResponse) - smpP = do - agentVersion <- smpP - smpP >>= \case - ... -- existing constructors - 'Q' -> do - (replyQueue, Tail requestPayload) <- smpP - pure AgentRequest {agentVersion, replyQueue, requestPayload} - 'P' -> do - (signature, Tail signedResponse) <- smpP - pure AgentResponse {agentVersion, signature, signedResponse} - -instance Encoding RequestReplyQueue where - smpEncode RequestReplyQueue {smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey} = - smpEncode (smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey) - smpP = do - (smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey) <- smpP - pure RequestReplyQueue {smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey} - -instance Encoding SignedResponse where - smpEncode SignedResponse {requestHash, prevMsgHash, more, responses} = - smpEncode (requestHash, prevMsgHash, more) <> smpEncodeList (map Large $ L.toList responses) - smpP = do - (requestHash, prevMsgHash, more) <- smpP - rs <- map unLarge <$> smpListP - responses <- maybe (fail "empty responses") pure $ L.nonEmpty rs - pure SignedResponse {requestHash, prevMsgHash, more, responses} -``` +Request (client), reusing the address-DR requester path (R2'/R3'): -Signing and verification follow the link data pattern (`Simplex.Messaging.Crypto.ShortLink`): +- Retrieve link data, reconstruct and negotiate the advertised `RcvE2ERatchetParamsUri`, create the reply queue Q_A (messaging mode, subscribed), establish the send ratchet (`generateSndE2EParams`, `pqX3dhSnd`, `initSndRatchet`, `createSndRatchet`). +- Send `AgentConfirmation {e2eEncryption_ = Just sndParams, ratchetKeyId = Just ratchetKeyId, encConnInfo = ratchetEncrypt(AgentServiceRequest (Q_A :| []) payload)}` to the address queue, unauthenticated (`agentCbEncryptOnce`). The client connection is `RcvConnection` (Q_A) with the send ratchet. -```haskell --- service: signing key is linkPrivSigKey from the address ShortLinkCreds -mkAgentResponse :: C.PrivateKeyEd25519 -> VersionSMPA -> SignedResponse -> AgentMsgEnvelope -mkAgentResponse rootPrivKey agentVersion resp = - let signedResponse = smpEncode resp - in AgentResponse {agentVersion, signature = C.sign' rootPrivKey signedResponse, signedResponse} - --- client: rootKey from the fixed link data -verifyAgentResponse :: C.PublicKeyEd25519 -> AgentMsgEnvelope -> Either AgentErrorType SignedResponse -verifyAgentResponse rootKey AgentResponse {signature, signedResponse} - | C.verify' rootKey signature signedResponse = parse smpP (AGENT A_MESSAGE) signedResponse - | otherwise = Left ... -- verification error -``` +Request (service), reusing the address-DR owner path (O1'/O2'): -## Reply queue - no new connection type +- `smpAddressConfirmation` selects the private keys by `ratchetKeyId`, `pqX3dhRcv`, `initRcvRatchet`, and `rcDecrypt` of `encConnInfo`, which also gives the connection its send side. The decrypted message is `AgentServiceRequest` (not `AgentConnInfoReply`), so the owner takes the RPC branch: create a `SndConnection` to Q_A (no Q_B, unidirectional) holding the ratchet, run idempotency (below), and either deliver `SREQ` or replay stored responses. This is receive-time establishment on unauthenticated input - the abuse bound of the address-DR plan ("Receive-time establishment, state, and abuse") applies unchanged. -The reply queue on the client is a receive connection (`RcvConnection`), extended to carry the KEM key and the computed hybrid secret. There is no new connection type and no flag on the connection: a client service request row (below) references the reply queue connection, and its presence identifies the connection as an RPC reply queue for dispatch and cleanup. +Response (service): send each response to Q_A under the ratchet. The first message to Q_A is `AgentConfirmation {e2eEncryption_ = Nothing, encConnInfo = ratchetEncrypt(AgentServiceResponse final bodies)}` (per-queue e2e is unestablished on Q_A - the address-DR first-message constraint), securing Q_A with `SKEY` using the service's own key (`agentSecureSndQueue`, Q_A is messaging mode); later messages are `AgentMsgEnvelope {encAgentMessage = ratchetEncrypt(AgentServiceResponse …)}`. After the `final = True` message, delete the send connection and its ratchet. -The reply queue reuses the existing receive queue fields. `e2ePrivKey` is its X25519 key, its public half is the queue address DH key sent in the request. Two nullable columns are added to `rcv_queues`: +Response (client): a message on Q_A (its `snd_service_requests` row marks it an RPC reply queue). The first is `AgentConfirmation … Nothing` and takes the address-DR `RcvConnection … Nothing` branch (R5'), extended to accept `AgentServiceResponse`; later ones are `AgentMsgEnvelope` and take the standard message path, extended to accept `AgentServiceResponse`. Each `agentRatchetDecrypt` advances the ratchet. The first response returns from the call; later responses go to the callback. On `final`, the deadline, or `cancelServiceRequest`, delete Q_A (`DEL`) and the reply connection. -- `reply_kem_priv_key` - the reply queue KEM private key. -- `reply_secret` - the hybrid secret, empty until the first reply, then set from the first reply's header, as the per-queue DH secret is set by `setRcvQueueConfirmedE2E` on the first message today. Later replies are decrypted with it by the standard receive path. +## Reply queue - the requester's DR connection -The service does not create a connection or queue for the reply queue. It sends replies directly to the reply queue address, as `sendInvitation` sends to a queue with no connection. +The reply queue is the address-DR requester connection: an `RcvConnection` whose receive queue is Q_A, with a ratchet. No `reply_kem_priv_key`/`reply_secret` columns (those were the queue-layer hybrid secret, not used here). A `snd_service_requests` row referencing this connection marks it an RPC reply queue for dispatch and cleanup; there is no new connection type. -## Database schema - `Agent/Store/SQLite/Migrations`, `Agent/Store/Postgres/Migrations` +## Database schema -One migration (`M20260712_service_rpc`), SQLite shown, PostgreSQL mirrors it; column types follow existing schema conventions. +One migration (`M20260712_service_rpc`), on top of the address-DR migration. SQLite shown, PostgreSQL mirrors it. ```sql --- address contact type (contact address queues), and reply queue keys (reply queues). --- NULL link_contact_type means 'A' (contact) for existing rows. -ALTER TABLE rcv_queues ADD COLUMN link_contact_type TEXT; -ALTER TABLE rcv_queues ADD COLUMN reply_kem_priv_key BLOB; -ALTER TABLE rcv_queues ADD COLUMN reply_secret BLOB; - --- service side: current and retired bundle private keys. -CREATE TABLE service_address_keys( - service_address_key_id INTEGER PRIMARY KEY AUTOINCREMENT, - conn_id BLOB NOT NULL REFERENCES connections ON DELETE CASCADE, - key_id BLOB NOT NULL, - dh_priv_key BLOB NOT NULL, - kem_priv_key BLOB NOT NULL, - created_at TEXT NOT NULL, - retired_at TEXT -- set on rotation, row deleted when the key retention period ends -); -CREATE UNIQUE INDEX idx_service_address_keys ON service_address_keys(conn_id, key_id); +-- service subtype on the address receive queue (address-DR adds the ratchet key columns) +ALTER TABLE rcv_queues ADD COLUMN link_contact_type TEXT; -- NULL = current contact type -- client side: one pending request per reply queue connection. CREATE TABLE snd_service_requests( snd_service_request_id INTEGER PRIMARY KEY AUTOINCREMENT, - conn_id BLOB NOT NULL REFERENCES connections ON DELETE CASCADE, -- reply queue connection - request_hash BLOB NOT NULL, - root_key BLOB NOT NULL, -- to verify replies - last_reply_hash BLOB, -- updated as reply messages arrive + conn_id BLOB NOT NULL REFERENCES connections ON DELETE CASCADE, -- reply queue (RcvConnection) with the ratchet deadline TEXT NOT NULL, created_at TEXT NOT NULL ); --- service side: one record per distinct request hash on an address. +-- service side: one record per distinct request hash on a service address. CREATE TABLE rcv_service_requests( rcv_service_request_id INTEGER PRIMARY KEY AUTOINCREMENT, - conn_id BLOB NOT NULL REFERENCES connections ON DELETE CASCADE, -- address connection + address_conn_id BLOB NOT NULL REFERENCES connections ON DELETE CASCADE, -- the service address connection request_hash BLOB NOT NULL, - ended INTEGER NOT NULL DEFAULT 0, -- a reply message with more = False was sent + ended INTEGER NOT NULL DEFAULT 0, -- a response with final = True was produced expires_at TEXT NOT NULL, -- created + retention created_at TEXT NOT NULL ); -CREATE UNIQUE INDEX idx_rcv_service_requests ON rcv_service_requests(conn_id, request_hash); +CREATE UNIQUE INDEX idx_rcv_service_requests ON rcv_service_requests(address_conn_id, request_hash); --- service side: ordered signed reply messages for a request; the signed envelope --- is independent of the reply queue, so it is stored once and encrypted per queue on send. -CREATE TABLE rcv_service_replies( - rcv_service_reply_id INTEGER PRIMARY KEY AUTOINCREMENT, +-- service side: ordered response payloads (plaintext) for a request; re-encrypted per reply +-- connection because each request establishes its own ratchet, so ciphertext is not reusable. +CREATE TABLE rcv_service_responses( + rcv_service_response_id INTEGER PRIMARY KEY AUTOINCREMENT, rcv_service_request_id INTEGER NOT NULL REFERENCES rcv_service_requests ON DELETE CASCADE, - reply_seq INTEGER NOT NULL, - more INTEGER NOT NULL, - signed_reply BLOB NOT NULL -- encoded AgentResponse + response_seq INTEGER NOT NULL, + final INTEGER NOT NULL, + response_bodies BLOB NOT NULL -- plaintext response payloads for this message (encoded NonEmpty MsgBody) ); -CREATE UNIQUE INDEX idx_rcv_service_replies ON rcv_service_replies(rcv_service_request_id, reply_seq); +CREATE UNIQUE INDEX idx_rcv_service_responses ON rcv_service_responses(rcv_service_request_id, response_seq); --- service side: reply queues subscribed under a request (the first, and any repeat). -CREATE TABLE rcv_service_reply_queues( - rcv_service_reply_queue_id INTEGER PRIMARY KEY AUTOINCREMENT, +-- service side: reply connections subscribed under a request (the first, and any repeat while pending +-- or after completion). Each is a SndConnection to a reply queue with its own ratchet (in ratchets table). +CREATE TABLE rcv_service_reply_conns( + rcv_service_reply_conn_id INTEGER PRIMARY KEY AUTOINCREMENT, rcv_service_request_id INTEGER NOT NULL REFERENCES rcv_service_requests ON DELETE CASCADE, - host TEXT NOT NULL, - port TEXT NOT NULL, - snd_id BLOB NOT NULL, - server_key_hash BLOB, - reply_secret BLOB NOT NULL, -- hybrid secret to this reply queue - secured INTEGER NOT NULL DEFAULT 0, -- reply queue secured with SSND + conn_id BLOB NOT NULL REFERENCES connections ON DELETE CASCADE, -- SndConnection to the reply queue, holds the ratchet last_sent_seq INTEGER NOT NULL DEFAULT 0 ); ``` +The service's address ratchet keys are the address-DR `address_ratchet_keys` table - not duplicated here. + ## Agent API - `Simplex.Messaging.Agent` Service side: ```haskell --- Creates a contact connection with CCTService link type, generates the root key --- and the first key bundle, composes and signs link data. +-- Creates a DR-advertising contact address with CCTService subtype (address-DR createServiceAddress +-- plus the subtype), generating the link ratchet key and the first RatchetKeys row. createServiceAddress :: AgentClient -> UserId -> UserLinkData -> AE (ConnId, ConnShortLink 'CMContact) --- Re-signs and updates user data with LSET, keeping the agent-owned key bundle. -updateServiceAddressData :: AgentClient -> ConnId -> UserLinkData -> AE () - --- Generates a new bundle, updates link data, retires the previous keys until the retention period ends. --- Also runs on a schedule (config) while the address is subscribed. -rotateServiceAddressKeys :: AgentClient -> ConnId -> AE () - --- Sends one reply message with a list of responses; more = False ends the exchange. --- The first message to each reply queue secures it with SSND; later messages use SEND. -sendServiceReply :: AgentClient -> ServiceRequestRef -> Bool -> NonEmpty ByteString -> AE () +-- Sends one response message with one or more payloads (final = True ends the exchange). The first +-- message to each reply connection secures the reply queue with SKEY; later ones use SEND. Appends to +-- rcv_service_responses. +sendServiceReply :: AgentClient -> ServiceRequestRef -> Bool -> NonEmpty MsgBody -> AE () ``` -Address deletion: existing `deleteConnection`. +Key rotation and update use the address-DR `rotateRatchetKeys`/link-data update; address deletion is `deleteConnection`. -Client side (name resolution to a link is an existing API; the link must have `CCTService` type and a key bundle in link data): +Client side (name resolution to a link is an existing API; the link must be a `CCTService` DR-advertising address): ```haskell --- Retrieves link data, creates the reply queue, sends the request, and waits for the first --- reply message up to the deadline. The callback receives later reply messages while the process runs. +-- Establishes the ratchet from the address, creates the reply queue, sends the request, and waits for +-- the first response up to the deadline. The callback receives later responses while the process runs. sendServiceRequest :: AgentClient -> UserId -> ConnShortLink 'CMContact -> UTCTime -> - ByteString -> (ServiceReply -> IO ()) -> AE ServiceReply + MsgBody -> (ServiceResponse -> IO ()) -> AE ServiceResponse --- Deletes the reply queue and the request row, and drops the callback. cancelServiceRequest :: AgentClient -> ConnId -> AE () -data ServiceReply = ServiceReply {responses :: NonEmpty ByteString, more :: Bool} +data ServiceResponse = ServiceResponse {bodies :: NonEmpty MsgBody, final :: Bool} ``` -The first reply message returns from `sendServiceRequest`; later messages are delivered through the callback. Both the waiting call and the callback are held in an in-memory map in `AgentClient`, keyed by the reply queue connection, and filled by the receive path. They do not survive a restart. +The waiting call and the callback are held in an in-memory map in `AgentClient`, keyed by the reply queue connection, filled by the receive path; they do not survive a restart. -Service side events (`AEvent`, entity is the address connection): +Service side event (`AEvent`, entity is the service address connection): ```haskell -SREQ :: ServiceRequestRef -> ByteString -> AEvent AEConn -- request received, payload for the bot +SREQ :: ServiceRequestRef -> MsgBody -> AEvent AEConn -- request payload for the bot ``` -`ServiceRequestRef` identifies the request record; the bot passes it to `sendServiceReply`. +`ServiceRequestRef` identifies the `rcv_service_requests` record; the bot passes it to `sendServiceReply`. + +Rejection reuses `AgentRejection` (communicating-rejection RFC) as an `AgentServiceResponse`-level refusal or a dedicated variant - decided with that RFC. -## Agent processing - `Simplex.Messaging.Agent`, `Simplex.Messaging.Agent.Client` +## Agent processing Client side: -- `sendServiceRequest`: retrieve link data (`LGET`, proxied per config) for every request - no caching; create the reply queue (`NEW`, messaging mode, subscribed) with an added KEM key; write the `snd_service_requests` row; send the request (proxied per config); wait on the in-memory sink for the first reply until the deadline. -- Reply processing in `processSMPTransmissions`: a message on a receive queue that has a `snd_service_requests` row is a reply. Decrypt (the first message sets `reply_secret` from its header), verify with `verifyAgentResponse`, check the request hash and the previous-message hash, update `last_reply_hash`, deliver to the waiting call or the callback. A message that fails verification is acknowledged and discarded. On `more = False`, or on the deadline, or on `cancelServiceRequest`, delete the reply queue (`DEL`) and the row. -- `cleanupManager` gains one step: delete `snd_service_requests` past the deadline and mark their reply queue connections deleted; the existing deleted-connections step sends `DEL` and removes them. After a restart every row is stale, so this step removes all reply queues left behind. +- `sendServiceRequest`: retrieve link data per request (proxied per config); establish the ratchet and create Q_A (address-DR R2'); write the `snd_service_requests` row; send the `AgentServiceRequest` confirmation (proxied per config); wait on the in-memory sink for the first response until the deadline. +- Response processing in `processSMPTransmissions`: a message on a queue with a `snd_service_requests` row is a response; `agentRatchetDecrypt` (advancing the ratchet), parse `AgentServiceResponse`, deliver to the waiting call or the callback. On `final`/deadline/cancel, delete Q_A and the reply connection. +- `cleanupManager`: delete `snd_service_requests` past the deadline and mark their reply connections deleted; the existing deleted-connections step sends `DEL`. After a restart every row is stale, so this removes reply queues left behind. Service side: -- Address queue message dispatch reads `link_contact_type`: a service address accepts only `AgentRequest` and rejects invitations and confirmations; a non-service address rejects `AgentRequest`. -- On `AgentRequest`: select bundle private keys by `keyId`, decrypt, compute the request hash. Look up `rcv_service_requests` by (address conn, hash): - - new: insert the request, insert a `rcv_service_reply_queues` row for the reply queue in the request, deliver `SREQ` to the bot. - - existing: insert a `rcv_service_reply_queues` row for the reply queue in this request, and send it every `rcv_service_replies` message already stored, in order. -- `sendServiceReply`: append a `rcv_service_replies` message (sign with the address root key), then send it to every reply queue of the request - `SSND` for a queue not yet secured, `SEND` after - encrypting the stored envelope to each queue's `reply_secret`; set `ended` when `more = False`. -- `cleanupManager` gains one step: delete `rcv_service_requests` past `expires_at` (cascading to replies and reply queues) and `service_address_keys` past the key retention period. +- Address-queue dispatch on `link_contact_type`: a service address routes `AgentConfirmation` with `ratchetKeyId` to the RPC handler (address-DR `smpAddressConfirmation` producing `AgentServiceRequest`); it rejects `AgentInvitation` and connection confirmations. +- On `AgentServiceRequest`: establish the ratchet (address-DR O2'), create the `SndConnection` to Q_A, compute the request hash, look up `rcv_service_requests` by (address conn, hash): + - new: insert the request, insert a `rcv_service_reply_conns` row, deliver `SREQ` to the bot. + - existing: insert a `rcv_service_reply_conns` row and send it every stored `rcv_service_responses` in order, each `AgentServiceResponse` re-encrypted under this connection's ratchet. +- `sendServiceReply`: append a `rcv_service_responses` row, then send `AgentServiceResponse` to every reply connection of the request under its ratchet (`SKEY`+`SEND` for the first message to a connection, `SEND` after); set `ended` on `final`. +- `cleanupManager`: delete `rcv_service_requests` past `expires_at` (cascading to responses and reply connections, and their ratchets/queues). + +Configuration (`AgentConfig`): default request deadline, request retention period (1 to 24 hours). Ratchet key rotation interval is the address-DR config. + +Errors reuse `AgentErrorType` (e.g. `CMD PROHIBITED` for a link that is not a service address). -Configuration (`AgentConfig`): default request deadline, request retention period (1 to 24 hours), key rotation interval. +## Idempotency -Errors: API failures reuse `AgentErrorType` (for example `CMD PROHIBITED` for a link that is not a service address). A new constructor is added only if the chat library needs to distinguish service request failures - decided at implementation. +The service identifies a request by its hash and keeps, for the retention period (config, not in link data), the ordered response payloads (`rcv_service_responses`) and the reply connections under that hash (`rcv_service_reply_conns`). A repeat request (same payload, therefore same hash) establishes its own ratchet and reply connection and does not reach the bot: while pending it is added and receives the responses so far and each later one; after completion it receives the whole stored sequence. Responses are stored as plaintext and re-encrypted per reply connection because each request has its own ratchet. This gives single execution over at-least-once delivery, bounded by the retention period. ## Correlation and chat -A reply is connected to its request by the reply queue: one request has one reply queue, and every message in that queue is a reply to that request. The request hash is used only in the reply signature, to bind a reply to the request content. The application ID is content inside `requestPayload`, used only by the application to make two requests equal or different; the agent does not read it. +A response is connected to its request by the reply queue (one request, one reply queue, one ratchet). The request hash is only the idempotency key. The application ID is content inside the request payload, used only by the application to make two requests equal or different; the agent does not read it. -Both ends are chat bots on the chat library. The chat library serializes a service command into `requestPayload` and deserializes the responses; the agent transports them and correlates by reply queue. The chat framework's `chatServiceCalls` correlation by `(AgentConnId, SharedMsgId)` is not used - the agent correlates by reply queue and the call returns the first reply directly. +Both ends are chat bots on the chat library, which serializes a service command into the request payload and deserializes the responses; the agent transports them and correlates by reply queue. The chat framework's `chatServiceCalls` correlation is not used. ## Tests -- Encoding roundtrips: envelopes, `ServiceKeyBundle` in `UserContactData`, `PubHeader` (all three variants), `SSND`, `SignedResponse` with one and several responses. -- Server: `SSND` on messaging and contact queues, repeated `SSND` before and after ACK, a different key, via proxy. -- Agent end-to-end: a request with one reply message; several responses in one message; responses in several messages delivered to the callback; a repeat request while pending coalesces onto the same operation; a repeat request after completion receives the stored messages without a second execution; key rotation with an old key still accepted while kept; a request whose key was deleted fails at the deadline; deadline; cancellation; a restart deletes reply queues; envelope rejection on both address types. +- Encoding roundtrips: `AgentServiceRequest`, `AgentServiceResponse` in `AgentMessage` (one and several response bodies); the service subtype in `UserContactData`/link. +- End to end (on the address-DR machinery): a request with one response; several responses streamed to the callback; `pqEncryption` on (hybrid) and off (X448-only) per the advertised keys; the request payload never travels under per-queue-only encryption. +- Idempotency: a repeat while pending coalesces onto the same operation; a repeat after completion receives the stored responses without a second `SREQ`; both under fresh ratchets. +- Lifecycle: the reply connection and the service ratchet are deleted after `final`; deadline; cancellation; a restart deletes client reply queues. +- Rejection and rejection of the wrong envelope on a service vs non-service address. ## Phases -1. SMP protocol: `SSND`, hybrid public header, version bumps, server dedup hash, server tests. -2. Agent: address type, key bundle in link data, envelopes, reply queue columns, schema migration, `createServiceAddress`, `sendServiceRequest`, reply processing, cleanup. -3. Agent: service-side request store, `sendServiceReply`, coalescing and repeats, key rotation, end-to-end tests. -4. Adoption: `SSND` in the fast connection handshake; the hybrid scheme for invitations and confirmations. +1. Service address subtype and dispatch on top of the address-DR flow; `AgentServiceRequest`/`AgentServiceResponse` messages; `createServiceAddress`. +2. Client: `sendServiceRequest`, reply-queue reception and callback, cleanup. +3. Service: request store, `sendServiceReply`, idempotency (coalescing and repeats under fresh ratchets), end-to-end and idempotency tests. diff --git a/plans/2026-07-12-address-dr-implementation.md b/plans/2026-07-12-address-dr-implementation.md new file mode 100644 index 000000000..dc10412dd --- /dev/null +++ b/plans/2026-07-12-address-dr-implementation.md @@ -0,0 +1,195 @@ +# Establishing the double ratchet from address data - implementation plan + +RFC: [../rfcs/2026-07-12-address-pqdr-keys.md](../rfcs/2026-07-12-address-pqdr-keys.md) + +All references are to the current tree. Names of new constructors, fields, tables and functions are provisional. + +Goal: a contact address advertises the owner's X3DH parameters in link data; a requester establishes the double ratchet in its first message, so that message and the profile in it are under the ratchet with post-quantum protection. The change reuses the invitation/confirmation machinery, with the requester in the joiner role and the owner in the initiator role - opposite to today's contact flow, but every message and code path below is reused. + +## Part 1 - the current contact-address handshake, step by step + +Requester Alice connects to owner Bob's contact address. Q_A is Alice's receive queue (Bob to Alice), Q_B is Bob's receive queue (Alice to Bob). + +Requester side, in `joinConnSrv … CRContactUri` (Agent.hs:1398-1428): + +- R1. `compatibleContactUri` (Agent.hs:1370) - version check, yields the address queue `SMPQueueInfo`. +- R2. `mkJoinInvitation` (Agent.hs:1411): creates or reuses the receive queue Q_A; `getRatchetX3dhKeys` or `generateRcvE2EParams` produces Alice's Rcv X3DH parameters, stored by `createRatchetX3dhKeys` (Agent.hs:1424); builds `cReq = CRInvitationUri crData aliceRcvParams` (Agent.hs:1426). +- R3. `sendInvitation` (Agent.hs:1408; Agent/Client.hs:1924-1934): sends `AgentInvitation {connReq = cReq, connInfo = aliceProfile}` to the address queue, per-queue encrypted with a fresh ephemeral key by `agentCbEncryptOnce` (Agent/Client.hs:1929-1934), unauthenticated. **`connInfo` (Alice's profile) is under the per-queue X25519 layer only - the gap this plan closes.** + +Owner side, receiving on the contact address: + +- O1. `processClientMsg` dispatch (Agent.hs:3185): state `(Nothing, Just e2ePubKey)`, `(PHEmpty, AgentInvitation {connReq, connInfo})` -> `smpInvitation` (Agent.hs:3186). +- O2. `smpInvitation` (Agent.hs:3610): stores an `Invitation`, emits `REQ` with Alice's `connInfo`. +- O3. `acceptContact'` (Agent.hs:1477): `getInvitation`, then `joinConn` with Alice's `connReq` (Agent.hs:1480). +- O4. `joinConnSrv … CRInvitationUri` (Agent.hs:1383) -> `startJoinInvitation` (Agent.hs:1395). +- O5. `startJoinInvitation` (Agent.hs:1310-1350): creates Bob's send queue to Q_A (`newSndQueue`, Agent.hs:1335); `createRatchet_` (Agent.hs:1343-1350) runs `generateSndE2EParams`, `pqX3dhSnd` against Alice's Rcv parameters, `initSndRatchet`, `createSndRatchet`. +- O6. `secureConfirmQueue` (Agent.hs:1396, 3747-3765): `agentSecureSndQueue` secures Q_A with `SKEY` (Agent.hs:3749); `mkAgentConfirmation` (Agent.hs:3780-3785) calls `createReplyQueue` to create Bob's receive queue Q_B and returns `AgentConnInfoReply (Q_B :| []) bobInfo`; `mkConfirmation` ratchet-encrypts it and wraps `AgentConfirmation {e2eEncryption_ = Just bobSndParams, encConnInfo}`; `sendConfirmation` sends it to Q_A. This is confirmation #1. + +Requester side, receiving confirmation #1 on Q_A: + +- R4. dispatch (Agent.hs:3181-3183): state `(Nothing, Just e2ePubKey)`, `AgentConfirmation` -> `smpConfirmation`. +- R5. `smpConfirmation`, initiating-party branch `RcvConnection … Just e2eEncryption` (Agent.hs:3405-3444): `getRatchetX3dhKeys`, `pqX3dhRcv` (Agent.hs:3408), `initRcvRatchet` (Agent.hs:3411), `createRatchet` (Agent.hs:3436), `setRcvQueueConfirmedE2E` (Agent.hs:3440); decrypts `AgentConnInfoReply` (Agent.hs:3420); `processConf` emits `CONF` (Agent.hs:3444). +- R6. `allowConnection'` (Agent.hs:1467-1474): `acceptConfirmation`, then `ICAllowSecure` secures Q_A with Bob's sender key. +- R7. `connectReplyQueues` (Agent.hs:3724-3737): `upgradeConn` creates Alice's send queue to Q_B; `agentSecureSndQueue` secures Q_B; `enqueueConfirmation … Nothing` (Agent.hs:3733) stores `AgentConnInfo aliceInfo` and sends `AgentConfirmation {e2eEncryption_ = Nothing, encConnInfo}` to Q_B. This is confirmation #2. + +Owner side, receiving confirmation #2 on Q_B: + +- O7. dispatch (Agent.hs:3182): `AgentConfirmation` -> `smpConfirmation`. +- O8. `smpConfirmation`, accepting-party branch `DuplexConnection … Nothing` (Agent.hs:3447-3462): `agentRatchetDecrypt` with the established ratchet; `AgentConnInfo` -> `INFO` (Agent.hs:3452); `ICDuplexSecure` or `CON`. + +`HELLO`/`CON` completion follows via `helloMsg` (Agent.hs:3466). + +## Part 2 - the DR-from-address handshake, mapped to Part 1 + +The address advertises Bob's Rcv X3DH parameters in link data (Part 3). Alice, when the address advertises them and versions are compatible, takes the joiner role; Bob takes the initiator role. + +Requester side - a new branch in `joinConnSrv … CRContactUri`, chosen when the fetched link data has `ratchetKeys`: + +- R2'. Replaces R2/R3. Read the advertised `linkRatchetKey`, `ratchetKeyId`, `e2eVRange`, `prekey`, and optional `kemKey` from link data; reconstruct Bob's `RcvE2ERatchetParamsUri 'C.X448` and negotiate the concrete version with `compatibleVersion` against the client e2e range, as `compatibleInvitationUri` does (Agent.hs:1362-1368). Create the receive queue Q_A subscribed (`newRcvQueue` with `subMode`), messaging mode so Bob can secure it. Run `generateSndE2EParams` (with `replyKEM_` to accept the KEM only when advertised), `pqX3dhSnd` against the negotiated parameters, `initSndRatchet`, `createSndRatchet` - the body of `createRatchet_` (Agent.hs:1343-1350), with parameters from link data rather than a received invitation. +- R3'. Build `AgentConfirmation {e2eEncryption_ = Just aliceSndParams, ratchetKeyId = Just ratchetKeyId, encConnInfo = ratchetEncrypt(AgentConnInfoReply (Q_A :| []) aliceProfile)}` - the `mkAgentConfirmation`/`mkConfirmation` bodies (Agent.hs:3780-3765) with the reply queue being Alice's own Q_A. Send it to the address queue unauthenticated with `agentCbEncryptOnce`, as `sendInvitation` sends (Agent/Client.hs:1929-1934). **Alice's profile is now inside `encConnInfo`, under the ratchet.** Alice's connection is `RcvConnection` (Q_A) with a send ratchet, until she receives Q_B. + +Owner side - a new dispatch branch and a new receive handler: + +- O1'. In `processClientMsg` (Agent.hs:3176-3187), add a branch in state `(Nothing, Just e2ePubKey)`: an `AgentConfirmation` with `ratchetKeyId = Just _` on a `ContactConnection` -> `smpAddressConfirmation` (new). It must be placed **before** the existing `(PHEmpty, AgentConfirmation) | senderCanSecure queueMode` case (Agent.hs:3182-3184), because a contact-address queue is `QMContact` (not `senderCanSecure`) and would otherwise fall into `prohibited "handshake: missing sender key"` (Agent.hs:3184). The address queue's `e2eDhSecret` stays `Nothing` (it is never set for a contact address - `smpInvitation` does not set it, Agent.hs:3609-3622), so every request is decrypted with its own ephemeral key via this `(Nothing, Just e2ePubKey)` path. +- O2'. `smpAddressConfirmation` (new, modeled on `smpConfirmation` initiating branch, Agent.hs:3405-3444): select the private prekey and, if any, KEM keypair by `ratchetKeyId` from `address_ratchet_keys`; `pqX3dhRcv` with the `linkRatchetKey` private key, the selected prekey private key, and the optional KEM keypair (as `PrivateRKParamsProposed`) against `aliceSndParams`; `initRcvRatchet` (its `PQSupport` follows from whether the address advertises a KEM and version compatibility, as `smpConfirmation` derives `pqSupport'`, Agent.hs:3410); `rcDecrypt` of `encConnInfo` performs the first ratchet step, giving the connection its send side too (as it does for the initiator today), so the owner can later reply; `createRatchet` stores the post-decrypt ratchet under a **new connection created now** for this request. Parse `AgentConnInfoReply (Q_A :| []) aliceProfile`, store a pending-request record referencing the new connection, its ratchet, and Q_A (not a `connReq`), and emit `REQ`. An unknown or expired `ratchetKeyId`, or a decryption failure: discard and acknowledge, as an undecryptable message is dropped today. This establishes ratchet state on unauthenticated input before the user accepts - see "Receive-time establishment, state, and abuse". +- O3'. `acceptContact'` for a DR-established request - a new branch that continues the ratchet instead of `joinConn`: reuse `mkAgentConfirmation` (Agent.hs:3780-3785) to create Bob's receive queue Q_B and return `AgentConnInfoReply (Q_B :| []) bobInfo`; create Bob's send queue to Q_A (`newSndQueue`, generating Bob's own sender key) and secure Q_A with `SKEY` using that key (`agentSecureSndQueue`, valid because Q_A is messaging mode) - the securing key is Bob's own, not taken from Alice's message; send the response to Q_A as `AgentConfirmation {e2eEncryption_ = Nothing, ratchetKeyId = Nothing, encConnInfo = ratchetEncrypt(AgentConnInfoReply (Q_B :| []) bobInfo)}`, per-queue encrypted with `agentCbEncryptOnce` (first message to Q_A). The reply content is `AgentConnInfoReply`, not `AgentConnInfo`: it takes the `mkAgentConfirmation` path with `e2eEncryption_ = Nothing`, not the `enqueueConfirmation` path (which produces `AgentConnInfo`, Agent.hs:3789). + +Requester side, receiving the response on Q_A: + +- R5'. `smpConfirmation` needs a new branch `RcvConnection … Nothing` (today only `RcvConnection … Just` and `DuplexConnection … Nothing` exist, Agent.hs:3403-3447): Alice already holds the send ratchet, so `agentRatchetDecrypt` advances it and creates the receive side; `setRcvQueueConfirmedE2E` on Q_A so later messages use the stored per-queue secret (as the current branches do, Agent.hs:3440,3455); parse `AgentConnInfoReply (Q_B :| []) bobInfo`; connect Q_B (create Alice's send queue to Q_B, `agentSecureSndQueue` it, upgrade `RcvConnection` to `DuplexConnection`); emit `INFO`. The accepting-party branch (Agent.hs:3447) must also accept `AgentConnInfoReply`, not only `AgentConnInfo` (Agent.hs:3451, 3462), for the reply-queue case. +- R6'/completion. The response went to Q_A, so Bob has no signal that Alice secured Q_B and would never reach `CON` on its own. The essential extra message is Alice -> Q_B: after R5' Alice sends `HELLO` on Q_B, and `helloMsg` (Agent.hs:3466) sets Q_B active on Bob's side and emits Bob's `CON`. Alice reaches `CON` from the response (Q_A active, Q_B secured). Whether Bob also needs to send `HELLO` on Q_A for Alice's `CON`, or the response suffices, follows the existing `helloMsg`/fast-path completion and is confirmed at implementation - the fixed point is that Alice must send on Q_B. + +Net code touch points: `joinConnSrv` (new requester branch), `processClientMsg` (new owner dispatch), `smpConfirmation` (new `RcvConnection … Nothing` branch and `AgentConnInfoReply` acceptance), `acceptContact'` (new continue-ratchet branch), a new `smpAddressConfirmation`, and the link data and storage of Part 3-4. + +### Receive-time establishment, state, and abuse + +This is the substantive departure from the current flow. Today `smpInvitation` creates only a lightweight `NewInvitation` and emits `REQ` (Agent.hs:3618-3621); no connection or ratchet exists until the user accepts. For DR the request is under the ratchet, so to show the requester's profile in `REQ` the owner must decrypt it, which means establishing the ratchet at **receive**, before accept. + +Design decision (Q1): decrypt at receive. Both use cases need the request content at `REQ` - a person decides to accept from the profile, and a service bot needs the request payload to act. Deferring decryption to accept would make `REQ` contentless and does not fit the service case, so it is not done. + +Consequences: + +- A connection is created at receive to hold the ratchet (ratchets are keyed by `conn_id`). O2' creates a `NewConnection`, stores the post-decrypt ratchet under it, and a pending-request record with Q_A's address; O3' (accept) adds Bob's send queue to Q_A and receive queue Q_B, upgrading to `DuplexConnection`; reject deletes the connection, its ratchet, and the record. +- Per incoming `AgentConfirmation` the owner does one `pqX3dhRcv` (three DH plus, with PQ, one `sntrup761` decapsulation) and one `rcDecrypt`, on unauthenticated input, and creates a connection row and a ratchet row - more CPU and state than the current `NewInvitation`. + +Abuse (Q2): a contact address already accepts and processes unauthenticated invitations today, so this is a degree-worse version of an existing surface, not a new class. It is bounded by the address queue quota (an attacker fills it, the owner drains and acknowledges) and, optionally, by basic auth on the address (already supported for contact addresses, `optBasicAuth`). The pending-request state is bounded by a request TTL: `cleanupManager` deletes pending DR connections never accepted or rejected past that TTL (Part 4 cleanup). Proof-of-work or a stricter gate can be added later; it is out of scope here and noted as a follow-up. + +The `acceptContact'` dispatch branches on the pending record: a classic invitation (`connReq` present) takes the current `joinConn` path (O3-O6); a DR request (pending connection and ratchet present, no `connReq`) takes the continue-ratchet path (O3'). + +## Part 3 - types and link data + +### Fixed data - link ratchet key + +Appended to `FixedLinkData` (Protocol.hs:1824); the encoding already stops at a trailing tail (Protocol.hs:1928), so earlier versions ignore it: + +```haskell +data FixedLinkData c = FixedLinkData + { agentVRange :: VersionRangeSMPA, + rootKey :: C.PublicKeyEd25519, + linkConnReq :: ConnectionRequestUri c, + linkEntityId :: Maybe ByteString, + linkRatchetKey :: Maybe C.PublicKeyX448 -- stable X448 key, the first X3DH key + } +``` + +### Mutable data - ratchet keys bundle + +Appended to `UserContactData` (Protocol.hs:1840); the encoding stops at a trailing tail (Protocol.hs:1981), so earlier versions ignore it: + +```haskell +data RatchetKeys = RatchetKeys + { ratchetKeyId :: ByteString, -- identifies this bundle; changes on rotation + e2eVRange :: VersionRangeE2E, -- advertised e2e version range, for negotiation + prekey :: C.PublicKeyX448, + kemKey :: Maybe KEMPublicKey -- opt-in, as PQ is opt-in in the ratchet (PQSupport) + } + +data UserContactData = UserContactData + { direct :: Bool, owners :: [OwnerAuth], relays :: [ConnShortLink 'CMContact], + userData :: UserLinkData, + ratchetKeys :: Maybe RatchetKeys + } +``` + +Reconstruction produces `RcvE2ERatchetParamsUri 'C.X448` (a version range, `E2ERatchetParamsUri VersionRangeE2E k1 k2 (Maybe kem)`), not concrete params: `e2eVRange` as the range, `linkRatchetKey` (from fixed data) as the first key, `prekey` as the second, `kemKey` as the optional KEM parameter. The requester negotiates the concrete version with `compatibleVersion` against its own e2e range, exactly as `compatibleInvitationUri` does for an invitation (Agent.hs:1362-1368), then uses the resulting `RcvE2ERatchetParams` in `pqX3dhSnd`. The KEM is optional: with `kemKey = Nothing` the ratchet is X448-only, as when `PQSupport` is off; with `Just` it is hybrid, matching `E2ERatchetParams … (Maybe (RKEMParams s))` (Ratchet.hs:220-221) and `generateRcvE2EParams`'s `PQSupport` gate (Ratchet.hs:439-445). + +The owner does not use `generateRcvE2EParams` (which generates both keys together, Ratchet.hs:439): `linkRatchetKey` (k1) is generated once at address creation, the prekey (k2) and KEM per rotation. To advertise it derives the public keys with `mkRcvE2ERatchetParams` (Ratchet.hs:412), whose argument is `(PrivateKey a, PrivateKey a, Maybe RcvPrivRKEMParams)`, from the stored link ratchet private key, the current prekey private key, and the current KEM keypair; the published `e2eVRange` is stored alongside, so the version in `mkRcvE2ERatchetParams` is only a carrier for the public keys. `ratchetKeys` and `linkRatchetKey` are set by the agent when it signs link data (`Crypto.ShortLink.encodeSignFixedData`/`encodeSignUserData`), not by the application. + +### Authentication of the advertised keys + +No signature is added on the keys: the link data already signs them. `decryptLinkData` (Crypto/ShortLink.hs:106-114) verifies `sig1` over fixed data and `sig2` over the mutable `UserContactData`, both by `rootKey`, and checks `linkKey = sha3_256(fixedData)`. So `linkRatchetKey` is hash-committed and root-signed, and `ratchetKeys` (prekey, KEM, `e2eVRange`) is root-signed as part of `UserContactData`. This is the X3DH anti-substitution property: an SMP server cannot substitute the prekey or KEM without forging the root signature or breaking the link hash. The signer is the root Ed25519 key, not `linkRatchetKey` (X448, which cannot sign) - the DH identity and the signing identity are separate keys, both anchored to the link hash. A malicious server can still serve an older but validly-signed mutable data (rollback to a retired prekey); this is bounded by the prekey retention window and by the ratchet advancing after the first message, and a signature does not prevent it. Inline ratchet params in a `CRInvitationUri` contact request are not in signed link data and remain unsigned - a separate change, out of scope here. + +### Request envelope + +`AgentConfirmation` (Protocol.hs:830-834) gains an optional `ratchetKeyId` - the `ratchetKeyId` of the `RatchetKeys` bundle the requester used, so the owner selects the matching private keys: + +```haskell +AgentConfirmation + { agentVersion :: VersionSMPA, + e2eEncryption_ :: Maybe (SndE2ERatchetParams 'C.X448), + ratchetKeyId :: Maybe ByteString, + encConnInfo :: ByteString + } +``` + +Encoding (extends Protocol.hs:853-866): from `addressDRVersion`, `smpEncode (agentVersion, 'C', e2eEncryption_, ratchetKeyId, Tail encConnInfo)`, where `ratchetKeyId` is `Just` for an address-DR confirmation and `Nothing` for the current joiner-to-initiator and initiator-to-joiner confirmations; earlier versions omit the field entirely and use `smpEncode (agentVersion, 'C', e2eEncryption_, Tail encConnInfo)`. Parsing gates the field on `agentVersion`. `CRInvitationUri` is unchanged - a connection request URI holds Rcv parameters and must not hold Snd parameters. + +## Part 4 - key rotation (separate concern) + +Rotation is required but independent of the handshake above. The link ratchet key is stable; the prekey and KEM pair rotate on a schedule. + +### Schema + +```sql +-- private stable link ratchet key on the contact address receive queue +ALTER TABLE rcv_queues ADD COLUMN link_ratchet_priv_key BLOB; -- X448 + +-- one row per ratchet-keys generation for an address; current plus retired-within-window +CREATE TABLE address_ratchet_keys( + address_ratchet_key_id INTEGER PRIMARY KEY AUTOINCREMENT, + conn_id BLOB NOT NULL REFERENCES connections ON DELETE CASCADE, + ratchet_key_id BLOB NOT NULL, -- the published id echoed by requests + prekey_priv_key BLOB NOT NULL, -- X448 private key (public derivable with publicKey) + kem_keypair BLOB, -- sntrup761 keypair (public + secret): the ratchet keeps the KEM keypair, and the retired public is not otherwise retained after LSET; NULL when PQ is off for this address + created_at TEXT NOT NULL, + retired_at TEXT -- set on rotation +); +CREATE UNIQUE INDEX idx_address_ratchet_keys ON address_ratchet_keys(conn_id, ratchet_key_id); +``` + +PostgreSQL mirrors this. Migration `M20260712_address_dr`. + +### Rotation logic + +`rotateRatchetKeys` (new), scheduled while the address is subscribed: + +1. Generate an X448 prekey pair, and an sntrup761 pair only if PQ is on for this address, with a fresh `ratchetKeyId`. +2. Recompute mutable link data with the new `RatchetKeys`, re-sign with the root key (`encodeSignUserData`), and `LSET` it to the address queue (`addSMPQueueLink`/`setConnShortLink` path). +3. Insert the new `address_ratchet_keys` row; set `retired_at` on the previous row. + +Retention window equals queue message retention (`storedMsgDataTTL`, Env/SQLite.hs:237), covering a request that used a just-retired prekey and is still in the address queue. Cadence is configuration; it bounds how long a recorded first message stays decryptable after a compromise of the current prekey - the link ratchet key alone decrypts nothing. + +### Cleanup + +`cleanupManager` (Agent.hs:2994) gains two steps: delete `address_ratchet_keys` rows with `retired_at` older than the window; and delete pending DR request connections (below) never accepted or rejected, past a request TTL. Both are batched like `deleteRcvMsgHashesExpired` (Agent.hs:3001). + +## Part 5 - backward compatibility + +- A requester older than `addressDRVersion`, or an address without `ratchetKeys`, uses R2/R3 (`AgentInvitation`); the owner uses O1-O8. Unchanged. +- The owner dispatches on the envelope: `AgentInvitation` -> `smpInvitation` (current); `AgentConfirmation` with `ratchetKeyId` on a `ContactConnection` -> `smpAddressConfirmation` (new). Both coexist. +- `AgentConfirmation` without `ratchetKeyId` remains the current confirmation on established connections. + +## Part 6 - tests + +- Encoding roundtrips: `FixedLinkData` with and without `linkRatchetKey`; `UserContactData` with and without `ratchetKeys`, and with `kemKey` present and absent; `AgentConfirmation` with and without `ratchetKeyId`, across versions. +- Address creation advertises `linkRatchetKey`, `prekey`, and optionally `kemKey`; `decryptLinkData` (Crypto/ShortLink.hs:100) verifies hash and signatures and reconstructs the advertised `RcvE2ERatchetParamsUri` (then negotiates to concrete) with and without the KEM. +- Both PQ modes: an address with `kemKey` gives a hybrid ratchet (`pqEncryption` on); an address without gives an X448-only ratchet. +- End to end: a DR-advertising address; a new requester establishes the ratchet, sends its profile under it, owner emits `REQ`, accepts, both reach `CON`; assert the profile never travels under per-queue-only encryption; assert `pqEncryption` on. +- Rotation: request against the current prekey; request against a just-retired prekey within the window still decrypts; request against a prekey past the window is discarded and the requester times out. +- Backward compatibility: old requester against a DR address connects via `AgentInvitation`; new requester against a non-DR address falls back to `AgentInvitation`. + +## Part 7 - phases + +1. Link data: `linkRatchetKey`, `RatchetKeys` (with optional `kemKey`), encodings, reconstruction, `encodeSignFixedData`/`encodeSignUserData`; address creation generating and storing the link ratchet private key and the first prekey row. +2. Handshake: requester R2'/R3', `AgentConfirmation.ratchetKeyId`, owner O1'/O2'/O3', `smpConfirmation` `RcvConnection … Nothing` branch and `AgentConnInfoReply` acceptance; end-to-end connection with the profile under the ratchet. +3. Rotation: schema migration, `rotateRatchetKeys`, retention window, cleanup step, rotation tests. diff --git a/rfcs/2026-07-11-service-rpc.md b/rfcs/2026-07-11-service-rpc.md index 074658e4f..2fc18520c 100644 --- a/rfcs/2026-07-11-service-rpc.md +++ b/rfcs/2026-07-11-service-rpc.md @@ -1,6 +1,12 @@ +--- +Proposed: 2026-07-11 +Protocol: agent-protocol (new version) +Depends on: 2026-07-12-address-pqdr-keys +--- + # One-off requests to service addresses -Implementation plan: [../plans/2026-07-11-service-rpc-implementation.md](../plans/2026-07-11-service-rpc-implementation.md) +Implementation plan: to follow, after this and the address-DR RFC are reviewed. ## Problem @@ -12,163 +18,93 @@ This is the wrong primitive for most service interactions: 2. Privacy. A connection is a stable pairwise pseudonym. If a service were to use a duplex connection, it could link all requests made over it into a profile: search history in the directory, blockchain operations linked even when different on-chain keys are used, telemetry that becomes longitudinal tracking. The client also accumulates history that can be recovered from the device. Where continuity is needed, it can be provided in the application protocol (e.g., a token included in requests), without a transport-level identity. -3. Encryption. Messages sent to contact addresses outside an established connection (invitations) have a single layer of X25519 encryption. This was acceptable while such messages contained only a connection link and profile (though it is planned to be improved); it is probably less acceptable for service requests. +3. Encryption. Messages sent to contact addresses outside an established connection have a single layer of X25519 encryption, with no post-quantum protection and no forward secrecy. This is not acceptable for service requests. -In-app service addresses should be stored as names resolving to links via the existing addressing layer (server host in the link authority, current link data retrieved with LGET), so that service links can be changed without redeploying the apps. Name resolution is already supported, and out of scope. +In-app service addresses should be stored as names resolving to links via the existing addressing layer (server host in the link authority, current link data retrieved with `LGET`), so that service links can be changed without redeploying the apps. Name resolution is already supported, and out of scope. ## Security objectives 1. Requests from the same client must not be linkable to each other by the service or by servers, and no long term state is created on either side in the transport layer. -2. Post-quantum resistant e2e encryption of requests and replies; decryptability of recorded requests must be bounded by key rotation, without changing the address. +2. Post-quantum resistant end-to-end encryption of requests and replies. 3. Reply authenticity must be verifiable against the link; substitution, replay, dropping or reordering of replies by servers must be detectable. 4. A repeated request for the same operation must not be executed twice. ## Solution -A service address is a contact address subtype. Both the client and the service are chat bots using the chat library over the agent. The chat library serializes a request into opaque bytes and calls the agent; the agent transports it and returns replies; the chat library deserializes them. The correlation of a reply with its request is the reply queue: each request has its own reply queue, and every message in that queue is a reply to that request. +A service address is an ordinary short-link contact address. The client sends one request to the address queue and receives replies in a reply queue it creates for the request. The double ratchet is established from the address's published keys (see the address-DR RFC): the request is the first ratchet message, and replies are subsequent ratchet messages. So a request-response exchange is a short-lived one-directional double ratchet connection, established from the first message and removed after the last. The exchange: -1. Resolve the service name to a short link and retrieve link data (`LGET`, via proxy when IP protection is needed). Link data holds the root key for verifying replies and the service key bundle for encrypting the request. -2. Create a reply queue (`NEW`, messaging mode, subscribed at creation). The reply queue is a receive queue with an added KEM key; the service encrypts replies to the reply queue keys. -3. Send the request to the service address queue once (unauthenticated `SEND`, via proxy when IP protection is needed). The request includes the reply queue address and its KEM key, and the application payload. It is encrypted to the service key bundle with hybrid X25519 + KEM encryption at the queue layer. There is no transport retry; a reply is the success signal, and a hard error fails the call. -4. The service decrypts the request, delivers the payload to the service application, secures the reply queue, and sends replies. Each reply message includes the request hash, the previous message hash, a flag whether more messages follow, and a non-empty list of application responses; it is signed with the root key. -5. The client verifies each reply message and delivers its responses to the application. The first reply message returns from the call; later reply messages are delivered through a callback the application registered. The client deletes the reply queue on the final message, on the request deadline, or when the application cancels the request. +1. Retrieve the address link data (`LGET`, via proxy when IP protection is needed): the root key, the identity key, the prekey with its id, and the KEM key. +2. Create a reply queue (`NEW`, subscribed). +3. Establish the sending ratchet from the published keys (`pqX3dhSnd`, `initSndRatchet`). Build the request: the reply queue, the requester's X3DH parameters, and the payload encrypted under the ratchet. Encrypt the whole request to the address queue and send it once (unauthenticated `SEND`, via proxy when IP protection is needed). There is no transport retry; a reply is the success signal, and a hard error fails the request. +4. The service establishes the receiving ratchet from its private keys and the request's X3DH parameters (`pqX3dhRcv`, `initRcvRatchet`), decrypts the payload, and delivers it to the service application. To reply it creates a send connection with the ratchet and sends reply messages to the reply queue, each encrypted under the ratchet. +5. The client decrypts and delivers each reply message to the application. The first reply message returns from the request; later reply messages are delivered through a callback the application registered. The exchange ends on a reply marked final. The client deletes the reply queue and the ratchet on the final message, the deadline, or when the application cancels. How this meets the objectives: -1. Unlinkability: fresh keys and a fresh reply queue per request; the sender's IP address and session are protected by existing private routing; the reply queue is deleted after the exchange; nothing is shared between two requests. -2. Encryption: hybrid X25519 + sntrup761 secret at the queue layer for the request and all replies. Request keys are published in mutable link data and rotated with `LSET`; the double ratchet is not used, because it would create per-request state before the service decides to reply and its properties serve long-lived sessions. -3. Authenticity: every reply message is signed with the root key committed in the immutable link data, over the request hash and the message; the previous-message hash in each message detects dropped and reordered messages. -4. Single execution: the service identifies a request by the hash of its payload and, within a fixed retention period, repeats the stored replies for a repeated request without running the operation again. +1. Unlinkability: fresh X3DH keys and a fresh reply queue per request; the sender's IP address and session are protected by existing private routing; the reply queue and both ratchets are removed after the exchange; nothing is shared between two requests. +2. Encryption: the double ratchet with its sntrup761 KEM, from the first message. +3. Authenticity: the ratchet is established against the identity key committed by the link hash, so a decryptable reply proves it came from the address owner; the ratchet message numbering detects dropped and reordered replies. No separate signature is needed. +4. Single execution: the service identifies a request by the hash of its decrypted payload and, within a fixed retention period, re-sends the stored replies for a repeated request without running the operation again. ## Design -Syntax below uses [ABNF][1] with [case-sensitive strings extension][2]. `shortString`, `largeString` and `x509encoded` are defined in the [SMP protocol](../protocol/simplex-messaging.md); `smpServer` and `agentVersion` - in the [agent protocol](../protocol/agent-protocol.md). All hashes below are SHA3-256. - -### Service address - -A new contact address type - char `s` in short links, next to existing `a`/`c`/`g`/`r`: +Syntax uses [ABNF][1] with [case-sensitive strings extension][2]. `SMPQueueInfo`, `agentVersion`, and the X3DH and ratchet types are as in the [agent protocol](../protocol/agent-protocol.md) and `Crypto.Ratchet`. -```abnf -contactType =/ %s"s" ; service address -``` +### Correlation and chat -Link data has the same structure as contact addresses: immutable fixed data (root key, connection request data with the queue address) committed by the link hash, and mutable user data signed by the root key. Fixed data and contact user data encodings ignore trailing bytes, so the field below is added without a new link format. +A reply is connected to its request by the reply queue: each request has its own reply queue, and every message in that queue is a reply to that request. The request hash (of the decrypted payload) is used only for idempotency. The application sets an id inside the payload to make two requests the same operation or different ones; the agent does not read it. -Agents dispatch on the address type and the envelope type: a normal address rejects requests, a service address rejects invitations and confirmations. - -### Service key bundle - -Mutable user data of a service address includes a key bundle, appended to the existing contact user data encoding: - -```abnf -userContactData = direct ownersList relaysList userLinkData [serviceKeys] -serviceKeys = %s"0" / (%s"1" serviceKeyBundle) ; absent in data from earlier versions -serviceKeyBundle = keyId reqDhKey reqKemKey -keyId = shortString ; identifies the bundle, included in requests -reqDhKey = length x509encoded ; X25519, used instead of the fixed-data queue key -reqKemKey = largeString ; sntrup761 encapsulation key, 1158 bytes -``` - -The service rotates the bundle with `LSET` and keeps the previous private keys long enough to decrypt requests already in the address queue - the queue message retention. The client retrieves link data for every request, so the published keys are current when the request is sent; a request becomes undecryptable only if it stays in the address queue longer than its key is kept, and such a request is not answered and fails at the client's deadline. Deleting old keys bounds the time recorded requests can be decrypted, so the retention is short and fixed. Nothing about retries or retention is published in link data. - -Link data is encrypted with a symmetric key derived from the link, which is not weakened by quantum computers, so publishing the bundle in link data does not weaken the scheme. - -Sizes: the KEM public key is 1158 bytes and user data is padded to 13784 bytes, so the bundle fits together with application data. - -### Queue-layer PQ encryption - -The single-shot queue encryption (today: crypto_box with a secret from the queue DH key and a sender ephemeral key, the ephemeral key in the message public header) is extended to a hybrid scheme in a new SMP client version. The public header gains a third variant: - -```abnf -smpPubHeaderHybrid = smpClientVersion %s"2" optKeyId senderPublicDhKey kemCiphertext -optKeyId = %s"0" / (%s"1" keyId) ; present in requests to select the bundle, absent in replies -senderPublicDhKey = length x509encoded ; sender ephemeral X25519 key -kemCiphertext = largeString ; sntrup761 ciphertext, 1039 bytes -``` - -The secret combines both shared secrets, and the body is encrypted with NaCl secret_box, padded as today: - -``` -secret = HKDF(dh(recipient key, sender ephemeral key) || KEM shared secret) -``` - -The header is readable by the destination server, as invitation headers are today; with proxied sending it is not readable by the proxy. - -For a request, the recipient keys are the service bundle keys selected by `keyId`. For a reply, the recipient keys are the reply queue keys included in the request. The first reply message includes the hybrid header; the client computes the secret once and stores it in the reply queue record, and later reply messages use it with the plain header. This is the one-secret-per-queue model of existing queues, and the secret is stored the same way the per-queue DH secret is stored on receiving the first message. Invitations and confirmations can adopt the same scheme independently of this design. +Both ends are chat bots on the chat library over the agent. The chat library serializes a service command into the request payload and deserializes the responses; the agent transports them, establishes and removes the ratchet, and correlates by reply queue. ### Request -A new agent envelope (alongside `agentConfirmation`, `agentMsgEnvelope`, `agentInvitation`, `agentRatchetKey`): +A new agent envelope, shaped like `AgentConfirmation` (X3DH parameters to establish the ratchet, plus a body encrypted under it): ```abnf -agentRequest = agentVersion %s"Q" replyQueue requestPayload -replyQueue = smpClientVersion smpServer senderId dhPublicKey kemPublicKey -smpClientVersion = 2*2 OCTET -senderId = shortString ; sender ID of the reply queue -dhPublicKey = length x509encoded ; reply queue X25519 key -kemPublicKey = largeString ; reply queue sntrup761 encapsulation key -requestPayload = *OCTET ; remaining bytes, opaque application payload +agentRequest = agentVersion %s"Q" replyQueues prekeyId sndE2EParams encRequest +replyQueues = length 1*SMPQueueInfo ; the first is used; more are for redundancy +prekeyId = shortString ; the published prekey used, from link data +sndE2EParams = +encRequest = ``` -The envelope type indicates to the receiving agent that no connection is created and replies are sent to the reply queue. The request does not include a key to secure the reply queue: the service generates its own sender key and secures the queue with it, as a sender does in the fast handshake. The reply queue address and keys are not part of the hashed payload. - -The request hash is the SHA3-256 of `requestPayload` - the same bytes the client sends and the service reads after the server encryption layer is removed. The application sets a request ID inside the payload; two requests are the same operation when their payloads, and therefore their hashes, are equal. A request must fit in one message; a larger payload is sent as an XFTP file description in the payload. +The whole `agentRequest` is encrypted to the address queue with the per-queue layer and sent with `SEND`, as an invitation is today. The reply queue keys and the X3DH parameters are not visible to servers. A request must fit one message; a larger payload is an XFTP file description in the payload. -Request delivery follows existing SEND semantics: a hard error (AUTH, QUOTA) fails the call, and the application decides whether to send a new request. +The request hash is the SHA3-256 of the decrypted payload - the same bytes on both sides. There is no transport retry; a hard error (`AUTH`, `QUOTA`) fails the request, and the application decides whether to send a new one. ### Replies -The service secures the reply queue and sends the first reply message with one combined SMP command (see below). A reply message uses a new agent envelope, following the signature-first structure of link data: +The service creates a send connection with the ratchet and sends reply messages to the reply queue, each encrypted under the ratchet. A reply message uses a new agent envelope: ```abnf -agentResponse = agentVersion %s"P" signature signedResponse -signature = length 64*64 OCTET ; root key signature of signedResponse -signedResponse = requestHash prevMsgHash more responses -requestHash = shortString ; hash of the request payload -prevMsgHash = shortString ; hash of the previous agentResponse message, empty in the first -more = %s"T" / %s"F" ; T - more reply messages follow +agentResponse = agentVersion %s"P" final responses +final = %s"T" / %s"F" ; T - no more reply messages follow responses = length 1*responseItem ; non-empty list of application responses responseItem = largeString ; opaque application response ``` -A reply message includes a list of responses, so responses known together are sent in one message rather than several; responses that become known over time are sent in separate messages. The signature covers the whole message. - -The client verifies each reply message: the signature against the root key from link data, the request hash against the sent request, the previous-message hash against the previous message. A message that fails verification is acknowledged and discarded. Forging a reply message requires the root key even if the reply queue keys are known; a signed message cannot be reused for a different request; the previous-message hash detects messages dropped or reordered by the reply queue server. - -The first reply message returns from the request call. Later reply messages are delivered through the callback the application registered with the request. The exchange ends on a message with `more` set to false. The client deletes the reply queue on that message, on the request deadline, or when the application cancels the request. Deleting the queue stops further replies: later SEND commands from the service fail with AUTH. Server queue and message expiry limit the lifetime of a reply queue left after a client restart. - -### Combined SKEY+SEND command - -A new SMP command combining `SKEY` and `SEND` in one transmission: - -```abnf -secureSend = %s"SSND " senderAuthPublicKey SP msgFlags SP smpEncMessage -senderAuthPublicKey = length x509encoded -``` - -The command is authorized with the key it sets, as `SKEY`, and only accepted on messaging-mode queues. The server responds `OK` or `ERR`. It is idempotent in both parts: +Each message includes a list of responses, so responses known together are sent in one message and responses that become known over time are sent in separate messages. The message is encrypted and numbered by the ratchet, which authenticates it against the committed identity key and detects dropped or reordered messages; no separate signature is used. -- key part: as `SKEY` today - a repeated command with the same key succeeds, a different key fails with AUTH. -- send part: the server keeps the hash of the message until it is acknowledged, and a repeated command with the same message within that period is reported as delivered without delivering it again. +The first reply message returns from the request. Later reply messages are delivered through the callback the application registered with the request, while the process runs. The exchange ends on a message with `final = T`. The client deletes the reply queue and the ratchet on that message, on the deadline, or when the application cancels. Deleting the reply queue stops further replies. The transport keeps no exchange across a client restart; the application keeps its own state and sends a new request when it needs to. -A repeat arriving after the message was acknowledged is delivered as a duplicate and discarded by the receiving agent by message hash, so the server-side hash covers the common case and the agent covers the rest. +### Rejection -The service uses this command for the first reply message. The joining party uses it in the fast connection handshake, where it replaces `SKEY` followed by the `SEND` confirmation, removing one command and one round trip. +The service refuses a request with the `AgentRejection` envelope from the [communicating rejection RFC](../../simplex-chat/docs/rfcs/2024-03-22-communicating-reject.md), sent to the reply queue under the ratchet, with an opaque application reason. The same envelope communicates refusal of a connection request, where today it is dropped silently. A rejection ends the exchange like a final reply. ### Idempotency -The service agent identifies a request by its hash and keeps, for a fixed retention period it chooses (in the 1 to 24 hour range, in service configuration, not in link data), a record of the ordered reply messages it has sent and the reply queues subscribed under that hash. A repeat request with the same hash does not reach the service application: +The service keeps, for a fixed retention period it chooses (1 to 24 hours, in service configuration, not in link data), the request hash, the ordered response messages it produced, and the reply queues and ratchets subscribed under that hash. A repeat request with the same hash does not reach the service application: -- while the first request is being answered, the repeat is added to the record and receives the reply messages already sent, and each later message is sent to every reply queue in the record. -- after the operation completed, the repeat receives the whole stored sequence of reply messages, in order. +- while the first request is being answered, the repeat establishes its own ratchet and reply queue, is added to the record, receives the responses already produced, and receives each later response too. +- after the operation completed, the repeat receives the whole stored sequence of responses, re-encrypted under its own ratchet. -This gives single execution over at-least-once delivery. The retention period bounds it: after the record is deleted, a request with the same hash is a new operation and runs again. The application does not rely on the transport for recovery across a restart; it keeps its own state and, after a restart, sends whatever request fits what it knows, which is often a different request (for example, "is this payment still pending" rather than a repeat of "start this payment"). +The stored responses are the application response bytes, not the ratchet ciphertext, because a repeat establishes a new ratchet and the responses are re-encrypted for it. This gives single execution over at-least-once delivery. After the retention period a request with the same hash is a new operation and runs again. ### Out of scope -- Recovery across restart: the transport keeps no exchange across a client restart. The application persists its own state and sends a new request when it needs to. +- Recovery across restart: the transport keeps no exchange across a client restart; the application persists its own state and sends a new request when it needs to. - Service-initiated messages: there is no standing channel; use a connection where the service must reach the client without a request. - Abuse protection beyond existing queue quotas: services can require application-level credentials (e.g., a badge) in the request payload; rate limiting is a separate discussion. - Scaling request reception: a single address queue bounds service throughput; distributing reception across multiple queues or relays (the existing `relays` field in contact link data) is a separate question, but it would fit well with name resolving to multiple addresses, both for redundancy, reliability and higher throughput. diff --git a/rfcs/2026-07-12-address-pqdr-keys.md b/rfcs/2026-07-12-address-pqdr-keys.md new file mode 100644 index 000000000..51e080f52 --- /dev/null +++ b/rfcs/2026-07-12-address-pqdr-keys.md @@ -0,0 +1,95 @@ +--- +Proposed: 2026-07-12 +Protocol: agent-protocol (new version) +--- + +# Establishing the double ratchet from address data + +## Problem + +A contact address receives a connection request as an `AgentInvitation` message, encrypted only with the per-queue X25519 layer. The double ratchet is established later: the address owner joins the requester's connection request, generates its X3DH keys, and sends them in the confirmation. So the first message to an address - the invitation and the profile in it - is not under the double ratchet, and has no post-quantum protection. + +The cause is that the address owner's X3DH contribution is generated per request and sent in the confirmation, so it cannot exist before the requester's first message. + +## Solution + +Publish the address owner's X3DH contribution in the address link data, so a requester can establish the double ratchet in its first message. The requester runs the existing `pqX3dhSnd` against the published keys, initializes a sending ratchet, and encrypts its first message under it. The owner runs the existing `pqX3dhRcv` against its stored private keys and the requester's X3DH keys from the message, initializes a receiving ratchet, and decrypts it. + +The published keys follow the X3DH structure: + +- The identity key goes in the immutable fixed link data, next to the root key, committed by the link hash. It is stable. +- The prekey and the KEM encapsulation key go in the mutable user data, signed by the root key, rotated with `LSET`. + +This is backward compatible. A requester that does not use the published keys sends a current `AgentInvitation` with its own X3DH keys, and the owner does what it does today: generates fresh X3DH keys and sends them in the confirmation. The owner branches on whether the incoming message uses the published keys. + +Two properties follow. The first message, including the profile, is under the double ratchet, which closes the profile gap and gives it post-quantum protection through the ratchet's sntrup761 KEM. And a decryptable message proves the sender established X3DH against the identity key committed by the link hash, so it authenticates the address owner without a separate signature. + +## Design + +Syntax uses [ABNF][1] with [case-sensitive strings extension][2]. Key and ratchet types are as in `Crypto.Ratchet` and the [agent protocol](../protocol/agent-protocol.md); all DH keys are X448, the KEM is sntrup761. + +### Published keys in link data + +The identity key is appended to fixed link data: + +```abnf +fixedData =/ addressIdentityKey ; appended, ignored by earlier versions +addressIdentityKey = %s"0" / (%s"1" length x509encoded) ; X448 +``` + +The prekey and KEM key are appended to contact user data: + +```abnf +userContactData =/ addressRatchetKeys ; appended, ignored by earlier versions +addressRatchetKeys = %s"0" / (%s"1" prekeyId prekey kemKey) +prekeyId = shortString ; identifies the prekey, echoed in the request +prekey = length x509encoded ; X448 +kemKey = largeString ; sntrup761 encapsulation key +``` + +Together with the version range in fixed data these reconstruct an `RcvE2ERatchetParams` (the owner's X3DH contribution): identity key as the first key, prekey as the second, KEM key as the KEM parameter. + +The owner keeps the private prekey and KEM key indexed by `prekeyId`. On rotation with `LSET` it publishes a new pair with a new id and keeps the previous private pair for a window covering queue message retention, so a request that used a just-rotated prekey still decrypts. The identity key is not rotated. + +### Existential ratchet in the request + +`CRInvitationUri` fixes the ratchet parameters to `RcvE2ERatchetParamsUri 'C.X448` today. It becomes existential over the establishment form, so a message can include either the current inline parameters or a reference to the published address keys: + +```haskell +data AddressE2EParams + = InlineE2EParams (RcvE2ERatchetParamsUri 'C.X448) -- current: requester's own keys + | PublishedE2EParams ByteString (SndE2ERatchetParams 'C.X448) -- prekeyId, requester's Snd params +``` + +`InlineE2EParams` is the current message. `PublishedE2EParams` names the prekey the requester used (so the owner selects the matching private keys) and includes the requester's Snd X3DH parameters (so the owner runs `pqX3dhRcv`). The owner branches on the constructor: `PublishedE2EParams` takes the published-key path with `pqX3dhRcv` against the stored private keys; `InlineE2EParams` takes the current path, generating fresh Snd params and sending them in the confirmation. + +### Establishing the ratchet + +Requester: + +1. Retrieve link data (`LGET`), read the identity key, the prekey with its id, and the KEM key, and reconstruct the owner's `RcvE2ERatchetParams`. +2. `generateSndE2EParams` for its own X3DH contribution. +3. `pqX3dhSnd` against the owner's parameters, then `initSndRatchet` - the sending ratchet. +4. Encrypt the first message under the ratchet, and include `PublishedE2EParams prekeyId sndParams`. + +Owner: + +1. On a message with `PublishedE2EParams prekeyId sndParams`, select the private prekey and KEM key by `prekeyId` (current or a retained previous pair). +2. `pqX3dhRcv` against `sndParams` with the identity, prekey, and KEM private keys, then `initRcvRatchet` - the receiving ratchet. +3. Decrypt, and reply under the ratchet. + +A request whose `prekeyId` is no longer retained cannot be decrypted; the owner does not learn the requester or its reply address, and the requester's attempt fails at its own timeout. + +### Authentication + +The ratchet is established against the identity key committed by the link hash, so a decryptable message proves the sender holds that identity key. Where a message today relies on a separate signature over its content for authenticity, this establishment provides it, and the signature is not needed. The identity key is a DH key and X3DH is over crypto_box, so deniability is preserved, and reuse of the identity key across requesters is consistent with the address already being a shared identifier. + +## Uses + +- Invitations to an address, and the profile in them, are under the double ratchet from the first message. +- The service RPC (see the RPC RFC) establishes the ratchet this way to send the request as the first ratchet message. + +This RFC depends on nothing else here. It replaces the need for the PQ-queue RFC in the address case, because the ratchet provides post-quantum protection for the first message; the PQ-queue RFC remains for first messages that do not establish a ratchet. + +[1]: https://tools.ietf.org/html/rfc5234 +[2]: https://tools.ietf.org/html/rfc7405 diff --git a/rfcs/2026-07-12-queue-pq-encryption.md b/rfcs/2026-07-12-queue-pq-encryption.md new file mode 100644 index 000000000..f82092323 --- /dev/null +++ b/rfcs/2026-07-12-queue-pq-encryption.md @@ -0,0 +1,42 @@ +--- +Proposed: 2026-07-12 +Protocol: smp-client (new version) +--- + +# Post-quantum encryption of the SMP queue layer + +## Problem + +A message sent to a queue outside an established double ratchet has one layer of end-to-end encryption: NaCl crypto_box over an X25519 DH secret. The sender generates an ephemeral X25519 key, computes the secret with the recipient's per-queue DH key, and puts its ephemeral key in the message public header (`agentCbEncryptOnce`). This layer protects invitations, confirmations, and the profile sent with them. + +It is not post-quantum. An adversary that records this traffic and later has a quantum computer can recover the X25519 secret and decrypt it. The double ratchet adds a post-quantum KEM once it is established, but the first message to a queue, before the ratchet, has only X25519. + +This RFC adds post-quantum protection to the single-shot queue encryption itself, for cases that do not establish a double ratchet from the first message. Where a double ratchet is established from the first message (see the address-DR RFC), the ratchet provides post-quantum protection and this layer is not needed. + +## Solution + +Extend the single-shot queue encryption to a hybrid X25519 + sntrup761 scheme, in a new SMP client version. The recipient publishes a KEM encapsulation key alongside its per-queue DH key. The sender encapsulates to it, combines the KEM shared secret with the X25519 DH secret, and encrypts the body with the combined secret. The KEM ciphertext travels in the message public header next to the ephemeral X25519 key. + +Recording the traffic and breaking X25519 later is not sufficient: without breaking sntrup761 as well, the combined secret is not recoverable. + +## Design + +The message public header (`PubHeader`) gains a hybrid variant, selected by a version and a tag, so older senders and the empty-header case are unchanged: + +```abnf +smpPubHeaderHybrid = smpClientVersion %s"2" senderPublicDhKey kemCiphertext +senderPublicDhKey = length x509encoded ; sender ephemeral X25519 key +kemCiphertext = largeString ; sntrup761 ciphertext, 1039 bytes +``` + +The secret combines both shared secrets, and the body is encrypted with NaCl secret_box (the DH-only path uses crypto_box today; the combined secret is no longer a plain DH result, so it is used as a secret_box key), padded to the same lengths: + +``` +secret = HKDF(dh(recipient key, sender ephemeral key) || KEM shared secret) +``` + +The recipient's KEM encapsulation key is distributed the same way its per-queue DH key is today - in the queue address for a connection request, and in link data for a short link. The KEM ciphertext is stored the same way the ephemeral DH key is: in the public header, readable by the destination server (and not by the proxy with proxied sending), which cannot derive the secret without the recipient's KEM private key. + +The recipient stores the computed secret the way the per-queue DH secret is stored on receiving the first message (`setRcvQueueConfirmedE2E`), and reuses it for later messages on the queue. + +Sizes: the KEM ciphertext is 1039 bytes and the encapsulation key 1158 bytes. Link data user data is padded to 13784 bytes, so the key fits with application data. This RFC is independent of the RPC, SSND, and address-DR RFCs. diff --git a/rfcs/2026-07-12-smp-secure-send.md b/rfcs/2026-07-12-smp-secure-send.md new file mode 100644 index 000000000..6bb444b09 --- /dev/null +++ b/rfcs/2026-07-12-smp-secure-send.md @@ -0,0 +1,55 @@ +--- +Proposed: 2026-07-12 +Protocol: smp (new version) +--- + +# SSND: combined secure-and-send command + +## Problem + +Two SMP flows secure a messaging queue and then immediately send the first message to it, as two commands and two round trips: + +- The fast connection handshake: the joining party secures the queue with `SKEY`, then sends the confirmation with `SEND`. +- Any first send to a sender-securable queue where the sender both secures it and delivers the first message. + +The two commands express one intent - "this is my key, and here is my first message" - so they can be one command and one round trip. The combination must be idempotent, because the first send is retried on network failure and a queue is often secured before the response is known. `SKEY` is already idempotent (a repeat with the same key succeeds). `SEND` is not, so a naive combination would deliver a duplicate message on retry. + +## Solution + +A new command `SSND` combines `SKEY` and `SEND` in one transmission, idempotent in both parts: + +- Key part: as `SKEY` - a repeat with the same key succeeds, a different key fails with `AUTH`. +- Send part: the server keeps the hash of the message until it is acknowledged, and reports a repeat of the same message as delivered without delivering it again. + +The server-side hash covers the common case, when the retry arrives before the message is acknowledged. A retry that arrives after the acknowledgement is delivered as a duplicate and discarded by the receiving agent by message hash, as duplicate messages are discarded today. + +## Design + +Syntax uses [ABNF][1] with [case-sensitive strings extension][2]. `senderAuthPublicKey`, `msgFlags` and `smpEncMessage` are as in the [SMP protocol](../protocol/simplex-messaging.md). + +```abnf +secureSend = %s"SSND " senderAuthPublicKey SP msgFlags SP smpEncMessage +senderAuthPublicKey = length x509encoded +``` + +`SSND` is a sender command, authorized with the key it sets, and accepted only on messaging-mode queues (`QMMessaging`), where the sender can secure the queue. The server responds `OK` or `ERR`. + +Server processing: + +1. Secure the queue with the key, as `SKEY`. A repeat with the same key succeeds; a different key returns `AUTH`. +2. If the queue holds an unacknowledged message whose stored hash equals the hash of this message, respond `OK` without storing it again. +3. Otherwise store the message, keep its hash with the queue until the message is acknowledged, and deliver it. + +The stored hash is one value per not-yet-acknowledged queue message. It is removed when the message is acknowledged. All queue store backends (in-memory, journal, PostgreSQL) keep it. + +`SSND` composes with the proxy protocol without change: `proxySMPCommand` forwards any sender command through `PFWD`/`RFWD`, so `SSND` is proxied as `SEND` and `SKEY` are today. + +## Uses + +- The fast connection handshake replaces `SKEY` then the `SEND` confirmation with one `SSND`. +- The service RPC response (see the RPC RFC) secures the reply queue and sends the first reply with one `SSND`. + +This RFC is independent of the RPC, PQ-queue, and address-DR RFCs and can be implemented on its own. + +[1]: https://tools.ietf.org/html/rfc5234 +[2]: https://tools.ietf.org/html/rfc7405 From 32351995fd798b9c4747de9b16c74e8b26d6566e Mon Sep 17 00:00:00 2001 From: "Evgeny @ SimpleX Chat" <259188159+evgeny-simplex@users.noreply.github.com> Date: Mon, 13 Jul 2026 11:21:31 +0000 Subject: [PATCH 5/6] update rfc --- rfcs/2026-07-12-address-pqdr-keys.md | 53 ++++++++++++++-------------- 1 file changed, 26 insertions(+), 27 deletions(-) diff --git a/rfcs/2026-07-12-address-pqdr-keys.md b/rfcs/2026-07-12-address-pqdr-keys.md index 51e080f52..d4846c7c6 100644 --- a/rfcs/2026-07-12-address-pqdr-keys.md +++ b/rfcs/2026-07-12-address-pqdr-keys.md @@ -17,12 +17,12 @@ Publish the address owner's X3DH contribution in the address link data, so a req The published keys follow the X3DH structure: -- The identity key goes in the immutable fixed link data, next to the root key, committed by the link hash. It is stable. -- The prekey and the KEM encapsulation key go in the mutable user data, signed by the root key, rotated with `LSET`. +- The link ratchet key - the stable first X3DH key - goes in the immutable fixed link data, next to the root key, committed by the link hash. +- The prekey, the KEM encapsulation key, and the e2e version range go in the mutable user data as a ratchet-keys bundle with an id, signed by the root key, rotated with `LSET`. This is backward compatible. A requester that does not use the published keys sends a current `AgentInvitation` with its own X3DH keys, and the owner does what it does today: generates fresh X3DH keys and sends them in the confirmation. The owner branches on whether the incoming message uses the published keys. -Two properties follow. The first message, including the profile, is under the double ratchet, which closes the profile gap and gives it post-quantum protection through the ratchet's sntrup761 KEM. And a decryptable message proves the sender established X3DH against the identity key committed by the link hash, so it authenticates the address owner without a separate signature. +Two properties follow. The first message, including the profile, is under the double ratchet, which closes the profile gap and gives it post-quantum protection through the ratchet's sntrup761 KEM. And a decryptable message proves the sender established X3DH against the link ratchet key committed by the link hash, so it authenticates the address owner without a separate signature. ## Design @@ -30,59 +30,58 @@ Syntax uses [ABNF][1] with [case-sensitive strings extension][2]. Key and ratche ### Published keys in link data -The identity key is appended to fixed link data: +The link ratchet key is appended to fixed link data: ```abnf -fixedData =/ addressIdentityKey ; appended, ignored by earlier versions -addressIdentityKey = %s"0" / (%s"1" length x509encoded) ; X448 +fixedData =/ linkRatchetKey ; appended, ignored by earlier versions +linkRatchetKey = %s"0" / (%s"1" length x509encoded) ; X448, stable, the first X3DH key ``` -The prekey and KEM key are appended to contact user data: +The ratchet-keys bundle is appended to contact user data: ```abnf -userContactData =/ addressRatchetKeys ; appended, ignored by earlier versions -addressRatchetKeys = %s"0" / (%s"1" prekeyId prekey kemKey) -prekeyId = shortString ; identifies the prekey, echoed in the request +userContactData =/ ratchetKeys ; appended, ignored by earlier versions +ratchetKeys = %s"0" / (%s"1" ratchetKeyId e2eVRange prekey kemKey) +ratchetKeyId = shortString ; identifies this bundle, changes on rotation, echoed in the request +e2eVRange = prekey = length x509encoded ; X448 -kemKey = largeString ; sntrup761 encapsulation key +kemKey = %s"0" / (%s"1" largeString) ; sntrup761 encapsulation key, optional (PQ is opt-in) ``` -Together with the version range in fixed data these reconstruct an `RcvE2ERatchetParams` (the owner's X3DH contribution): identity key as the first key, prekey as the second, KEM key as the KEM parameter. +Together these reconstruct the owner's X3DH contribution as an `RcvE2ERatchetParamsUri` - a version-range form: `linkRatchetKey` (from fixed data) as the first key, `prekey` as the second, `kemKey` as the optional KEM parameter, `e2eVRange` as the version range. The requester negotiates the concrete version against its own e2e range, as it does for an invitation, then runs `pqX3dhSnd` against the result. -The owner keeps the private prekey and KEM key indexed by `prekeyId`. On rotation with `LSET` it publishes a new pair with a new id and keeps the previous private pair for a window covering queue message retention, so a request that used a just-rotated prekey still decrypts. The identity key is not rotated. +The owner keeps the private prekey and, when PQ is on, the KEM keypair, indexed by `ratchetKeyId`. On rotation with `LSET` it publishes a new bundle with a new `ratchetKeyId` and keeps the previous private keys for a window covering queue message retention, so a request that used a just-rotated prekey still decrypts. The link ratchet key is stable and not rotated. -### Existential ratchet in the request +### Request confirmation -`CRInvitationUri` fixes the ratchet parameters to `RcvE2ERatchetParamsUri 'C.X448` today. It becomes existential over the establishment form, so a message can include either the current inline parameters or a reference to the published address keys: +A requester that uses the published keys establishes the sending ratchet before its first message, so it sends that message as a confirmation, not an invitation - the same envelope a joining party sends in a connection. The confirmation gains an optional `ratchetKeyId` naming the bundle the requester used, so the owner selects the matching private keys: -```haskell -data AddressE2EParams - = InlineE2EParams (RcvE2ERatchetParamsUri 'C.X448) -- current: requester's own keys - | PublishedE2EParams ByteString (SndE2ERatchetParams 'C.X448) -- prekeyId, requester's Snd params +```abnf +agentConfirmation =/ ratchetKeyId ; the bundle the requester used, echoed; absent on other confirmations ``` -`InlineE2EParams` is the current message. `PublishedE2EParams` names the prekey the requester used (so the owner selects the matching private keys) and includes the requester's Snd X3DH parameters (so the owner runs `pqX3dhRcv`). The owner branches on the constructor: `PublishedE2EParams` takes the published-key path with `pqX3dhRcv` against the stored private keys; `InlineE2EParams` takes the current path, generating fresh Snd params and sending them in the confirmation. +The confirmation holds the requester's Snd X3DH parameters (so the owner runs `pqX3dhRcv`) and, encrypted under the ratchet, the first message. A confirmation with `ratchetKeyId` on a contact address takes the published-key path; an `AgentInvitation`, as today, takes the current path where the owner generates fresh X3DH keys and returns them in its own confirmation. A connection-request URI is unchanged: it advertises the requester's Rcv parameters and must not include Snd parameters. ### Establishing the ratchet Requester: -1. Retrieve link data (`LGET`), read the identity key, the prekey with its id, and the KEM key, and reconstruct the owner's `RcvE2ERatchetParams`. -2. `generateSndE2EParams` for its own X3DH contribution. +1. Retrieve link data (`LGET`), read the link ratchet key, the prekey with its `ratchetKeyId`, the `e2eVRange`, and the optional KEM key, reconstruct the owner's `RcvE2ERatchetParamsUri`, and negotiate the concrete version against its own e2e range. +2. `generateSndE2EParams` for its own X3DH contribution (with the KEM only when the address advertises one). 3. `pqX3dhSnd` against the owner's parameters, then `initSndRatchet` - the sending ratchet. -4. Encrypt the first message under the ratchet, and include `PublishedE2EParams prekeyId sndParams`. +4. Encrypt the first message under the ratchet, and send a confirmation with its Snd parameters and `ratchetKeyId`. Owner: -1. On a message with `PublishedE2EParams prekeyId sndParams`, select the private prekey and KEM key by `prekeyId` (current or a retained previous pair). -2. `pqX3dhRcv` against `sndParams` with the identity, prekey, and KEM private keys, then `initRcvRatchet` - the receiving ratchet. +1. On a confirmation with `ratchetKeyId` on a contact address, select the private prekey and, if any, KEM keypair by `ratchetKeyId` (current or a retained previous generation). +2. `pqX3dhRcv` against the requester's Snd parameters with the link ratchet, prekey, and optional KEM private keys, then `initRcvRatchet` - the receiving ratchet. Decrypting the first message advances the ratchet and gives it a send side, so the owner can reply. 3. Decrypt, and reply under the ratchet. -A request whose `prekeyId` is no longer retained cannot be decrypted; the owner does not learn the requester or its reply address, and the requester's attempt fails at its own timeout. +A request whose `ratchetKeyId` is no longer retained cannot be decrypted; the owner does not learn the requester or its reply address, and the requester's attempt fails at its own timeout. ### Authentication -The ratchet is established against the identity key committed by the link hash, so a decryptable message proves the sender holds that identity key. Where a message today relies on a separate signature over its content for authenticity, this establishment provides it, and the signature is not needed. The identity key is a DH key and X3DH is over crypto_box, so deniability is preserved, and reuse of the identity key across requesters is consistent with the address already being a shared identifier. +The ratchet is established against the link ratchet key committed by the link hash and the prekey signed by the root key, so a decryptable message proves the sender established X3DH against the owner's published, link-committed keys. Where a message today relies on a separate signature over its content for authenticity, this establishment provides it, and the signature is not needed. The link ratchet key is a DH key (X448) and X3DH is over crypto_box, so deniability is preserved; the signing identity (the root key) and the DH key are separate keys, both anchored to the link hash. Reusing the link ratchet key across requesters is consistent with the address already being a shared identifier. ## Uses From 88ab41266e1a73528bc3258e59d0d831ead75d2a Mon Sep 17 00:00:00 2001 From: "Evgeny @ SimpleX Chat" <259188159+evgeny-simplex@users.noreply.github.com> Date: Mon, 13 Jul 2026 18:58:34 +0000 Subject: [PATCH 6/6] update rfc and plan --- plans/2026-07-12-address-dr-implementation.md | 161 +++++++++++------- rfcs/2026-07-12-address-pqdr-keys.md | 41 ++--- 2 files changed, 119 insertions(+), 83 deletions(-) diff --git a/plans/2026-07-12-address-dr-implementation.md b/plans/2026-07-12-address-dr-implementation.md index dc10412dd..86d72beb4 100644 --- a/plans/2026-07-12-address-dr-implementation.md +++ b/plans/2026-07-12-address-dr-implementation.md @@ -6,6 +6,8 @@ All references are to the current tree. Names of new constructors, fields, table Goal: a contact address advertises the owner's X3DH parameters in link data; a requester establishes the double ratchet in its first message, so that message and the profile in it are under the ratchet with post-quantum protection. The change reuses the invitation/confirmation machinery, with the requester in the joiner role and the owner in the initiator role - opposite to today's contact flow, but every message and code path below is reused. +Version: `addressDRVersion = VersionSMPA 8`, a plain agent-layer bump; `currentSMPAgentVersion` goes 7 → 8 (Agent/Protocol.hs:317-324). It gates the `AgentConfirmation.ratchetKeyId` field and the DR-from-address behavior. The receive-at-address path relies on ratchet-on-confirmation, already present since `ratchetOnConfSMPAgentVersion = 7` (Agent/Protocol.hs:317), so there is no cross-layer version dependency; the SMP and e2e-encryption versions are unchanged. + ## Part 1 - the current contact-address handshake, step by step Requester Alice connects to owner Bob's contact address. Q_A is Alice's receive queue (Bob to Alice), Q_B is Bob's receive queue (Alice to Bob). @@ -37,7 +39,7 @@ Owner side, receiving confirmation #2 on Q_B: - O7. dispatch (Agent.hs:3182): `AgentConfirmation` -> `smpConfirmation`. - O8. `smpConfirmation`, accepting-party branch `DuplexConnection … Nothing` (Agent.hs:3447-3462): `agentRatchetDecrypt` with the established ratchet; `AgentConnInfo` -> `INFO` (Agent.hs:3452); `ICDuplexSecure` or `CON`. -`HELLO`/`CON` completion follows via `helloMsg` (Agent.hs:3466). +Completion is direct `CON` on `senderCanSecure` (SKEY) messaging-mode queues (the sender on `AgentConnInfo`, Agent.hs:2252; the receiver with no `senderKey`, Agent.hs:3459-3461); the separate `HELLO` via `helloMsg` (Agent.hs:3466) is the older non-`senderCanSecure` (duplexHandshake v2, in-band-securing) path. ## Part 2 - the DR-from-address handshake, mapped to Part 1 @@ -45,21 +47,21 @@ The address advertises Bob's Rcv X3DH parameters in link data (Part 3). Alice, w Requester side - a new branch in `joinConnSrv … CRContactUri`, chosen when the fetched link data has `ratchetKeys`: -- R2'. Replaces R2/R3. Read the advertised `linkRatchetKey`, `ratchetKeyId`, `e2eVRange`, `prekey`, and optional `kemKey` from link data; reconstruct Bob's `RcvE2ERatchetParamsUri 'C.X448` and negotiate the concrete version with `compatibleVersion` against the client e2e range, as `compatibleInvitationUri` does (Agent.hs:1362-1368). Create the receive queue Q_A subscribed (`newRcvQueue` with `subMode`), messaging mode so Bob can secure it. Run `generateSndE2EParams` (with `replyKEM_` to accept the KEM only when advertised), `pqX3dhSnd` against the negotiated parameters, `initSndRatchet`, `createSndRatchet` - the body of `createRatchet_` (Agent.hs:1343-1350), with parameters from link data rather than a received invitation. -- R3'. Build `AgentConfirmation {e2eEncryption_ = Just aliceSndParams, ratchetKeyId = Just ratchetKeyId, encConnInfo = ratchetEncrypt(AgentConnInfoReply (Q_A :| []) aliceProfile)}` - the `mkAgentConfirmation`/`mkConfirmation` bodies (Agent.hs:3780-3765) with the reply queue being Alice's own Q_A. Send it to the address queue unauthenticated with `agentCbEncryptOnce`, as `sendInvitation` sends (Agent/Client.hs:1929-1934). **Alice's profile is now inside `encConnInfo`, under the ratchet.** Alice's connection is `RcvConnection` (Q_A) with a send ratchet, until she receives Q_B. +- R2'. Replaces R2/R3. Read the advertised bundle from link data - `ratchetKeyId` and `e2eParams :: RcvE2ERatchetParamsUri 'C.X448` - and negotiate the concrete version with `compatibleVersion` against the client e2e range, as `compatibleInvitationUri` does (Agent.hs:1362-1368). Create the receive queue Q_A subscribed (`newRcvQueue` with `subMode`), messaging mode so Bob can secure it. Choose the requester's KEM with `replyKEM_ v ownerKem_ pqSup` (Ratchet.hs:839): if the bundle advertises a KEM (owner `IKUsePQ`) the requester encapsulates to it (`AcceptKEM`, PQ from message 1); if not and the requester wants PQ, it proposes its own (`ProposeKEM`, PQ from message 2 if the owner supports it). Run `generateSndE2EParams g v (replyKEM_ …)`, `pqX3dhSnd` against the negotiated parameters, `initSndRatchet`, `createSndRatchet` - the body of `createRatchet_` (Agent.hs:1343-1350), with parameters from link data rather than a received invitation. +- R3'. Build `AgentConfirmation {e2eEncryption_ = Just aliceSndParams, ratchetKeyId = Just ratchetKeyId, encConnInfo = ratchetEncrypt(AgentConnInfoReply (Q_A :| []) aliceProfile)}` - the `mkAgentConfirmation`/`mkConfirmation` bodies (Agent.hs:3780-3765) with the reply queue being Alice's own Q_A. Send it to the address queue unauthenticated with `agentCbEncryptOnce`, as `sendInvitation` sends (Agent/Client.hs:1929-1934). Retry is chat re-invoking the join, reusing the prepared connection's keys via the existing `startJoinInvitation`-style reuse (Agent.hs:1314) - exactly as a classic contact request; a resend is not deduplicated (it may produce another `REQ` on the owner, as a resent classic invitation does). The connect UX (premature failure, lost-reply recovery) is a separate chat-layer concern, out of scope here. **Alice's profile is now inside `encConnInfo`, under the ratchet.** Alice's connection is `RcvConnection` (Q_A) with a send ratchet, until she receives Q_B. Owner side - a new dispatch branch and a new receive handler: - O1'. In `processClientMsg` (Agent.hs:3176-3187), add a branch in state `(Nothing, Just e2ePubKey)`: an `AgentConfirmation` with `ratchetKeyId = Just _` on a `ContactConnection` -> `smpAddressConfirmation` (new). It must be placed **before** the existing `(PHEmpty, AgentConfirmation) | senderCanSecure queueMode` case (Agent.hs:3182-3184), because a contact-address queue is `QMContact` (not `senderCanSecure`) and would otherwise fall into `prohibited "handshake: missing sender key"` (Agent.hs:3184). The address queue's `e2eDhSecret` stays `Nothing` (it is never set for a contact address - `smpInvitation` does not set it, Agent.hs:3609-3622), so every request is decrypted with its own ephemeral key via this `(Nothing, Just e2ePubKey)` path. -- O2'. `smpAddressConfirmation` (new, modeled on `smpConfirmation` initiating branch, Agent.hs:3405-3444): select the private prekey and, if any, KEM keypair by `ratchetKeyId` from `address_ratchet_keys`; `pqX3dhRcv` with the `linkRatchetKey` private key, the selected prekey private key, and the optional KEM keypair (as `PrivateRKParamsProposed`) against `aliceSndParams`; `initRcvRatchet` (its `PQSupport` follows from whether the address advertises a KEM and version compatibility, as `smpConfirmation` derives `pqSupport'`, Agent.hs:3410); `rcDecrypt` of `encConnInfo` performs the first ratchet step, giving the connection its send side too (as it does for the initiator today), so the owner can later reply; `createRatchet` stores the post-decrypt ratchet under a **new connection created now** for this request. Parse `AgentConnInfoReply (Q_A :| []) aliceProfile`, store a pending-request record referencing the new connection, its ratchet, and Q_A (not a `connReq`), and emit `REQ`. An unknown or expired `ratchetKeyId`, or a decryption failure: discard and acknowledge, as an undecryptable message is dropped today. This establishes ratchet state on unauthenticated input before the user accepts - see "Receive-time establishment, state, and abuse". -- O3'. `acceptContact'` for a DR-established request - a new branch that continues the ratchet instead of `joinConn`: reuse `mkAgentConfirmation` (Agent.hs:3780-3785) to create Bob's receive queue Q_B and return `AgentConnInfoReply (Q_B :| []) bobInfo`; create Bob's send queue to Q_A (`newSndQueue`, generating Bob's own sender key) and secure Q_A with `SKEY` using that key (`agentSecureSndQueue`, valid because Q_A is messaging mode) - the securing key is Bob's own, not taken from Alice's message; send the response to Q_A as `AgentConfirmation {e2eEncryption_ = Nothing, ratchetKeyId = Nothing, encConnInfo = ratchetEncrypt(AgentConnInfoReply (Q_B :| []) bobInfo)}`, per-queue encrypted with `agentCbEncryptOnce` (first message to Q_A). The reply content is `AgentConnInfoReply`, not `AgentConnInfo`: it takes the `mkAgentConfirmation` path with `e2eEncryption_ = Nothing`, not the `enqueueConfirmation` path (which produces `AgentConnInfo`, Agent.hs:3789). +- O2'. `smpAddressConfirmation` (new, modeled on `smpConfirmation` initiating branch, Agent.hs:3405-3444): select the private triple `(pk1, pk2, pKem)` by `ratchetKeyId` from `address_ratchet_keys`; `pqX3dhRcv pk1 pk2 pKem aliceSndParams`; `initRcvRatchet` with the address connection's stored `PQSupport` (`connPQEncryption` of the address `InitialKeys` - `On` for `IKUsePQ` and `IKPQOn`, `Off` for `IKPQOff`; this is what lets `IKPQOn` accept the requester's proposed KEM), combined with version compatibility as `smpConfirmation` derives `pqSupport'` (Agent.hs:3410); `rcDecrypt` of `encConnInfo` performs the first ratchet step, giving the ratchet its send side too (as it does for the initiator today), so the owner can later reply. Parse `AgentConnInfoReply (Q_A :| []) aliceProfile`. Store the request with `createInvitation` on the address connection (`contact_conn_id`), exactly as a classic invitation - except the request value is the `CRConfirmation` variant (Part 3) carrying the post-decrypt ratchet state and Q_A, and `recipient_conn_info` is `aliceProfile` - so **no connection or `ratchets` row is created at receive**, as with a classic invitation. Emit `REQ` with the `invitation_id`. A resend is not deduplicated: like a resent classic invitation it produces another `REQ` (the connect-UX fix for that is separate chat work). An unknown or expired `ratchetKeyId`, or a decryption failure: discard and acknowledge, as an undecryptable message is dropped today. This establishes ratchet state on unauthenticated input before the user accepts - see "Receive-time establishment, state, and abuse". +- O3'. `acceptContact'` for a DR request - a new branch that continues the ratchet instead of `joinConn`. `getInvitation` returns the request; its `CRConfirmation` variant gives the stored ratchet state and Q_A. Create the connection now (as `joinConn` does for a classic invitation) and `createRatchet` (AgentStore.hs:1419) from the stored ratchet state. Reuse `mkAgentConfirmation` (Agent.hs:3780-3785) to create Bob's receive queue Q_B and return `AgentConnInfoReply (Q_B :| []) bobInfo`; create Bob's send queue to Q_A (`newSndQueue`, generating Bob's own sender key) and secure Q_A with `SKEY` using that key (`agentSecureSndQueue`, valid because Q_A is messaging mode) - the securing key is Bob's own, not taken from Alice's message; send the response to Q_A as `AgentConfirmation {e2eEncryption_ = Nothing, ratchetKeyId = Nothing, encConnInfo = ratchetEncrypt(AgentConnInfoReply (Q_B :| []) bobInfo)}` via `sendConfirmation` (`agentCbEncrypt` over Bob's send queue to Q_A, `PHEmpty` because Q_A is `senderCanSecure`) - exactly the current contact msg 2 path (Client.hs:1916), not `agentCbEncryptOnce`. The reply content is `AgentConnInfoReply`, not `AgentConnInfo`: it takes the `mkAgentConfirmation` path with `e2eEncryption_ = Nothing`, not the `enqueueConfirmation` path (which produces `AgentConnInfo`, Agent.hs:3789). `rejectContact'` deletes the `conn_invitations` row (the current behaviour), discarding the inline ratchet; no connection was created, so there is nothing else to clean up. Requester side, receiving the response on Q_A: -- R5'. `smpConfirmation` needs a new branch `RcvConnection … Nothing` (today only `RcvConnection … Just` and `DuplexConnection … Nothing` exist, Agent.hs:3403-3447): Alice already holds the send ratchet, so `agentRatchetDecrypt` advances it and creates the receive side; `setRcvQueueConfirmedE2E` on Q_A so later messages use the stored per-queue secret (as the current branches do, Agent.hs:3440,3455); parse `AgentConnInfoReply (Q_B :| []) bobInfo`; connect Q_B (create Alice's send queue to Q_B, `agentSecureSndQueue` it, upgrade `RcvConnection` to `DuplexConnection`); emit `INFO`. The accepting-party branch (Agent.hs:3447) must also accept `AgentConnInfoReply`, not only `AgentConnInfo` (Agent.hs:3451, 3462), for the reply-queue case. -- R6'/completion. The response went to Q_A, so Bob has no signal that Alice secured Q_B and would never reach `CON` on its own. The essential extra message is Alice -> Q_B: after R5' Alice sends `HELLO` on Q_B, and `helloMsg` (Agent.hs:3466) sets Q_B active on Bob's side and emits Bob's `CON`. Alice reaches `CON` from the response (Q_A active, Q_B secured). Whether Bob also needs to send `HELLO` on Q_A for Alice's `CON`, or the response suffices, follows the existing `helloMsg`/fast-path completion and is confirmed at implementation - the fixed point is that Alice must send on Q_B. +- R5'. `smpConfirmation` needs a new branch `RcvConnection … Nothing` (today only `RcvConnection … Just` and `DuplexConnection … Nothing` exist, Agent.hs:3403-3447): Alice already holds the send ratchet, so `agentRatchetDecrypt` advances it and creates the receive side; `setRcvQueueConfirmedE2E` on Q_A so later messages use the stored per-queue secret (as the current branches do, Agent.hs:3440,3455); parse `AgentConnInfoReply (Q_B :| []) bobInfo`; emit `INFO` (Bob's profile); then `connectReplyQueues` (Agent.hs:3724) - create Alice's send queue to Q_B, secure Q_B with `SKEY` (`agentSecureSndQueue`, Q_B is messaging mode), upgrade to `DuplexConnection`, and `enqueueConfirmation … Nothing` to send the third message, `AgentConnInfo` to Q_B (`encConnInfo = ratchetEncrypt(AgentConnInfo aliceInfo)`). Because Q_B is sender-securable, sending `AgentConnInfo` completes Alice with `CON` (Agent.hs:2252) - no `HELLO`. +- R6'/completion. Unchanged from the current contact handshake, and modern (no `HELLO`). The exchange is three agent↔agent wire messages - Alice → address queue (msg 1), Bob → Q_A (msg 2, an `AgentConfirmation` carrying `AgentConnInfoReply` with Q_B), Alice → Q_B (msg 3, an `AgentConfirmation` carrying `AgentConnInfo`) - the same shape as the current contact flow, where msg 1 was `AgentInvitation`; here it is the ratchet-establishing `AgentConfirmation`. (`CON` is not a wire message - it is the agent→app event; `HELLO` and `AgentConnInfo` are the wire messages.) `HELLO` belongs to the older non-`senderCanSecure` path (duplexHandshake v2, before SKEY): there the confirmation secures the queue in-band (`PHConfirmation` carries the sender key, Client.hs:1918) and the receiver replies with `HELLO` (`ICDuplexSecure` → `enqueueDuplexHello`, Agent.hs:3457-3458). Both Q_A and Q_B here are messaging-mode - Q_A by R2', Q_B via `createReplyQueue` → `SCMInvitation` → `QMMessaging` (Agent.hs:1233,1458,3783) - so the sender secures with SKEY and sends `PHEmpty` (Client.hs:1918), the dispatch takes the `senderCanSecure` branch (Agent.hs:3182-3184), and each agent raises the `CON` app event locally off msg 3 - Bob on receiving it (`senderKey = Nothing`, Agent.hs:3459-3461), Alice on sending it (Agent.hs:2252) - with no separate `HELLO` wire message. (msg 2's `AgentConnInfoReply` only sets Q_A `Confirmed`, Agent.hs:2254.) Invitations are two messages because the initiator's queue is already in the link; a contact address needs three because Bob's receive queue Q_B is only delivered in msg 2. The third message no longer has a ratchet role: Bob's X3DH params are pre-published, so the agreement is complete once Bob receives msg 1 (in the current flow Bob's Snd params instead arrive in msg 2). msg 2 and msg 3 are queue setup - msg 2 delivers Q_B, msg 3 secures Q_B so Alice can send to Bob and signals Bob's `CON`; neither negotiates the ratchet. A one-directional exchange (the RPC) needs no Q_B and is two messages. -Net code touch points: `joinConnSrv` (new requester branch), `processClientMsg` (new owner dispatch), `smpConfirmation` (new `RcvConnection … Nothing` branch and `AgentConnInfoReply` acceptance), `acceptContact'` (new continue-ratchet branch), a new `smpAddressConfirmation`, and the link data and storage of Part 3-4. +Net code touch points: `joinConnSrv` (new requester branch), `processClientMsg` (new owner dispatch), `smpConfirmation` (new `RcvConnection … Nothing` branch and `AgentConnInfoReply` acceptance), `acceptContact'` (new continue-ratchet branch), a new `smpAddressConfirmation` reusing `createInvitation`/`getInvitation` with the sum request value, and the link data and storage of Part 3-4. `rejectContact'` is unchanged (it deletes the `conn_invitations` row either way). ### Receive-time establishment, state, and abuse @@ -69,59 +71,82 @@ Design decision (Q1): decrypt at receive. Both use cases need the request conten Consequences: -- A connection is created at receive to hold the ratchet (ratchets are keyed by `conn_id`). O2' creates a `NewConnection`, stores the post-decrypt ratchet under it, and a pending-request record with Q_A's address; O3' (accept) adds Bob's send queue to Q_A and receive queue Q_B, upgrading to `DuplexConnection`; reject deletes the connection, its ratchet, and the record. -- Per incoming `AgentConfirmation` the owner does one `pqX3dhRcv` (three DH plus, with PQ, one `sntrup761` decapsulation) and one `rcDecrypt`, on unauthenticated input, and creates a connection row and a ratchet row - more CPU and state than the current `NewInvitation`. +- No connection is created at receive, exactly as for a classic invitation. O2' stores the request with `createInvitation` on the address connection; the post-decrypt ratchet state and Q_A live inline in the `CRConfirmation` request value (`cr_invitation`). O3' (accept) creates the connection, `createRatchet` from the stored state, and adds Bob's queues, becoming a `DuplexConnection`; `rejectContact'` deletes the `conn_invitations` row. +- Per incoming `AgentConfirmation` the owner does one `pqX3dhRcv` (three DH plus, with PQ, one `sntrup761` decapsulation) and one `rcDecrypt`, on unauthenticated input, and writes one `conn_invitations` row - more CPU than the current `NewInvitation`, the same order of state (no connection, no `ratchets` row until accept). -Abuse (Q2): a contact address already accepts and processes unauthenticated invitations today, so this is a degree-worse version of an existing surface, not a new class. It is bounded by the address queue quota (an attacker fills it, the owner drains and acknowledges) and, optionally, by basic auth on the address (already supported for contact addresses, `optBasicAuth`). The pending-request state is bounded by a request TTL: `cleanupManager` deletes pending DR connections never accepted or rejected past that TTL (Part 4 cleanup). Proof-of-work or a stricter gate can be added later; it is out of scope here and noted as a follow-up. +Abuse (Q2): a contact address already accepts and processes unauthenticated invitations today, so this is a degree-worse version of an existing surface, not a new class. It is bounded by the address queue quota (an attacker fills it, the owner drains and acknowledges) and, optionally, by basic auth on the address (already supported for contact addresses, `optBasicAuth`). The per-request state is a single `conn_invitations` row - the same class as a classic contact request - so it is subject to the same limits and lifecycle, with no DR-specific dedup or TTL. Proof-of-work or a stricter gate can be added later; it is out of scope here and noted as a follow-up. -The `acceptContact'` dispatch branches on the pending record: a classic invitation (`connReq` present) takes the current `joinConn` path (O3-O6); a DR request (pending connection and ratchet present, no `connReq`) takes the continue-ratchet path (O3'). +`acceptContact'`/`rejectContact'` keep taking the `invitation_id` from `REQ` unchanged; the only difference is that `getInvitation` returns a request that is either a `CRInvitation` URI (current `joinConn` path, O3-O6) or a `CRConfirmation` (continue-ratchet path, O3'). Nothing in the `REQ`/accept/reject flow or the chat client changes - the change is contained in the agent. -## Part 3 - types and link data +### The four communication layers, per message (verified against code) -### Fixed data - link ratchet key +Layers, outermost (server-visible) first: -Appended to `FixedLinkData` (Protocol.hs:1824); the encoding already stops at a trailing tail (Protocol.hs:1928), so earlier versions ignore it: +- **L1 `ClientMsgEnvelope`** (Protocol.hs:1089), `PubHeader {phVersion, phE2ePubDhKey :: Maybe PublicKeyX25519}` (1096) - **this is where per-queue encryption is agreed** (not L2). `phE2ePubDhKey` is the sender's e2e DH public key; the recipient combines it with the queue's e2e private key: `(e2eDhSecret, e2ePubKey_) -> (Nothing, Just e2ePubKey) -> e2eDh = dh' e2ePubKey e2ePrivKey` (Agent.hs:3172-3178). `agentCbEncryptOnce` (Client.hs:2214) puts a **fresh ephemeral** pubkey (generated 2217, set 2223) - used when the sender has no send queue (the address queue), whose `e2eDhSecret` stays `Nothing`, so it decrypts every message with the per-message ephemeral. `agentCbEncrypt` (Client.hs:2203) puts the **send queue's persistent** e2e pubkey (`Just` on a confirmation, 2210); the recipient stores the secret via `setRcvQueueConfirmedE2E`, and *later* messages send `phE2ePubDhKey = Nothing` (`sendAgentMessage`, 2080). +- **L2 `ClientMessage PrivHeader`** (Protocol.hs:1113), `PrivHeader = PHConfirmation APublicAuthKey | PHEmpty` (1115) - **queue securing / authorization, not encryption**. `PHConfirmation` carries the sender's AUTH key for in-band securing (v2, non-`senderCanSecure`); `PHEmpty` when the sender secured the queue with SKEY out-of-band. `PHEmpty` on every message here is about securing, and says nothing about encryption (that is L1). Set in `sendConfirmation` (Client.hs:1918), `sendInvitation` (1934), `sendAgentMessage` (2079). +- **L3 `AgentMsgEnvelope`** (Agent/Protocol.hs:829, encoding 851) - outside the ratchet. `AgentConfirmation` ('C') carries `e2eEncryption_` (Snd X3DH params, agrees DR) + `encConnInfo`; `AgentInvitation` ('I') carries `connReq` (Rcv X3DH params) + plaintext `connInfo` (no DR); `AgentMsgEnvelope` ('M') carries `encAgentMessage`. +- **L4 `AgentMessage`** (Agent/Protocol.hs:883, encoding 893) - inside the ratchet. `AgentConnInfo` ('I'), `AgentConnInfoReply` ('D', reply queues + info), `AgentMessage APrivHeader AMessage` ('M'; `AMessage` includes `HELLO`, Agent/Protocol.hs:1018-1020). **Absent when L3 is `AgentInvitation`** (that profile is per-queue-only - the gap this plan closes). -```haskell -data FixedLinkData c = FixedLinkData - { agentVRange :: VersionRangeSMPA, - rootKey :: C.PublicKeyEd25519, - linkConnReq :: ConnectionRequestUri c, - linkEntityId :: Maybe ByteString, - linkRatchetKey :: Maybe C.PublicKeyX448 -- stable X448 key, the first X3DH key - } -``` +Send routing: msg 1 (to address) → `sendInvitation` today / a new `agentCbEncryptOnce` confirmation send for DR; msg 2 → `secureConfirmQueue` → `sendConfirmation` (Agent.hs:3747); msg 3 → `connectReplyQueues` → `enqueueConfirmation` → delivery worker `AM_CONN_INFO` → `sendConfirmation` (Agent.hs:3733,3789,2183). `AM_CONN_INFO`/`AM_CONN_INFO_REPLY` both go through `sendConfirmation` (2183-2184); other `AMessage`s go through `sendAgentMessage` wrapping `AgentMsgEnvelope` 'M' (2192-2193). + +Current contact handshake (address does **not** advertise DR): + +| msg | L1 `PubHeader.phE2ePubDhKey` (per-queue enc) | L2 `PrivHeader` (securing) | L3 `AgentMsgEnvelope` | L4 `AgentMessage` | +|---|---|---|---|---| +| 1 Alice→addr | `Just` fresh ephemeral, `agentCbEncryptOnce` (Client.hs:1933,2223) | `PHEmpty` (1934) | `AgentInvitation` {connReq = Alice Rcv params, connInfo = profile} (Client.hs:1932) | — none (profile per-queue only) | +| 2 Bob→Q_A | `Just` Bob's send-queue e2e pubkey, `agentCbEncrypt` (1920,2210) | `PHEmpty` [`senderCanSecure`] (1918) | `AgentConfirmation` {e2eEncryption_ = **Just Bob Snd params**, encConnInfo} (Agent.hs:3765) | `AgentConnInfoReply` (Q_B) bobInfo, DR-enc (Agent.hs:3785) | +| 3 Alice→Q_B | `Just` Alice's send-queue e2e pubkey, `agentCbEncrypt` (1920,2210) | `PHEmpty` [`senderCanSecure`] (1918) | `AgentConfirmation` {e2eEncryption_ = **Nothing**, encConnInfo} (Agent.hs:3802) | `AgentConnInfo` aliceInfo, DR-enc (Agent.hs:3789) | + +New DR handshake (address advertises DR): + +| msg | L1 `PubHeader.phE2ePubDhKey` (per-queue enc) | L2 `PrivHeader` (securing) | L3 `AgentMsgEnvelope` | L4 `AgentMessage` | +|---|---|---|---|---| +| 1 Alice→addr | `Just` fresh ephemeral, `agentCbEncryptOnce` [same] | `PHEmpty` [same] | **`AgentConfirmation`** {e2eEncryption_ = **Just Alice Snd params**, **ratchetKeyId = Just**, encConnInfo} [was `AgentInvitation`] | **`AgentConnInfoReply`** (Q_A) aliceProfile, **DR-enc** [was plaintext connInfo] | +| 2 Bob→Q_A | `Just` Bob's send-queue e2e pubkey, `agentCbEncrypt` [same] | `PHEmpty` [same] | `AgentConfirmation` {e2eEncryption_ = **Nothing**, ratchetKeyId = Nothing, encConnInfo} [was Just Bob Snd params] | `AgentConnInfoReply` (Q_B) bobInfo, DR-enc [same] | +| 3 Alice→Q_B | `Just` Alice's send-queue e2e pubkey, `agentCbEncrypt` [same] | `PHEmpty` [same] | `AgentConfirmation` {e2eEncryption_ = Nothing, encConnInfo} [same] | `AgentConnInfo` aliceInfo, DR-enc [same] | + +Net difference: **only msg 1 and msg 2's L3/L4 change.** msg 1's L3 becomes `AgentConfirmation` (was `AgentInvitation`) carrying Alice's Snd params + `ratchetKeyId`, and the profile moves from plaintext L3 to DR-encrypted L4 (`AgentConnInfoReply`) - the whole point of the change. msg 2 drops `e2eEncryption_` (Bob no longer sends Snd params - the ratchet is agreed from msg 1). msg 3 is unchanged. L1 (per-queue encryption - each queue agrees its own secret via the sender's e2e pubkey in the `PubHeader` on the first message to it) and L2 (securing, `PHEmpty` because SKEY is used) are unchanged throughout; the DR change is entirely at L3/L4. The only new send code is msg 1 (an `AgentConfirmation` fired to the address with `agentCbEncryptOnce`, like `sendInvitation` but with a confirmation envelope). + +## Part 3 - types and link data + +### Fixed data - unchanged + +`FixedLinkData` (Protocol.hs:1824) is not touched. The double-ratchet keys go entirely in mutable data, so an existing address advertises them without a new link (the fixed data is hash-committed and cannot change). Fixed data keeps only `agentVRange`, `rootKey`, `linkConnReq`, `linkEntityId`. ### Mutable data - ratchet keys bundle Appended to `UserContactData` (Protocol.hs:1840); the encoding stops at a trailing tail (Protocol.hs:1981), so earlier versions ignore it: ```haskell -data RatchetKeys = RatchetKeys - { ratchetKeyId :: ByteString, -- identifies this bundle; changes on rotation - e2eVRange :: VersionRangeE2E, -- advertised e2e version range, for negotiation - prekey :: C.PublicKeyX448, - kemKey :: Maybe KEMPublicKey -- opt-in, as PQ is opt-in in the ratchet (PQSupport) +data AddressRatchetKeys = AddressRatchetKeys + { ratchetKeyId :: ByteString, -- identifies this bundle; changes on rotation, echoed in the request + e2eParams :: CR.RcvE2ERatchetParamsUri 'C.X448 -- version range + both X3DH keys + optional KEM } data UserContactData = UserContactData { direct :: Bool, owners :: [OwnerAuth], relays :: [ConnShortLink 'CMContact], userData :: UserLinkData, - ratchetKeys :: Maybe RatchetKeys + ratchetKeys :: Maybe AddressRatchetKeys } ``` -Reconstruction produces `RcvE2ERatchetParamsUri 'C.X448` (a version range, `E2ERatchetParamsUri VersionRangeE2E k1 k2 (Maybe kem)`), not concrete params: `e2eVRange` as the range, `linkRatchetKey` (from fixed data) as the first key, `prekey` as the second, `kemKey` as the optional KEM parameter. The requester negotiates the concrete version with `compatibleVersion` against its own e2e range, exactly as `compatibleInvitationUri` does for an invitation (Agent.hs:1362-1368), then uses the resulting `RcvE2ERatchetParams` in `pqX3dhSnd`. The KEM is optional: with `kemKey = Nothing` the ratchet is X448-only, as when `PQSupport` is off; with `Just` it is hybrid, matching `E2ERatchetParams … (Maybe (RKEMParams s))` (Ratchet.hs:220-221) and `generateRcvE2EParams`'s `PQSupport` gate (Ratchet.hs:439-445). +`e2eParams` is the existing `RcvE2ERatchetParamsUri 'C.X448` (`E2ERatchetParamsUri VersionRangeE2E k1 k2 (Maybe (RKEMParams s))`, Ratchet.hs:282-286) - the same type a `CRInvitationUri` advertises - with `StrEncoding`/`Encoding` already defined (Ratchet.hs:302-374). There is no bespoke key type and no reconstruction: the requester negotiates the concrete version with `compatibleVersion` against its own e2e range, exactly as `compatibleInvitationUri` does for an invitation (Agent.hs:1362-1368), giving `RcvE2ERatchetParams` for `pqX3dhSnd`. The KEM is optional: `Nothing` gives an X448-only ratchet (as when `PQSupport` is off), `Just` a hybrid one, matching `generateRcvE2EParams`'s `PQSupport` gate (Ratchet.hs:439-445). + +The address-creation parameter is `InitialKeys` (Ratchet.hs:864) - the same 3-way choice as invitations, not a bare `PQSupport`. Currently `IKUsePQ` is prohibited for `SCMContact` (Agent.hs:990,1198) because a contact address carries no owner keys; this change lifts that prohibition. The bundle plays the published-contact-request role, so its KEM follows `initialPQEncryption False pqInitKeys` (Ratchet.hs:882) - exactly as the requester's contact request does today (Agent.hs:1422): -The owner does not use `generateRcvE2EParams` (which generates both keys together, Ratchet.hs:439): `linkRatchetKey` (k1) is generated once at address creation, the prekey (k2) and KEM per rotation. To advertise it derives the public keys with `mkRcvE2ERatchetParams` (Ratchet.hs:412), whose argument is `(PrivateKey a, PrivateKey a, Maybe RcvPrivRKEMParams)`, from the stored link ratchet private key, the current prekey private key, and the current KEM keypair; the published `e2eVRange` is stored alongside, so the version in `mkRcvE2ERatchetParams` is only a carrier for the public keys. `ratchetKeys` and `linkRatchetKey` are set by the agent when it signs link data (`Crypto.ShortLink.encodeSignFixedData`/`encodeSignUserData`), not by the application. +- `IKUsePQ` - the bundle advertises the KEM; the requester encapsulates to it, so PQ from message 1. +- `IKPQOn` (`IKLinkPQ PQSupportOn`) - the bundle is X448-only (no KEM advertised), but the owner's ratchet supports PQ (`connPQEncryption` = On, Ratchet.hs:888); the requester proposes its own KEM (R2'), so PQ from message 2. +- `IKPQOff` (`IKLinkPQ PQSupportOff`) - X448-only, and the owner's ratchet does not support PQ even if the requester proposes it. + +Advertising the KEM adds ~1158 B to the rotated, widely-fetched link data, which is why `IKPQOn` exists (PQ one round later, without the size cost). The owner generates the bundle with `generateRcvE2EParams g v (initialPQEncryption False pqInitKeys)` (Ratchet.hs:439), stores the private triple `(pk1, pk2, pKem)` (Part 4), and advertises `e2eParams` by wrapping the public `E2ERatchetParams` in the address's e2e version range (`toVersionRangeT`; or `mkRcvE2ERatchetParams` from the stored privates, Ratchet.hs:412) - the same private-key shape `createRatchetX3dhKeys`/`getRatchetX3dhKeys` already store (AgentStore.hs:1362-1367). `ratchetKeys` is set by the agent when it signs mutable link data (`Crypto.ShortLink.encodeSignUserData`), not by the application. ### Authentication of the advertised keys -No signature is added on the keys: the link data already signs them. `decryptLinkData` (Crypto/ShortLink.hs:106-114) verifies `sig1` over fixed data and `sig2` over the mutable `UserContactData`, both by `rootKey`, and checks `linkKey = sha3_256(fixedData)`. So `linkRatchetKey` is hash-committed and root-signed, and `ratchetKeys` (prekey, KEM, `e2eVRange`) is root-signed as part of `UserContactData`. This is the X3DH anti-substitution property: an SMP server cannot substitute the prekey or KEM without forging the root signature or breaking the link hash. The signer is the root Ed25519 key, not `linkRatchetKey` (X448, which cannot sign) - the DH identity and the signing identity are separate keys, both anchored to the link hash. A malicious server can still serve an older but validly-signed mutable data (rollback to a retired prekey); this is bounded by the prekey retention window and by the ratchet advancing after the first message, and a signature does not prevent it. Inline ratchet params in a `CRInvitationUri` contact request are not in signed link data and remain unsigned - a separate change, out of scope here. +No signature is added on the keys: the mutable link data already signs them. `decryptLinkData` (Crypto/ShortLink.hs:106-114) verifies `sig2` over the mutable `UserContactData` by `rootKey`, so `ratchetKeys` is root-signed. This is the X3DH anti-substitution property: an SMP server cannot substitute the keys without forging the root signature. The signer is the root Ed25519 key (the address's signing identity); the X3DH keys are separate DH keys (X448, which cannot sign). A single owner signs address data ("we don't use multiple owners"), so the root signature alone is sufficient - no per-key signature. A malicious server can still serve an older but validly-signed `UserContactData` (rollback to a retired bundle); this is bounded by the retention window and by the ratchet advancing after the first message, and a signature does not prevent it. Inline ratchet params in a `CRInvitationUri` contact request are not in signed link data and remain unsigned - a separate change, out of scope here. ### Request envelope -`AgentConfirmation` (Protocol.hs:830-834) gains an optional `ratchetKeyId` - the `ratchetKeyId` of the `RatchetKeys` bundle the requester used, so the owner selects the matching private keys: +`AgentConfirmation` (Protocol.hs:830-834) gains an optional `ratchetKeyId` - the `ratchetKeyId` of the `AddressRatchetKeys` bundle the requester used, so the owner selects the matching private keys: ```haskell AgentConfirmation @@ -134,62 +159,84 @@ AgentConfirmation Encoding (extends Protocol.hs:853-866): from `addressDRVersion`, `smpEncode (agentVersion, 'C', e2eEncryption_, ratchetKeyId, Tail encConnInfo)`, where `ratchetKeyId` is `Just` for an address-DR confirmation and `Nothing` for the current joiner-to-initiator and initiator-to-joiner confirmations; earlier versions omit the field entirely and use `smpEncode (agentVersion, 'C', e2eEncryption_, Tail encConnInfo)`. Parsing gates the field on `agentVersion`. `CRInvitationUri` is unchanged - a connection request URI holds Rcv parameters and must not hold Snd parameters. +### Stored request - the invitation record + +The `conn_invitations` record stays; only the type of the stored request widens. From chat's point of view a DR request is still an invitation - it "contains a confirmation" instead of an invitation URI - so `REQ`, `acceptContact'`/`rejectContact'`, and the chat side are unchanged; the change is contained in the agent. The `NewInvitation`/`Invitation` request field (`cr_invitation`, stays `NOT NULL`) becomes a sum: + +```haskell +data ContactRequest + = CRInvitation (ConnectionRequestUri 'CMInvitation) -- classic: joinConn on accept (O3-O6) + | CRConfirmation DRRequest -- DR: continue the ratchet on accept (O3') + +data DRRequest = DRRequest + { drRatchet :: RatchetX448, -- post-decrypt receiving ratchet (with send side), stored inline + drReplyQueue :: SMPQueueInfo -- Q_A, where the owner replies + } +``` + +`recipient_conn_info` holds the profile in both cases. `getInvitation`/`createInvitation` carry `ContactRequest`; `acceptContact'` branches on the constructor. There is no dedup column: a resent request produces another `REQ`, exactly as a resent classic invitation does. + ## Part 4 - key rotation (separate concern) -Rotation is required but independent of the handshake above. The link ratchet key is stable; the prekey and KEM pair rotate on a schedule. +Rotation is required but independent of the handshake above. The whole ratchet-keys bundle - both X448 keys and the KEM - rotates on a schedule, generated fresh each time. ### Schema ```sql --- private stable link ratchet key on the contact address receive queue -ALTER TABLE rcv_queues ADD COLUMN link_ratchet_priv_key BLOB; -- X448 - --- one row per ratchet-keys generation for an address; current plus retired-within-window +-- one row per ratchet-keys generation for an address; current plus retired-within-window. +-- private side of the advertised RcvE2ERatchetParamsUri - same shape as the ratchets x3dh +-- columns and createRatchetX3dhKeys (AgentStore.hs:1362-1367). CREATE TABLE address_ratchet_keys( address_ratchet_key_id INTEGER PRIMARY KEY AUTOINCREMENT, conn_id BLOB NOT NULL REFERENCES connections ON DELETE CASCADE, - ratchet_key_id BLOB NOT NULL, -- the published id echoed by requests - prekey_priv_key BLOB NOT NULL, -- X448 private key (public derivable with publicKey) - kem_keypair BLOB, -- sntrup761 keypair (public + secret): the ratchet keeps the KEM keypair, and the retired public is not otherwise retained after LSET; NULL when PQ is off for this address + ratchet_key_id BLOB NOT NULL, -- the published id echoed by requests + x3dh_priv_key_1 BLOB NOT NULL, -- X448 + x3dh_priv_key_2 BLOB NOT NULL, -- X448 + pq_priv_kem BLOB, -- RcvPrivRKEMParams (sntrup761 keypair); NULL when PQ is off for this address created_at TEXT NOT NULL, - retired_at TEXT -- set on rotation + retired_at TEXT -- set on rotation ); CREATE UNIQUE INDEX idx_address_ratchet_keys ON address_ratchet_keys(conn_id, ratchet_key_id); + +-- a DR request stays in conn_invitations with NO schema change: cr_invitation now holds a ContactRequest +-- sum (an invitation URI or a confirmation carrying the post-decrypt ratchet + reply queue), so it stays +-- NOT NULL - no nullable change, no new column on conn_invitations, no new table for the request. ``` -PostgreSQL mirrors this. Migration `M20260712_address_dr`. +`cr_invitation` stays `NOT NULL` - only its decoded value gains a variant (Part 3), so the invitations flow, `REQ`, and chat are unchanged; the only new storage is the `address_ratchet_keys` table. The link signing key is already on the address queue (`rcv_queues.link_priv_sig_key`, M20250322), so nothing is added there - rotation and retrofit re-sign mutable data with it. PostgreSQL mirrors this. Migration `M20260712_address_dr`. ### Rotation logic -`rotateRatchetKeys` (new), scheduled while the address is subscribed: +`rotateRatchetKeys` (new), run on address subscription, at most every 2 weeks (skip if the current generation is younger): -1. Generate an X448 prekey pair, and an sntrup761 pair only if PQ is on for this address, with a fresh `ratchetKeyId`. -2. Recompute mutable link data with the new `RatchetKeys`, re-sign with the root key (`encodeSignUserData`), and `LSET` it to the address queue (`addSMPQueueLink`/`setConnShortLink` path). -3. Insert the new `address_ratchet_keys` row; set `retired_at` on the previous row. +1. `generateRcvE2EParams` (Ratchet.hs:439) for a fresh generation - two X448 keys, and an sntrup761 keypair only if PQ is on for this address - with a fresh `ratchetKeyId`. +2. Recompute mutable link data with the new `AddressRatchetKeys` (the public `e2eParams`), re-sign with the root key (`encodeSignUserData`, key from `rcv_queues.link_priv_sig_key`), and `LSET` it to the address queue (`setConnShortLink` path). +3. Insert the new `address_ratchet_keys` row (`x3dh_priv_key_1`, `x3dh_priv_key_2`, `pq_priv_kem`); set `retired_at` on the previous row. -Retention window equals queue message retention (`storedMsgDataTTL`, Env/SQLite.hs:237), covering a request that used a just-retired prekey and is still in the address queue. Cadence is configuration; it bounds how long a recorded first message stays decryptable after a compromise of the current prekey - the link ratchet key alone decrypts nothing. +Retention window equals queue message retention (`storedMsgDataTTL`, Env/SQLite.hs:237), covering a request that used a just-retired bundle and is still in the address queue. The 2-week cadence bounds how long a recorded first message stays decryptable after a compromise of the current private keys - a retired generation is deleted after the window and then decrypts nothing. ### Cleanup -`cleanupManager` (Agent.hs:2994) gains two steps: delete `address_ratchet_keys` rows with `retired_at` older than the window; and delete pending DR request connections (below) never accepted or rejected, past a request TTL. Both are batched like `deleteRcvMsgHashesExpired` (Agent.hs:3001). +`cleanupManager` (Agent.hs:2994) gains one step: delete `address_ratchet_keys` rows with `retired_at` older than the window, batched like `deleteRcvMsgHashesExpired` (Agent.hs:3001). Unaccepted DR request rows in `conn_invitations` are handled exactly like unaccepted classic invitation requests - no DR-specific cleanup (a DR request is one `conn_invitations` row, the same class of state as a classic contact request). ## Part 5 - backward compatibility - A requester older than `addressDRVersion`, or an address without `ratchetKeys`, uses R2/R3 (`AgentInvitation`); the owner uses O1-O8. Unchanged. - The owner dispatches on the envelope: `AgentInvitation` -> `smpInvitation` (current); `AgentConfirmation` with `ratchetKeyId` on a `ContactConnection` -> `smpAddressConfirmation` (new). Both coexist. - `AgentConfirmation` without `ratchetKeyId` remains the current confirmation on established connections. +- An existing address gains `ratchetKeys` when the app updates the address - with the user's confirmation and the current profile - combined with the full→short address migration, not by the agent alone (which has neither the profile nor the user's intent). The update is an `LSET` of mutable data re-signed with `rcv_queues.link_priv_sig_key`; no new link is issued, because the keys are in mutable, not fixed, data. Requesters that fetch the updated data use DR; older ones still use `AgentInvitation`. ## Part 6 - tests -- Encoding roundtrips: `FixedLinkData` with and without `linkRatchetKey`; `UserContactData` with and without `ratchetKeys`, and with `kemKey` present and absent; `AgentConfirmation` with and without `ratchetKeyId`, across versions. -- Address creation advertises `linkRatchetKey`, `prekey`, and optionally `kemKey`; `decryptLinkData` (Crypto/ShortLink.hs:100) verifies hash and signatures and reconstructs the advertised `RcvE2ERatchetParamsUri` (then negotiates to concrete) with and without the KEM. -- Both PQ modes: an address with `kemKey` gives a hybrid ratchet (`pqEncryption` on); an address without gives an X448-only ratchet. +- Encoding roundtrips: `UserContactData` with and without `ratchetKeys`, and with the KEM present and absent; `AgentConfirmation` with and without `ratchetKeyId`, across versions. +- Address creation advertises `ratchetKeys` (the `RcvE2ERatchetParamsUri`); `decryptLinkData` (Crypto/ShortLink.hs:100) verifies signatures and the requester negotiates the advertised params to a concrete version, with and without the KEM. +- Both PQ modes: an address whose bundle carries a KEM gives a hybrid ratchet (`pqEncryption` on); one without gives an X448-only ratchet. - End to end: a DR-advertising address; a new requester establishes the ratchet, sends its profile under it, owner emits `REQ`, accepts, both reach `CON`; assert the profile never travels under per-queue-only encryption; assert `pqEncryption` on. -- Rotation: request against the current prekey; request against a just-retired prekey within the window still decrypts; request against a prekey past the window is discarded and the requester times out. +- Rotation and retrofit: request against the current bundle; against a just-retired bundle within the window still decrypts; against a bundle past the window is discarded and the requester times out; an address that adds `ratchetKeys` via `LSET` is then reached by DR while an old requester still uses `AgentInvitation`. - Backward compatibility: old requester against a DR address connects via `AgentInvitation`; new requester against a non-DR address falls back to `AgentInvitation`. ## Part 7 - phases -1. Link data: `linkRatchetKey`, `RatchetKeys` (with optional `kemKey`), encodings, reconstruction, `encodeSignFixedData`/`encodeSignUserData`; address creation generating and storing the link ratchet private key and the first prekey row. -2. Handshake: requester R2'/R3', `AgentConfirmation.ratchetKeyId`, owner O1'/O2'/O3', `smpConfirmation` `RcvConnection … Nothing` branch and `AgentConnInfoReply` acceptance; end-to-end connection with the profile under the ratchet. -3. Rotation: schema migration, `rotateRatchetKeys`, retention window, cleanup step, rotation tests. +1. Link data: `AddressRatchetKeys` in `UserContactData` (reusing `RcvE2ERatchetParamsUri`), encoding, `encodeSignUserData`; `AgentConfirmation.ratchetKeyId`; address creation taking `InitialKeys` (lifting the `IKUsePQ`-for-`SCMContact` prohibition), generating (`generateRcvE2EParams`, KEM per `initialPQEncryption False`) and storing the first `address_ratchet_keys` row. +2. Handshake: requester R2'/R3'; owner O1'/O2'/O3' storing the DR request as a `conn_invitations` row whose request value is the `CRConfirmation` variant; `smpConfirmation` `RcvConnection … Nothing` branch and `AgentConnInfoReply` acceptance; end-to-end connection with the profile under the ratchet. +3. Rotation and retrofit: schema migration, `rotateRatchetKeys`, retention window, cleanup step, app-driven `LSET` retrofit (with the full→short address migration), rotation/retrofit tests. diff --git a/rfcs/2026-07-12-address-pqdr-keys.md b/rfcs/2026-07-12-address-pqdr-keys.md index d4846c7c6..dad2bb76d 100644 --- a/rfcs/2026-07-12-address-pqdr-keys.md +++ b/rfcs/2026-07-12-address-pqdr-keys.md @@ -15,14 +15,11 @@ The cause is that the address owner's X3DH contribution is generated per request Publish the address owner's X3DH contribution in the address link data, so a requester can establish the double ratchet in its first message. The requester runs the existing `pqX3dhSnd` against the published keys, initializes a sending ratchet, and encrypts its first message under it. The owner runs the existing `pqX3dhRcv` against its stored private keys and the requester's X3DH keys from the message, initializes a receiving ratchet, and decrypts it. -The published keys follow the X3DH structure: +Publish the owner's X3DH contribution - two X448 keys and an optional sntrup761 KEM key, with the e2e version range - as one bundle in the mutable contact user data, signed by the root key. The bundle is the existing `RcvE2ERatchetParamsUri` type that a one-time invitation already advertises, with an id for rotation. -- The link ratchet key - the stable first X3DH key - goes in the immutable fixed link data, next to the root key, committed by the link hash. -- The prekey, the KEM encapsulation key, and the e2e version range go in the mutable user data as a ratchet-keys bundle with an id, signed by the root key, rotated with `LSET`. +This is backward compatible. A requester that does not use the published bundle sends a current `AgentInvitation` with its own X3DH keys, and the owner does what it does today: generates fresh X3DH keys and sends them in the confirmation. The owner branches on whether the incoming message uses the published bundle. -This is backward compatible. A requester that does not use the published keys sends a current `AgentInvitation` with its own X3DH keys, and the owner does what it does today: generates fresh X3DH keys and sends them in the confirmation. The owner branches on whether the incoming message uses the published keys. - -Two properties follow. The first message, including the profile, is under the double ratchet, which closes the profile gap and gives it post-quantum protection through the ratchet's sntrup761 KEM. And a decryptable message proves the sender established X3DH against the link ratchet key committed by the link hash, so it authenticates the address owner without a separate signature. +Three properties follow. The first message, including the profile, is under the double ratchet, which closes the profile gap and gives it post-quantum protection through the ratchet's sntrup761 KEM. A decryptable message proves the sender established X3DH against the root-signed keys, so it authenticates the address owner without a separate signature. And because the bundle is in mutable data, an existing address can advertise the double ratchet by updating its mutable data - no new link. ## Design @@ -30,27 +27,18 @@ Syntax uses [ABNF][1] with [case-sensitive strings extension][2]. Key and ratche ### Published keys in link data -The link ratchet key is appended to fixed link data: - -```abnf -fixedData =/ linkRatchetKey ; appended, ignored by earlier versions -linkRatchetKey = %s"0" / (%s"1" length x509encoded) ; X448, stable, the first X3DH key -``` - -The ratchet-keys bundle is appended to contact user data: +The owner's X3DH contribution is a ratchet-keys bundle appended to the mutable contact user data, signed by the root key. Nothing is added to the immutable fixed link data: ```abnf userContactData =/ ratchetKeys ; appended, ignored by earlier versions -ratchetKeys = %s"0" / (%s"1" ratchetKeyId e2eVRange prekey kemKey) +ratchetKeys = %s"0" / (%s"1" ratchetKeyId e2eParams) ratchetKeyId = shortString ; identifies this bundle, changes on rotation, echoed in the request -e2eVRange = -prekey = length x509encoded ; X448 -kemKey = %s"0" / (%s"1" largeString) ; sntrup761 encapsulation key, optional (PQ is opt-in) +e2eParams = ``` -Together these reconstruct the owner's X3DH contribution as an `RcvE2ERatchetParamsUri` - a version-range form: `linkRatchetKey` (from fixed data) as the first key, `prekey` as the second, `kemKey` as the optional KEM parameter, `e2eVRange` as the version range. The requester negotiates the concrete version against its own e2e range, as it does for an invitation, then runs `pqX3dhSnd` against the result. +`e2eParams` is the existing `RcvE2ERatchetParamsUri` - the same type a one-time invitation advertises - so a requester reads it, negotiates the concrete e2e version against its own range, and runs `pqX3dhSnd` against it, exactly as it does for an invitation. The KEM key is optional (the params' KEM field is a `Maybe`), controlled by the address's initial-keys mode, the same 3-way choice as invitations: advertise the KEM (post-quantum from the first message), advertise X448-only but still support post-quantum if the requester proposes its own KEM (one message later, avoiding the ~1158-byte key in link data), or no post-quantum. -The owner keeps the private prekey and, when PQ is on, the KEM keypair, indexed by `ratchetKeyId`. On rotation with `LSET` it publishes a new bundle with a new `ratchetKeyId` and keeps the previous private keys for a window covering queue message retention, so a request that used a just-rotated prekey still decrypts. The link ratchet key is stable and not rotated. +The owner keeps the private side of each bundle - the two X448 private keys and, when PQ is on, the KEM keypair - indexed by `ratchetKeyId`. On rotation with `LSET` it publishes a new bundle with a new `ratchetKeyId` and keeps the previous private keys for a window covering queue message retention, so a request that used a just-rotated bundle still decrypts. Because the bundle is in mutable data, an existing address advertises the double ratchet by updating its mutable data - no new link, no re-creation. ### Request confirmation @@ -66,26 +54,27 @@ The confirmation holds the requester's Snd X3DH parameters (so the owner runs `p Requester: -1. Retrieve link data (`LGET`), read the link ratchet key, the prekey with its `ratchetKeyId`, the `e2eVRange`, and the optional KEM key, reconstruct the owner's `RcvE2ERatchetParamsUri`, and negotiate the concrete version against its own e2e range. -2. `generateSndE2EParams` for its own X3DH contribution (with the KEM only when the address advertises one). -3. `pqX3dhSnd` against the owner's parameters, then `initSndRatchet` - the sending ratchet. +1. Retrieve link data (`LGET`), read the bundle - its `ratchetKeyId` and `e2eParams` (`RcvE2ERatchetParamsUri`) - and negotiate the concrete e2e version against its own range. +2. `generateSndE2EParams` for its own X3DH contribution - encapsulating to the bundle's KEM if it advertises one, or proposing its own KEM if the requester wants post-quantum and the bundle is X448-only. +3. `pqX3dhSnd` against the bundle's parameters, then `initSndRatchet` - the sending ratchet. 4. Encrypt the first message under the ratchet, and send a confirmation with its Snd parameters and `ratchetKeyId`. Owner: -1. On a confirmation with `ratchetKeyId` on a contact address, select the private prekey and, if any, KEM keypair by `ratchetKeyId` (current or a retained previous generation). -2. `pqX3dhRcv` against the requester's Snd parameters with the link ratchet, prekey, and optional KEM private keys, then `initRcvRatchet` - the receiving ratchet. Decrypting the first message advances the ratchet and gives it a send side, so the owner can reply. +1. On a confirmation with `ratchetKeyId` on a contact address, select the private X3DH keys and, if any, KEM keypair by `ratchetKeyId` (current or a retained previous generation). +2. `pqX3dhRcv` against the requester's Snd parameters with those private keys, then `initRcvRatchet` - the receiving ratchet. Decrypting the first message advances the ratchet and gives it a send side, so the owner can reply. 3. Decrypt, and reply under the ratchet. A request whose `ratchetKeyId` is no longer retained cannot be decrypted; the owner does not learn the requester or its reply address, and the requester's attempt fails at its own timeout. ### Authentication -The ratchet is established against the link ratchet key committed by the link hash and the prekey signed by the root key, so a decryptable message proves the sender established X3DH against the owner's published, link-committed keys. Where a message today relies on a separate signature over its content for authenticity, this establishment provides it, and the signature is not needed. The link ratchet key is a DH key (X448) and X3DH is over crypto_box, so deniability is preserved; the signing identity (the root key) and the DH key are separate keys, both anchored to the link hash. Reusing the link ratchet key across requesters is consistent with the address already being a shared identifier. +The ratchet-keys bundle is in the mutable link data, signed by the root key. A decryptable message proves the sender established X3DH against those root-signed keys: an SMP server cannot substitute them without forging the root signature (`decryptLinkData` verifies it), which is the X3DH anti-substitution property. Where a message today relies on a separate signature over its content for authenticity, this establishment provides it, and the signature is not needed. The root Ed25519 key is the address's signing identity; the X3DH keys are separate DH keys (X448), and X3DH is over crypto_box, so deniability is preserved. A malicious server can still serve an older but validly-signed bundle (rollback to a retired generation); this is bounded by the retention window and by the ratchet advancing after the first message, and a per-key signature would not prevent it. Reusing a bundle across requesters is consistent with the address already being a shared identifier. ## Uses - Invitations to an address, and the profile in them, are under the double ratchet from the first message. +- An existing address gains the double ratchet when the app updates its mutable link data (`LSET`, with the user's confirmation and current profile, combined with the full→short address migration); no new link is issued. - The service RPC (see the RPC RFC) establishes the ratchet this way to send the request as the first ratchet message. This RFC depends on nothing else here. It replaces the need for the PQ-queue RFC in the address case, because the ratchet provides post-quantum protection for the first message; the PQ-queue RFC remains for first messages that do not establish a ratchet.