Configurable pacing rate modes via TransportConfig

[stablebits]

Add PacingRateMode enum to control rate calculation:

• RttDependent: rate = cwnd × 1.25 / RTT (default, standard QUIC)
• Fixed: constant bytes/sec rate, ignoring RTT and cwnd
• RttDependentWithFloor: max(floor, cwnd × 1.25 / RTT)

Fixed and RttDependentWithFloor are useful for latency-sensitive high-RTT connections.

With default Quinn's RTT-dependent pacing, cwnd bytes worth of traffic are spread over 80% of the RTT window. This approach doesn't impact overall throughput for large downloads, but it does impact latency. For example, given two connections with RTT of 10ms and 100ms sending 128K of data, and assuming both congestion and flow control allow this in 1 RTT, it will take 13ms (5ms one-way + 8ms pacing) and 130ms (50ms one-way + 80ms pacing) respectively until the last byte is delivered to the server. High-RTT clients are penalized twice: once by RTT, and again by the slower pacing rate.

With fixed-rate pacing (Fixed(rate)), both connections use the same pacing rate, so the delivery spread is independent of RTT.

[djc]

Thanks for working on this. I think this could be made easier to review:

• One commit for adding/clarifying documentation for the existing state
• One commit to replace the current logic with the new way of defining the current logic
• Any commits that add/extend tests that apply with the current logic
• One commit per new pacing mode

Please rebase and force push (avoiding merge commits). Note that our style prescribes top-down item ordering, so constants go near the bottom and high-level logic goes near the top.

[djc]

Also note

• Introduce max_outgoing_bytes_per_second option #2556

Does it still make sense to do both?

[Ralith]

Whoops, sorry, this slipped off my stack!

The motivation here makes sense to me in abstract, but it's not obvious to me that this patch is the best solution. Some thoughts:

• Do we really need three different cases here? A single option representing a lower bound to pacing rate seems sufficient for any use case I can think of.
• Is pacing at a higher rate than your true average link capacity actually a good idea? Choosing to pace at a high rate increases risk of packet loss and might interact poorly with other network users.
• Is pacing at a higher rate than your true average link capacity actually desired here, or is the problem that the initial congestion window estimate is too far below the size it should have? Have you experimented with adjusting the initial congestion window size?

Does it still make sense to do [this and max_outgoing_bytes_per_second]?

I think max_outgoing_bytes_per_second solves a different problem -- it sets an upper bound, whereas this PR is more concerned with various lower bounds.

[stablebits]

Whoops, sorry, this slipped off my stack!

The motivation here makes sense to me in abstract, but it's not obvious to me that this patch is the best solution.

Thank you for the feedback!

Here is some additional context for our use case:

Communication nodes are expected to have very good links (at least no last mile issues). Think of a distributed financial database. Senders have a short slot [t, t + duration) where duration is on the order of hundreds of ms to deliver their transactions, and the earlier they deliver them, the better.

Some senders have high RTT, which already puts them at a disadvantage. They can of course use a larger initial cwnd, or try to keep connections "hot" cwnd-wise one way or another. But the issue here is different: this is about sending the existing cwnd faster, not allowing more bytes in flight (which can be restricted on purpose by flow control, so cwnd won't grow further).

What we want here is to reduce the extra latency from spreading that already-allowed data over RTT.
New modes are meant as an explicit opt-in for (semi)controlled environments where lower latency for short bursts matters more than conservative RTT-scaled pacing, and where those burst can be tolerated.

Some senders even choose to relax or disable congestion control, since slow-start (or classical CC algos in general) is arguably not a great fit for this kind of workload. In that case, Fixed may actually be a cleaner option, since it at least gives them explicit pacing at a configured rate instead of effectively ending up with no pacing at all.

[Ralith]

It makes sense to me that, on a known network, a minimum pacing rate is useful. I still think we can simplify this API:

• RttDependent is RttDependentWithFloor with a floor of 0
• Fixed is RttDependentWithFloor with max_outgoing_bytes_per_second set to the same value.

Hence, it seems like the only thing we need to cover all cases is a "minimum pacing rate" setting. Does that make sense?