streaming-tts-serving

VITS on Triton with a decoupled C++ backend, TensorRT FP16, and a Go gateway. Everything below is a real artifact from the benchmark runs, not a mockup.

Source on GitHub · measured on 2× NVIDIA RTX 6000 Ada

113.8 ms

p99 time to first audio

3,200

held sessions, 10% duty

6.91×

whole-pipeline GPU time

underruns / rejections

Does FP16 still sound right?

The optimization only counts if the audio survives it. Same text, same latents, same seed — decoded through stock PyTorch FP32 and through the TensorRT FP16 engine. Measured SNR between them is 42.3 dB, log-spectral distance 1.52 dB. Play them back to back.

Sure, I can help with that.

PyTorch FP32

TensorRT FP16

Your flight leaves at four fifteen from gate B twelve, and boarding starts about forty minutes before that.

PyTorch FP32

TensorRT FP16

Doctor Chen's invoice came to twelve hundred forty seven dollars and fifty cents, due on March fourteenth.

Written input was Dr. Chen's invoice came to $1,247.50, due on 3/14. — the text normalizer expands currency, dates and abbreviations before the model sees them.

PyTorch FP32

TensorRT FP16

The difference between them, amplified

FP32 minus FP16, gained back up to audible level. This is what 42 dB of SNR actually is: a faint ghost of the sentence, with the error concentrated on fricatives and onsets rather than spread evenly. Quiet enough not to matter, structured enough to prove it is precision loss and not noise.

Audio starts before synthesis finishes

A normal Triton model returns one response per request, so the first sample arrives only after the last one is computed. This one runs decoupled: the C++ backend decodes the utterance in overlapping chunks and pushes each one out as it is ready. The clip below is 6.26 seconds long. The first chunk left the server at 132 ms.

Streamed output, reassembled from 10 chunks

chunk arrival

Ten chunks, all delivered within 178 ms, for six seconds of speech. Chunk size ramps up after the first so latency is spent where a listener notices it. Single client, warmed, NVIDIA A40 (results/m5_stream.json).

Table view

Chunk	Arrival (ms)	Gap (ms)
1	132	—
2	139	7
3	147	8
4	154	7
5	157	3
6	160	3
7	163	3
8	166	3
9	172	6
10	178	6

Latency under load

Held sessions against time-to-first-audio, two GPUs, 10% duty cycle. The p50 barely moves as load triples — it is the tail that pays, which is the whole reason the target was written as p99 rather than as a mean.

p50 p90 p99

Server-side latency: the load generator runs on the same host, so wide-area network time is excluded. 15,599 requests at the top level, zero errors (results/m12_3200_2gpu.json).

Table view

Held sessions	p50	p90	p99	max	Requests
1,600	24.2	32.7	44.6	113.7	7,700
2,400	25.3	34.7	60.7	164.3	11,678
3,200	26.3	37.6	113.8	214.8	15,599

What it cost to get there

Whole pipeline, PyTorch against TensorRT, CUDA-event timed on the same utterances in the exact chunk geometry the server uses. Weighted by audio produced rather than by utterance count, so a two-word reply does not count the same as a ten-second one.

Utterance	Audio	PyTorch	TensorRT	Speedup
short	1.82 s	66.2 ms	10.2 ms	6.46×
medium	5.94 s	101.1 ms	14.3 ms	7.09×
long	9.79 s	133.3 ms	19.3 ms	6.92×
list	6.56 s	104.0 ms	14.8 ms	7.05×
weighted	24.11 s	404.6 ms	58.5 ms	6.91×

That is 85.5% less GPU time per request, or $0.000252 down to $0.000036 per audio-minute at $0.90/GPU-hour — counting GPU occupancy alone. Real cost falls by less, because CPU, network and the headroom deliberately kept to protect the tail do not shrink with kernel time.

Read this before quoting any of it

The load generator runs on the same host as the gateway. These are server-side latencies and exclude wide-area network time.
A held session synthesizes 10% of the time, like a voice agent taking turns. 3,200 held is roughly 320 speaking at once. At 100% duty, one card holds 128.
All numbers are RTX 6000 Ada. The same code went from 4 to 128 continuously speaking sessions moving from an A40, so serving ceilings do not transfer between cards. An A100 measured 16% worse per audio-minute at double the price.
The model is facebook/mms-tts-eng — 36M parameters, 16 kHz. A larger model moves every number here.
Cross-request batching is off in the TensorRT front half; the batched alignment is wrong for B>1 because each item predicts its own frame count. It costs throughput at 100% duty and nothing at 10%.

Full write-ups: RESULTS · ARCHITECTURE · PROFILE · NOTES