How to Reduce LLM Inference Latency: Flash Attention, Speculative Decoding, and KV Cache Optimisation
The Three Components of LLM Latency LLM inference latency has three distinct components that require different optimisation strategies. Time to first token (TTFT) is the delay between sending a request and receiving the first token of the response — dominated by prefill time, the cost of processing all input tokens. Time per output token (TPOT) … Read more