Why GPU Memory Is Already Full Before It Has To Be
A short answer can tie up a lot of GPU memory. Paged attention splits the KV cache into blocks that a request receives when it needs them.
A short answer can tie up a lot of GPU memory. Paged attention splits the KV cache into blocks that a request receives when it needs them.