Articles
How models work and what turns a model call into a reliable application.
An extraction reliably pulls values out of contracts, until one field is missing. The same call still returns a number, cleanly formatted, just invented. Why a language model is not a database, and what that means for checking its output.
A token is the smallest unit a language model splits text into. Why the model sees no letters, and what that one fact explains about cost and errors.
A server can reuse the opening of a request instead of recomputing it. That only works if the opening is identical, token for token. From the first differing token on, the saving is gone.
The question hides three different questions: where the data goes, how the model behaves, and what the company allows. Only the first one is settled by where the model runs.
For every single token, the GPU re-reads the entire set of model weights just to compute that one token. Speculative decoding lets a small model guess ahead and saves exactly that wait.
Fewer bits save space for model weights. Comparing the intended tasks shows whether answers stay good enough and whether the model actually runs faster.
A short answer can tie up a lot of GPU memory. Paged attention splits the KV cache into blocks that a request receives when it needs them.
Eight requests start at the same time on the same server. The shortest one is done after three tokens. Its answer still does not arrive until the longest of the eight has written its last word.
In June an open model cracked 50 points on the leading independent AI benchmark for the first time; six weeks later a second one came within three points of the frontier. Why the average number is still almost irrelevant for your own decision.
Same prompt, same model, twice, two different answers. Not a bug: the model draws from a probability distribution at every word. What temperature, top-p, and top-k actually set.
One request answers instantly, then drips out word by word. Another takes a while before the first word appears, then runs through fast. The reason: two phases with different bottlenecks.
An AI model never sees an image as pixels. It cuts it into a grid of tiles that become numbers, and reads them like tokens. Why the fine print disappears first.
Is your own server for open AI models worth it, or is a subscription cheaper? The honest cost and hardware math for the Mittelstand, as of 2026.
Same model, same question. Every answer costs more the longer the conversation runs. The reason has a name: KV cache.
