Query, key, value
The three vectors an attention layer computes for every word, each from its own learned linear layer. Each word's query is compared with every word's key to produce a matrix of scores for how strongly the words relate, and those scores, turned into weights, decide how much of each word's value flows into the result. The recipe is scaled dot-product attention. Multiply queries by keys, divide by the square root of the key dimension (the length of each key vector in an attention head), apply softmax, and use the resulting weights to combine the values.
An IT team sizes an on-premises server for a language model from the model's size alone, and a pilot with short questions from a few users runs smoothly. Once clinicians start sending whole admission histories, responses slow and requests queue. While generating, the model keeps the key and value vectors for every token of every active request in GPU memory, so memory use grows with document length and with the number of simultaneous users. Ask for memory figures at your expected record length and peak concurrency, then load test with real-length notes.
Terms like this come up in real clinical scenarios across the HelloAI courses: bite-sized modules with verifiable certificates. An account takes one minute, no password needed.
Sign in →