Position encoding
Extra information that tells a transformer the order of the words it reads. A transformer takes in all the words at once, which is fast but loses their order, and order changes meaning: "no fever, cough" says something different from "fever, no cough". The original transformer added to each word's embedding a wave-based signature unique to that word's position. Many current language models use a rotary scheme instead, which builds position into the way attention compares words.
A vendor announces that its chart summarizer now accepts four times as much text as before. A longer reach often comes from adjusting the position encodings, usually with extra training, so the model can handle positions it rarely or never met in pretraining. Accepting the longer input is one property, and using facts from every part of it reliably is another, which has to be tested. Before relying on it, place a known fact such as a documented allergy at the start, middle and end of long test records and check that the summary catches it each time.
Terms like this come up in real clinical scenarios across the HelloAI courses: bite-sized modules with verifiable certificates. An account takes one minute, no password needed.
Sign in →