Multi-head attention
Running several attention layers in parallel so each head can form a different filter and focus on a different part of the input. The heads' outputs are concatenated and passed through a linear layer to combine them.
A vendor demo highlights the words in a pathology report that the model "attended to", and the highlights look sensible. A model of modest size can run 12 attention heads in each of 12 layers, which gives 144 different weightings of the same text, and the demo shows one of them or an average. Each weighting shows how information was mixed at that point, which can differ from what drove the final result. Ask how the displayed map was chosen, and whether the vendor has tested that removing the highlighted words actually changes the output.
Terms like this come up in real clinical scenarios across the HelloAI courses: bite-sized modules with verifiable certificates. An account takes one minute, no password needed.
Sign in →