Attention mechanism
The core component of a transformer that lets the model weigh how much every part of the input matters to every other part. It builds an attention filter from the inputs and uses it to give some words more weight than others when forming a representation.
A tool that moves diagnoses from clinic notes onto the problem list has to tell "no history of diabetes", "family history of breast cancer" and "breast cancer, on tamoxifen" apart. Attention lets the representation of each word draw on the words around it, so the model can register that "no" or "family history" changes what follows, which a plain keyword search cannot. It still slips on phrasing it rarely saw in training. Before go-live, build a test set full of negated, historical and family-history mentions and count the errors by type.
Terms like this come up in real clinical scenarios across the HelloAI courses: bite-sized modules with verifiable certificates. An account takes one minute, no password needed.
Sign in →