Masked language modeling
A self-supervised pre-training task where part of the input tokens are hidden and the model is trained to predict them from the surrounding unmasked tokens. Because it can use context on both sides, it produces bidirectional encoders. It is the task BERT is pre-trained with.
An informatics team wants to flag radiology reports that recommend follow-up imaging, and picks an encoder that was pre-trained by hiding words in millions of clinical notes and learning to fill them back in. Because it learned to use the words on both sides of each gap, it reads each phrase in light of the whole sentence, which suits sorting reports into categories. An encoder trained this way is not built to write running text, so if a later proposal uses it to draft patient letters, ask which component actually generates the words.
Terms like this come up in real clinical scenarios across the HelloAI courses: bite-sized modules with verifiable certificates. An account takes one minute, no password needed.
Sign in →