Learn in the textbookTransfer of Learning and Representative Deep Learning Models
DefinitionPre-training (事前学習) is the process of training a model on large amounts of data before tailoring it to a specific use, so that it acquires features shared by many uses.
ExplanationFor example, a model may read a vast amount of text to learn how words connect, or view a wide variety of images to grasp features of shape and color. The resulting trained model (学習済みモデル) later serves as the base for adapting to other purposes. Because that base can be shared, there is no need to gather large amounts of data and train from scratch for every use.
AnalogyIt is like learning reading, writing and arithmetic at school, basics every job relies on, before taking up any particular job.
Memory hookThe "P" in GPT, a name used for conversational AI, is the initial of Pre-trained, from the same word as pre-training.