Pre-training (事前学習) is the process of training a model on large amounts of data before tailoring it to a specific use, so that it acquires features shared by many uses.
For example, a model may read a vast amount of text to learn how words connect, or view a wide variety of images to grasp features of shape and color. The resulting trained model (学習済みモデル) later serves as the base for adapting to other purposes. Because that base can be shared, there is no need to gather large amounts of data and train from scratch for every use.
AnalogyIt is like learning reading, writing and arithmetic at school, basics every job relies on, before taking up any particular job.
How to rememberThe "P" in GPT, a name used for conversational AI, is the initial of Pre-trained, from the same word as pre-training.
Exam points
ExamA large-scale model that has completed pre-training (事前学習) on diverse, massive data and can later be adjusted and reused for many purposes is called a foundation model (基盤モデル).
Fine-tuning (ファインチューニング) means giving a pre-trained modeladditional training on data suited to the target task so that it is adjusted for that use.
Pre-trained model
Prepare data suited to the goal
Adjust weights by further training
Model for the specific use
The result is a model that is strong at that task and still keeps the features it learned.
4 / 4
Instead of rebuilding the model from the start, it keeps the features already learned and shifts parameters such as the weights toward the goal. As a result, it tends to need less data and less computing time than training from scratch. One use is to have a language model trained on general text learn a hospital's consultation records so that its answers handle medical terms well.
AnalogyIt is like a new employee with a solid grounding from school learning, through a short training period, the way their assigned department does its work.
How to remember"Fine" means precise, and "tuning" is what you do to a musical instrument. The name likens it to the last careful adjustment of a finished instrument's pitch.
Exam points
ExamBy adjusting a foundation model (基盤モデル) for each use through fine-tuning (ファインチューニング) or other additional training, many kinds of AI can be built from a single base.
Mix-upPre-training (事前学習) is the first training, which builds a base from broad data, while fine-tuning (ファインチューニング) is the later training done on that base with goal-specific data.
Transfer learning (転移学習) is a method that carries the knowledge of a model trained on one task over to a related, different task, so that learning the new task goes efficiently.
Examples of how transfer learning is done
Use the trained part as it is
Train only the layers near the output on new data
Also adjust the trained part's weights
Update the weights bit by bit with fine-tuning
For example, an image recognition model trained to tell many kinds of objects apart can be put to a different task: spotting scratches on products in a factory. Even with little data for the new task, it can draw on the shape and pattern features it has already learned, so it tends to reach good accuracy in less time than training from scratch. Fine-tuning (ファインチューニング) with a trained model as the base is one of the main ways to achieve transfer learning.
AnalogyIt is like someone who has learned the piano using their feel for the keys and their ability to read music to pick up the electronic organ quickly.
How to rememberIn English it is transfer learning. "Transfer" is the same word used for changing trains, so you can picture knowledge switching over to another line.
Exam points
Mix-upTransfer learning (転移学習) refers to the idea of moving knowledge to another task, while fine-tuning (ファインチューニング) refers to the concrete procedure of adjusting the already-trained weights at the destination.
CNN stands for Convolutional Neural Network, a deep learning model widely used for image recognition.
It stacks many layers of processing that pick up features from each small area of an image, building step by step from simple shapes such as lines and corners to whole shapes such as eyes or wheels. It is used to classify objects in photos, find where they are, and recognize faces. Its inner workings are covered under its full name, convolutional neural network (畳み込みニューラルネットワーク).
AnalogyIt is like dividing a large map into small squares, checking each one for roads or rivers, and then working out from the whole which area the map shows.
How to rememberThe C is for Convolution, and the R in RNN is for Recurrent, so the first letter alone tells you how the two work differently.
Exam points
Mix-upA CNN is good at capturing the features of data laid out in a two-dimensional grid, like images, while an RNN is good at capturing the flow of data laid out in order, like text.
Convolutional neural network (畳み込みニューラルネットワーク) is the full name of CNN: a neural network that extracts the local features of an image through a process called convolution.
Input image
Convolution layer (extracts local features)
Pooling layer (shrinks features)
Fully connected layer (decides what is shown)
Based on the collected features, the last layer outputs the classification.
4 / 4
In convolution (畳み込み), a small filter is slid across the image bit by bit, and whether a particular pattern appears in each area is turned into a number. The pooling (プーリング) step that follows merges nearby values to shrink the features, and after several such layers, the last layer decides what the image shows. Because the same feature is easy to find even when an object shifts slightly, it suits work that deals with images.
AnalogyIt is like sliding a small paper template across a cloth from one edge to the other and marking each spot where the same design appears.
How to rememberThe Japanese term 畳み込み translates the mathematical term convolution, which has long named the calculation that applies a filter to sound and image signals.
Exam points
Mix-upThe core of a convolutional neural network (畳み込みニューラルネットワーク) is the convolution layer, which slides filters across the input to extract local features.
RNN stands for Recurrent Neural Network, a deep learning model suited to data whose order matters, such as text, speech and time-series data.
It keeps information from processing the previous input inside itself and handles it together with the next input, so it can make judgments that take the flow so far into account. For example, it is used to predict the next word in a sentence, turn speech into text, and forecast what comes next from a sales trend.
How to rememberRecurrent is an English word for something that keeps happening again. The name comes straight from the way information circulates repeatedly inside the network.
Exam points
Mix-upA weakness of an RNN is that older information fades over long sequences, and LSTM makes up for this by adding a mechanism that controls whether to keep or forget information.
CNN vs. RNN
Aspect
CNN
RNN
Key differenceData it handles well
Data in a two-dimensional grid, such as images
Ordered data such as text and speech
Core mechanism
Convolution, which extracts local features with filters
A loop that feeds the previous step's state into the next
Recurrent neural network (リカレントニューラルネットワーク) is the full name of RNN: a neural network that feeds the internal state created at one time step back into the processing of the next step.
Process the first input
Pass the state to the next step
Process the next input with the previous state
Repeat to the last input, then output
This repeats to the end, so the output reflects the flow up to that point.
4 / 4
Because there is a path that sends each result back into the next round of processing, the internal state is passed along as a summary of the data seen so far. The same weights are reused at every time step, so texts and time series of different lengths can be handled by the same mechanism.
AnalogyIt is like rereading the previous day's page when writing a diary, then adding today's entry so that it follows on from it.
Exam points
Mix-upA recurrent neural network (リカレントニューラルネットワーク) has a path that sends a layer's output back to itself, which sets it apart from networks where data only flows one way from input to output.
Key points
Starting from a general-purpose model built by pre-training makes it easier to create a model suited to the target task, even with little data.
Fine-tuning does not rebuild the model from scratch; it adjusts the already-trained weights toward the goal.
Transfer learning is the idea of putting knowledge gained on one task to work on a related, different task.
Convolutional neural network is the full name of CNN and recurrent neural network is the full name of RNN; each refers to the same model as its abbreviation.