Learning curve (machine learning)

Learning curve plot of training set size vs training score (loss) and cross-validation score

In machine learning (ML), a learning curve (or training curve) is a graphical representation that shows how a model's performance on a training set (and usually a validation set) changes with the number of training iterations (epochs) or the amount of training data. Typically, the number of training epochs or training set size is plotted on the x-axis, and the value of the loss function (and possibly some other metric such as the cross-validation score) on the y-axis.

Synonyms include error curve, experience curve, improvement curve and generalization curve.

More abstractly, learning curves plot the difference between learning effort and predictive performance, where learning effort usually means the number of training samples, and predictive performance means accuracy on testing samples.

Learning curves have many useful purposes in ML, including:

• choosing model parameters during design,
• adjusting optimization to improve convergence,
• diagnosing problems such as overfitting (or underfitting),
• and determining the amount of data used for training.

Formal definition

When creating a function to approximate the distribution of some data, it is necessary to define a loss function $L(f_{\theta }(X),Y)$ to measure how good the model output is (e.g., accuracy for classification tasks or mean squared error for regression). We then define an optimization process which finds model parameters $\theta$ such that $L(f_{\theta }(X),Y)$ is minimized, referred to as $\theta ^{*}$.

Training curve for amount of data

If the training data is

$\{x_{1},x_{2},\dots ,x_{n}\},\{y_{1},y_{2},\dots y_{n}\}$

and the validation data is

$\{x_{1}',x_{2}',\dots x_{m}'\},\{y_{1}',y_{2}',\dots y_{m}'\}$,

a learning curve is the plot of the two curves

1. $i\mapsto L(f_{\theta ^{*}(X_{i},Y_{i})}(X_{i}),Y_{i})$
2. $i\mapsto L(f_{\theta ^{*}(X_{i},Y_{i})}(X_{i}'),Y_{i}')$

where $X_{i}=\{x_{1},x_{2},\dots x_{i}\}$

Training curve for number of iterations

Many optimization algorithms are iterative, repeating the same step (such as backpropagation) until the process converges to an optimal value. Gradient descent is one such algorithm. If $\theta _{i}^{*}$ is the approximation of the optimal $\theta$ after $i$ steps, a learning curve is the plot of

1. $i\mapsto L(f_{\theta _{i}^{*}(X,Y)}(X),Y)$
2. $i\mapsto L(f_{\theta _{i}^{*}(X,Y)}(X'),Y')$

Choosing the size of the training dataset

It is a tool to find out how much a machine model benefits from adding more training data and whether the estimator suffers more from a variance error or a bias error. If both the validation score and the training score converge to a value that is too low with increasing size of the training set, it will not benefit much from more training data.

In the machine learning domain, there are two implications of learning curves differing in the x-axis of the curves, with experience of the model graphed either as the number of training examples used for learning or the number of iterations used in training the model.

See also

• Overfitting
• Bias–variance tradeoff
• Model selection
• Cross-validation (statistics)
• Validity (statistics)
• Verification and validation
• Double descent