I had a small discussion with my friends on overfitting and we became confused over the two terms: "training accuracy" and "training loss (or cost)". This is the first time I've heard the term training accuracy. So far, I have only calculated accuracy on the validation and test sets.
My understanding is that training accuracy and training cost are just one thing, and more generally accuracy is only applied for classification problems.
Is that correct?
The concepts of "loss" and "accuracy" are NOT the same.
The loss function is what you minimize during the training of your model. There are many types of loss functions. You usually choose the loss function depending on the "task" you are facing; for instance, binary classification uses binary cross-entropy as loss function, multi-class classification uses categorical cross-entropy, regression uses mean squared-error (MSE) or mean absolute error (MAE).
Accuracy is a concept from classification (either binary classification or multiclass classification). It is defined as the percentage of correctly classified elements.
Of course, during training, when the loss decreases, we expect that the accuracy increases. This, however, is not always the case. That's why it is important to monitor both loss function and accuracy in both the training and validation data, so that we understand how our model actually performs.
Note that accuracy has its own problems representing the performance of a model. For instance, if a binary classification dataset has 95% positive elements and 5% negative elements, a classifier that ALWAYS classifies as positive will obtain a 95% accuracy. There are other measures that account for this, like the area under the ROC curve (AUC) or the F1.
These terms are all used interchangeably (unfortunately) but they are all referring to the same concepts ("training loss" and "training accuracy", which is not to be confused with a specific metric also called by that name - "accuracy"). "Accuracy" is the evaluation estimate of some metric, usually this would be using the same cost function that was used in training the model, however sometimes people use a different metric for this purpose. "Accuracy" is just a general english word meaning the quality or state of being correct or precise, that is how well the model predicts with the cost function of choice (on whichever data set), and I believe this is what was meant in your case as well, when referring to "training accuracy".
Cost is short for "value of the cost function", while accuracy is short for "predictive accuracy". Both of these terms are general and they do not represent any specific measures. Both of these can be, and usually are, computed on all sets (training/validation/testing, although only certain results may be reported in the end, as people usually do not care about training "cost").
The cost function is the specific measure you are using, such as MSE, and the estimate of it on some data is the "cost" or "predictive accuracy". While these are synonyms it might be better to use "cost" instead of "accuracy", as that is often times confused with the metric "accuracy", which is a specific measure used in classification.
A possible reason why people use "cost" for reporting training cost and "accuracy" when evaluating it is because they use different metrics on the same data. For ex. for a classification task someone might choose to train a model using binary cross-entropy but then evaluate it using accuracy metric (because cross-entropy is not as easy to interpret as accuracy), which would give you two different values for the same model and data. This is not good practice, you should evaluate a model using the same metric which was used for training it, as that is what was actually optimized by the model.
