Q.What is the difference between cost function and loss function?
â€śLoss functionâ€ť is the loss for a single training set as compared to the â€ścost functionâ€ť which is the square root of mean of squares MSE for all the training sets taken together.
Hi Soumyadeep,
Actually, all of the three terms loss function, objective function and cost function are used interchangeably usually.
From â€śDeep Learningâ€ť book by Ian Goodfellow, Yoshua Bengio, Aaron Courville in section 4.3:
â€śThe function we want to minimize or maximize is called the objective function, or criterion. When we are minimizing it, we may also call it the cost function, loss function, or error function. In this book, we use these terms interchangeably, though some machine learning publications assign special meaning to some of these terms.â€ť
As per this book, at least, loss and cost are the same.

Yeah this point has been explained in one of the videos during the live QuestionAnswer Session , but donâ€™t recall it at this moment. However, I believe that this topic has been lucidly explained in the 1st video recording of Training Models by the Faculty Trainer  Sandeep Sirâ€¦

Following is the reply for your doubts: â€”

Note that Cost/Loss/Criterion function are one and the same. There is no difference between these terminologies.

These terms are used interchangeably

This is determined by how well the algorithm performs on the given training model or the training dataset.

The final outcome is that costfunction needs to be minimized i.e. y^ should be closer to y (i.e. predicted value should be close to actual value) i.e. y  y^ (Actual Value  Predicted Value).

Looking at the concepts explained explained prior to Training Models topic, SGD (Stochastic Gradient Descent) is basically a Costfunction

SGD  It is basically an Optimization Algorithm technique used for Training the datasets. In other words, when we are training a dataset, SGD works from behind the scenes.

Furthermore, SGD is used for minimizing the Cost Function. CostFunction is basically the Average Outcome of the Errors calculated from the given model. or dataset sample.

RSME( (Root Square Mean Error), MAE (Mean Average Error) are some of the other Cost Functions used in ML apart from SGD technique.