Regression techniques - Evaluating a Model's efficiency

Course Discussions Machine Learning
#1
  1. To evaluate a model’s efficiency using Regression ML technique, the following are also evaluated viz.:
    a) R-square
    b) Adjusted R-Square

Could you share some more gyaan about it? How do we incorporate and use this in Python?

  1. How do we explain the below terms to an Interviewer or to a colleague at office:
    a) RMSE (Root Mean Square Error)
    b) MAE (Mean Average Error)
    c) MAPE (Mean Absolute Percentage Error)
    d) MSLE (Mean Square Logarithmic Error)

  2. While discussing the Performance Metrics for Optimization techniques in SGD (Stochastic Gradient Descent), there is a Precision-Recall Tradeoff that is used. I believe F-statistic is associated with this.
    a) What is this F-statistic value???
    b) How is this F-Statistic value useful?

#2

Hi,

R^2 helps to find how model had fit numerically. Using the ratio of two sum of squares.

  1. Total Sum of the Squares(TSS) of (ground_truth-predicted_vallue)^2 or Vi [ (y-y’)^2] for all the inputs. let it be (S_total).
    Note :- here y’ is mean where Vi = For all i.

  2. The Residual Sum of Squares (RSS) is the Vi [ (y-y’)^2]
    Note :- ** here y’ is the predicted value of the model.

R^2 = ( 1-(RSS/TSS) )

Case1 :- In Best Case if the modeled values match exactly the observed values then RSS=0 so R^2=1. Mean all the points are on the line
Case2 :- A model that always predicts y’(mean)(Hypothetical case) have R^2=0.
Case3:- In worst case If Residuals RSS > TSS then it may go for negative R^2 values too.
Means all the points are far from the line.
Make sure R^2 Values should be more closer to 1 better the linear regressions model is.

Disadvantages –

When the number of independent variables (x1,x2,x3,x4) or features increases then there is possibility of more random noise. So, it leads to overfitting the model and gives High value of R^2.

So, we go for Adjusted R^2. or (R’)^2.

(R’)^2 = 1 - [ (1 - R^2) ( (n-1)/(n-p-1)) ]

p = Total number of the variables .
n = Sample size.

In terms of TSS and RSS.

(R’)^2 = 1 - [ (RSS)/dfe) / (RSS)/dfe) ]

Here dft= is what called degree of freedom (n-1).
dfe= Degrees of freedom (n-p-1).

So, Adjusted R"2 can be interpreted as an unbiased (or less biased) estimator of the population R2, whereas the observed sample R2 is a positively biased estimate.

And for the Tyes of the errors you can refer to the sklearn user guide.

https://scikit-learn.org/stable/modules/model_evaluation.html#mean-squared-error

F-static I suppose you are referring to F1 scores value is a measure of score in binary classification, which is the Harmonic mean of the “Precision” and “Recall” value.

It is a good measure of accuracy as it will consider both the Precisions and Recall into account.
Let Precisions§ = TP/(TP+FP)
Let Recall ® = TP/(TP+FN)

F1= [ 2*(P*R)/(P+R) ]

Best value is 1, and worst value is 0.
It tell how best our classifier has classified the data points.
and it is used in the Natural language processing for word segmentations and NER( Named entity recognition) that you will study later.

More read – https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html

All the best!

1 Like
#3

Good and precise explanation…Appreciate your efforts Satyajit!!!

All the Best in your endeavours…!!!

1 Like