Supervised Machine Learning is the process of determining the relationship between a given set of features (or variables) and a target value, which is also known as a label or a classification. Which means building ML Models that can take in certain input data and spit out a predicted value.

Let’s understand this by taking the example of – “Whether a bank should give House Loan to an applicant or not?” based upon information submitted by applicant to bank. Let’s assume that in the House Load Application, applicant submitted Age, Sex, Education Level, Income Level, Marital Status, Demographics, Previous Loans paid.

Age	Sex	Education Level	Income Level (per year)	Marital Status	Demographics	Previous Load Paid
30	Female	Undergraduate	98,000 USD	Yes	New York	Yes
27	Male	Undergraduate	120,000 USD	Yes	Austin	No
29	Female	High School	67,000 USD	No	LA	Yes
45	Male	None	54,000 USD	Yes	Baltimore	No
24	Female	None	43,000 USD	No	Georgia	No
64	Male	Undergraduate	180,000 USD	No	Bay Area	Yes

Based upon this data, a Supervised Machine Learning Model can be trained which can provide a yes or no answer to question “Give the loan to applicant?” for newer applicants.

Supervised Machine Learning models can be further divided into Classification Tasks and Regression Tasks.

Classification Tasks in Supervised Machine Learning

Classification Tasks are used to build models out of data with discrete categories as labels. For example – A classification task can be used to predict whether a person will pay back loan or not. You have more than two discrete categories, such as predicting ranking of a horde in a race, but they must be a finite number.

In the above image, Machine Learning Model is classifying observation in dataset as yellow, blue, blackish or pink.
Most classification tasks output the prediction as probability of an instance to belong to each output label. The assigned label is one with highest probability.

Supervised Classification Machine Learning Algorithms

1. Decision Trees – This algorithm follow a tree-like architecture that simulates decision process following a series of decisions, considering one variable at a time.
2. Naive Bayes Classifier – This algorithm relies on a group of probabilistic equations based on Bayes’ theorem, which assumes independence among features. It has ability to consider several attributes.
3. Artificial Neural Networks (ANNs) – These replicate the structure and performance of a biological neural network to perform pattern recognition tasks. An ANN consists of interconnected neurons, laid out with a set architecture. They pass information to one another until a result is achieved.

Regression Tasks in Supervised Machine Learning

Regression Tasks are used for data with continuous quantities as labels. For example – A regression task can be used for predicting house prices. This means that value is represented by a quantity and not by a set of possible outputs. Output labels can be of integer or float types.

Some commonly used Supervised Machine Learning algorithms for Regression Tasks are Linear Regression, Regression Trees, Support Vector Regression, Artificial Neural Networks (ANNs).

Evaluating Performance of a Supervised Machine Learning Model

Model evaluation is essential for the development of successful models that perform well not just on the data that was used to train the model, but also on data that it has not seen yet.  When dealing with supervised machine learning problems, the process of assessing the model is made particularly simple as ground truth is already known which can be compared to prediction of model.

When applying a model to unseen data that does not have a label class to compare it to, determining the accuracy percentage of the model is essential. For example, a model with an accuracy of 95% may lead you to believe that the chances of making an accurate forecast are great, and as a result, the model should be considered trustworthy. But definitely that assumption can be wrong as well because metric “accuracy of model” implies what? is known. Moreover a specific performance measurement metric for a Supervised Machine Learning Model should be selected on case by case basis. Because for some models it would be better to use one metric while same metric can imply something else for other model. So be careful while selecting a specific metric to measure performance of a model.

Evaluating models’ performance should be done on two types of datasets – Validation DataSet to fine-tune the model and Testing DataSet to evaluate how well model will function when applied to data which it does not know about.

Metrics used for Measuring Performance of a Supervised Classification Machine Learning Model

Confusion Metric

Confusion Matrix is a table that contains information about performance of a model. In Confusion Metric table columns represent instances which belong to a predicted class, while rows represent instances that actually belong to a class.

Let’s understand what exactly is “Confusion Metric” by taking an example of how many images in a given dataset of 500 images are of dogs.

Prediction👉 Actual👇	Dog Image	Not Dog Image
Dog Image	270	230
Not Dog Image	150	350

Each cell in a Confusion Matrix can be classified as True Positives (TP), False Positives (FP), True Negatives (TN), False Negatives (FN).

Cell of Confusion Metric	Description	Example
True Positives (TP)	Refers to instances that model correctly classified the event as positive	Correctly classifying image of the dog as god image
False Positives (FP)	Refers to the instances that model incorrectly classified the event as positive	Images of other animals being classified as Dog Images by model
True Negatives (TN)	Refers to the instances that model correctly classified event as negative	Images of other animals are being classified as not images of Dog by model
False Negatives (FN)	Refers to the instances that model incorrectly classified the event as negative	Images of dog being classified as not images of Dog by model

Accuracy

Accuracy of a model measures its capability to correctly classify all instances. It can be calculated by summing up number of True Positives (TP) and True Negatives (TN) then dividing by total number of instances.

Accuracy = (TP + TN)/Total Number of Instances

Precision

Precision measures the model’s ability to correctly classify positive labels by comparing it with total number of instances predicted as positive.

Precision can be calculated by taking ratio of True Positives (TP) and sum of True Positives (TP) and False Positives (FP).

Precision = TP divided by sum of TP, FP

Recall

Recall measures the number of correctly predicted positive labels against all positive labels.

Recall can be calculated by taking ratio of True Positives (TP) and sum of True Positives (TP) and False Negatives (FN).

Recall = TP divided by sum of TP, FN

Metrics used for Measuring Performance of a Supervised Regression Machine Learning Model

Considering that regression tasks are those where final output is continuous rather than being categorical, the performance of model can be measured by comparing predicted value with actual value.

For example – The performance of a Supervised Regression Machine Learning Model which makes prediction about price of a house in a locality can be measured by comparing actual price of house with that predicted by model.

Let’s say actual price of a house in a locality is 700,000 USD but our model is predicting price of that being 699,999 USD which is pretty close, so we can say that given model is efficient enough as difference between predicted and actual value is quite low.

For measuring this difference between predicted and actual values Mean Absolute Error (MAE) or Root Mean Squared Error (RMSE) can be used.

Mean Absolute Error (MAE)

It measures average absolute difference between predicted values and actual values, without taking into account the direction of error.

   

• m = Number of total instances
• y_i = Actual Value
• ŷ_i = Predicted Value

Actual House Price (yi)	Predicted House Price (ŷi)	Predicted – Actual (yi – ŷi)
500,000 USD	499,012 USD	988 USD
650,000 USD	590,918 USD	59082 USD
839,193 USD	832,039 USD	7154 USD
127,092 USD	120,043 USD	7049 USD
983,028 USD	980,832 USD	2196 USD
		= 76469 USD

MAE = 76469/5 = 15293.8 USD

Root Mean Square Error (RMSE)

It measures average magnitude of error between actual value and predicted value. It can be calculated by taking square root of average of squared difference between actual, predicted values.

   

• m = Number of total instances
• y_i = Actual Value
• ŷ_i = Predicted Value

Actual House Price (yi)	Predicted House Price (ŷi)	Predicted – Actual (yi – ŷi)	Predicted – Actual (yi – ŷi)2
500,000 USD	499,012 USD	988 USD	976144 USD
650,000 USD	590,918 USD	59082 USD	3490682724 USD
839,193 USD	832,039 USD	7154 USD	51179716 USD
127,092 USD	120,043 USD	7049 USD	49688401 USD
983,028 USD	980,832 USD	2196 USD	4822416 USD
			= 3597349401

RMSE = Square Root (3597349401/5) = Square Root (719469880.2) = 26822.93 USD

Which one metric out of MAE, RMSE to use for measuring the performance of Supervised Regression Machine Learning Model?

Both of Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) expresses the average error, in a range from 0 to infinity, where lower the value the better the performance of model. The main difference between these two metrics is that MAE assigns same weight of importance to all errors, while RMSE squares the error, assigning higher weight to larger errrors.

Consider this, RMSE metric is especially useful in cases where larger errors should be penalized, meaning that outliers are taken into account in the measurement of performance. For example – RMSE metric can be used when a value that is off by 4 is more than twice as bad as being off by 2. The MAE, on the other hand, is used when a value that is off by 4 is just twice as bad as a value off by 2.