How Confusion Matrix helps in Cyber Security

Have you been in a situation where you expected your machine learning model to perform really well but it sputtered out a poor accuracy? You’ve done all the hard work — so where did the classification model go wrong? How can you correct this?

There are plenty of ways to gauge the performance of your classification model but none have stood the test of time like the confusion matrix. It helps us evaluate how our model performed, where it went wrong and offers us guidance to correct our path.

In this story we will be exploring how confusion matrix a holistic view of the performance of our model. How, it can give a detailed insights of the model prediction and can be really useful for ML models in the security world.

What’s a Confusion Matrix?

A Confusion Matrix is a N x N matrix used for evaluating the performance of a classification model, where N is the no of target classes. This matrix clearly compares the actual target values with those predicted by the ML Model. This will give a holistic idea of how well our model is performing and where it is making errors.

Now, as binary classification has two values for target variable i.e 0 or 1/True or False. So, we have 2 x 2 matrix.

Now, here what is TP,FP,FN and TN?

Let’s start understanding one by one.

1. True Positive (TP): Here, if the machine predicted positive and then the actual value is also positive. This means the predicted value matches the actual value.
2. True Negative (TN): Here, the actual value is negative and the model predicted as negative. It also matches the output but it’s value is negative.
3. False Positive (FP): Here, the actual value is negative but the model predicted as Positive.
4. False Negative (FN): The actual value is positive but the model predicted as Negative. That means the predicted values are falsely predicted.

False Positive(FP) are known as Type 1 error and are most dangerous in the usecase like covid test or Cyber Attack. While False Negative is considered as Type 2 error.

We can derive accuracy using below formula:

Precision: It tells us how many of the correctly predicted cases actually turned out to be positive.

Recall: It tells us about the actual positive cases we were able to predict correctly with our model.

Note: Precision is a useful metric in cases where False Positive is a higher concern than False Negatives. While Recall is useful when False Negative trumps False Positive.

How ML Algos help in intrusion detection systems and how metrics play an important role

Cyber-attacks are becoming more sophisticated and thereby presenting increasing challenges in accurately detecting intrusions. Failure to prevent the intrusions could degrade the credibility of security services, e.g. data confidentiality, integrity, and availability. Numerous intrusion detection methods have been proposed in the literature to tackle computer security threats, which can be broadly classified into Signature-based Intrusion Detection Systems (SIDS) and Anomaly-based Intrusion Detection Systems (AIDS)

Machine learning is the process of extracting knowledge from large quantities of data. Machine learning models comprise of a set of rules, methods, or complex “transfer functions” that can be applied to find interesting data patterns, or to recognise or predict behaviour. Machine learning techniques have been applied extensively in the area of AIDS. Several algorithms and techniques such as clustering, neural networks, association rules, decision trees, genetic algorithms, and nearest neighbour methods, have been applied for discovering the knowledge from intrusion datasets.

There are majorly four main types of attack which can be classified. Denial of Service(DoS), Remote to User(R2L), User to Root (U2R), Probing(Information Gathering). In the KDD Cup 99, the criteria used for evaluation of the participant entries is the Cost Per Test (CPT) computed using the confusion matrix and a given cost matrix .A Confusion Matrix (CM) is a square matrix in which each column corresponds to the predicted class, while rows correspond to the actual classes. An entry at row i and column j, CM (i, j), represents the number of misclassified instances that originally belong to class i, although incorrectly identified as a member of class j. The entries of the primary diagonal, CM (i, i), stand for the number of properly detected instances. Cost matrix is similarly defined, as well, and entry C (i, j) represents the cost penalty for misclassifying an instance belonging to class i into class j. Cost matrix values employed for the KDD Cup 99 classifier learning contest are shown in Table 2. A Cost Per Test (CPT) is calculated by using the following formula:

Where CM and C is confusion matrix and cost matrix, respectively, and N represents the total number of test instances, m is the number of the classes in classification. The accuracy is based on the Percentage of Successful Prediction (PSP) on the test data set.

SVM implements the principle of Structural Risk Minimization by constructing an optimal separating hyper plane in the hidden feature space, using quadratic programming to find a unique solution. Originally SVM was developed for pattern recognition problems. Recently, a regression version of SVM has emerged as an alternative and powerful technique to solve regression problems by introducing an alternative loss function. Although SVM has been successfully applied in many fields, there is a conspicuous problem appeared in the practical application of SVM. In parallel SVM machine first we reduced nonclassified features data by distance matrix of binary pattern. From this concept, the cascade structure is developed by initializing the problem with a number of independent smaller optimizations and the partial results are combined in later stages in a hierarchical way, supposing the training data subsets and are independent among each other.

Result

True Positive (TP): The amount of attack detected when it is actually attack.

True Negative (TN): The amount of normal detected when it is actually normal.

False Positive (FP): The amount of attack detected when it is actually normal (False alarm).

False Negative (FN): The amount of normal detected when it is actually attack.

Comparison of detection rate: Detection Rate (DR) is given by.

Comparison of False Alarm Rate: False Alarm Rate (FAR) refers to the proportion that normal data is falsely detected as attack behavior.

Confusion matrix contains information actual and predicted classifications done by a classifier. The performance of cyber attack detection system is commonly evaluated using the data in a matrix.

Thank You 😃