Don’t Waste Your Labeled Anomalies: 3 Practical Strategies to Boost Anomaly Detection Performance

algorithms assume you’re working with fully unlabeled information.

However in case you’ve truly labored on these issues, you understand the truth is usually completely different. In observe, anomaly detection duties typically include no less than a number of labeled examples, possibly from previous investigations, or your subject material professional flagged a few anomalies that can assist you outline the issue extra clearly.

In these conditions, if we ignore these priceless labeled examples and stick to these purely unsupervised strategies, we’re leaving cash on the desk.

So the query is, how can we truly make use of these few labeled anomalies?

For those who search the tutorial literature, you’ll find it is filled with intelligent options, particularly with all the brand new deep studying strategies popping out. However let’s be actual, most of these options require adopting solely new frameworks with steep studying curves. They often contain a painful quantity of unintuitive hyperparameter tuning, and nonetheless may not carry out nicely in your particular dataset.

On this put up, I wish to share three sensible methods that you may begin utilizing straight away to spice up your anomaly detection efficiency. No fancy frameworks required. I’ll additionally stroll by a concrete instance on fraud detection information so you’ll be able to see how one in every of these approaches performs out in observe.

By the top, you’ll have a number of actionable strategies for making higher use of your restricted labeled information, plus a real-world implementation you’ll be able to adapt to your personal use instances.

1. Threshold Tuning

Let’s begin with the lowest-hanging fruit.

Most unsupervised fashions output a steady anomaly rating. It’s solely as much as you to resolve the place to attract the road to differentiate the “regular” and “irregular” courses.

This is a crucial step for a sensible anomaly detection resolution, as choosing the improper threshold may end up in both lacking essential anomalies or overwhelming operators with false alarms. Fortunately, these few labeled irregular examples can present some steering in correctly setting this threshold.

The important thing perception is that you should utilize these labeled anomalies as a validation set to quantify detection efficiency underneath completely different threshold selections.

Right here’s how this works in observe:

Step (1): Proceed along with your standard mannequin coaching & thresholding on the dataset excluding these labeled anomalies. If in case you have curated a pure regular dataset, you may wish to set the brink as the utmost anomaly rating noticed within the regular information. In case you are working with unlabeled information, you’ll be able to set the brink by selecting a percentile (e.g., ninety fifth or 99th percentile) that corresponds to your tolerated false constructive charge.

Step (2): Together with your labeled anomalies put aside, you’ll be able to calculate concrete detection metrics underneath your chosen threshold. These embody recall (what proportion of identified anomalies could be caught), precision, and recall@ok (helpful when you’ll be able to solely examine the highest ok alerts). These metrics offer you a quantitative measure of whether or not your present threshold yields acceptable detection efficiency.

💡Professional Tip: If the variety of your labeled anomalies is small, the estimated metrics (e.g., recall) would have excessive variances. A extra sturdy means right here could be to report its uncertainty through bootstrapping. Primarily, you’re creating many “pseudo-datasets” by randomly sampling identified anomalies with substitute, re-compute the metrics for each replicate, and derive the boldness interval from the distribution (e.g., seize the two.5-th and 97.5-th percentiles, which supplies you 95% confidence interval). These uncertainty estimates would provide the trace of how reliable these computed metrics are.

Step (3): In case you are not glad with the present detection efficiency, now you can actively tune the brink primarily based on these metrics. In case your recall is just too low (which means that you simply’re lacking too many identified anomalies), you’ll be able to decrease the brink. For those who’re catching most anomalies however the false constructive charge is greater than acceptable, you’ll be able to increase the brink and measure the trade-off. The underside line is that you may now discover the optimum steadiness between false positives and false negatives to your particular use case, primarily based on actual efficiency information.

✨ Takeaway

The power of this method lies in its simplicity. You’re not altering your anomaly detection algorithm in any respect – you’re simply utilizing your labeled examples to intelligently tune a threshold you’ll have needed to set anyway. With a handful of labeled anomalies, you’ll be able to flip threshold choice from guesswork into an optimization drawback with measurable outcomes.

2. Mannequin Choice

In addition to tuning the brink, the labeled anomalies may information the number of higher mannequin selections and configurations.

Mannequin choice is a typical ache level each practitioner faces: with so many anomaly detection algorithms on the market, every with their very own hyperparameters, how have you learnt which mixture will truly work nicely to your particular drawback?

To successfully reply this query, we’d like a concrete solution to measure how nicely completely different fashions and configurations carry out on the dataset we’re investigating.

That is precisely the place these labeled anomalies turn out to be invaluable. Right here’s the workflow:

Step (1): Prepare your candidate mannequin (with a particular set of configurations) on the dataset, excluding these labeled anomalies, similar to what we did with the brink tuning.

Step (2): Rating the complete dataset and calculate the typical anomaly rating percentile of your identified anomalies. Particularly, for every of the labeled anomalies, you calculate what percentile it falls into of the distribution of the scores (e.g., if the rating of a identified anomaly is greater than 95% of all information factors, it’s on the ninety fifth percentile). Then, you common these percentiles throughout all of your labeled anomalies. This fashion, you acquire a single metric that captures how nicely the mannequin pushes identified anomalies towards the highest of the rating. The upper this metric is, the higher the mannequin performs.

Step (3): You’ll be able to apply this method to establish essentially the most promising hyperparameter configurations for a particular mannequin sort you take into consideration (e.g., Native Outlier Issue, Gaussian Combination Fashions, Autoencoder, and so on.), or to pick out the mannequin sort that greatest aligns along with your anomaly patterns.

💡Professional Tip: Ensemble studying is more and more frequent in manufacturing anomaly detection programs. This paradigm means as an alternative of counting on one single detection mannequin, a number of detectors, presumably with completely different mannequin varieties and completely different mannequin configurations, run concurrently to catch various kinds of anomalies. On this case, these labeled irregular samples might help you gauge which candidate mannequin occasion truly deserve a spot in your last ensemble.

✨ Takeaway

In comparison with the earlier threshold tuning technique, this present mannequin choice technique strikes from “tuning what you’ve got” to “selecting what to make use of.”

Concretely, through the use of the typical percentile rating of your identified anomalies as a efficiency metric, you’ll be able to objectively examine completely different algorithms and configurations by way of how nicely they establish the varieties of anomalies you truly encounter. Because of this, your mannequin choice is not a trial-and-error course of, however a data-driven decision-making course of.

3. Supervised Ensembling

Up to now, we’ve been discussing methods the place the labeled anomalies are primarily used as a validation device, both for tuning the brink or choosing promising fashions. We are able to, after all, put them to work extra straight within the detection course of itself.

That is the place the concept of supervised ensembling is available in.

To higher perceive this method, let’s first talk about the instinct behind this technique.

We all know that completely different anomaly detection strategies typically disagree about what seems to be suspicious. One algorithm may flag “anomaly” at a knowledge level whereas one other may say it’s completely regular. However right here’s the factor: these disagreements are fairly informative, as they inform us so much about that information level’s anomaly signature.

Let’s think about the next situation: Suppose now we have two information factors, A and B. For information level A, it triggers alarms in a density-based methodology (e.g., Gaussian Combination Fashions) however passes by an isolation-based one (e.g., Isolation Forest). For information level B, nevertheless, each detectors set off the alarm. Then, we’d typically imagine these two factors carry fully completely different signatures, proper?

Now the query is the right way to seize these signatures in a scientific means.

Fortunately, we are able to resort to supervised studying. Right here is how:

Step (1): Begin by coaching a number of base anomaly detectors in your unlabeled information (excluding your treasured labeled examples, after all).

Step (2): For every information level, accumulate the anomaly scores from all these detectors. This turns into your characteristic vector, which is actually the “anomaly signatures” we purpose to mine from. To present a concrete instance, let’s say you used three base detectors (e.g., Isolation Forest, GMM, and PCA), then the characteristic vector for a single information level i would appear to be this:

X_i=[iForest_score, GMM_score, PCA_score]

The label for every information level is easy: 1 for the identified anomalies and 0 for the remainder of the samples.

Step (3): Prepare an ordinary supervised classifier utilizing these newly composed characteristic vectors as inputs and the labels because the goal outputs. Though any off-the-shelf classification algorithm may in precept work, a typical suggestion is to make use of gradient-boosted tree fashions, reminiscent of XGBoost, as they’re adept at studying complicated, non-linear patterns within the options, and they’re sturdy towards the “noisy” labels (understand that most likely not all of the unlabeled samples are regular).

As soon as educated, this supervised “meta-model” is your last anomaly detector. At inference time, you run new information by all base detectors and feed their outputs to your educated meta-model for the ultimate resolution, i.e., regular or irregular.

✨ Takeaway

With the supervised ensembling technique, we’re shifting the paradigm from utilizing the labeled anomalies as passive validation instruments to creating them lively members within the detection course of. The meta-classifier mannequin we constructed learns how completely different detectors reply to anomalies. This not solely improves detection accuracy, however extra importantly, provides us a principled solution to mix the strengths of a number of algorithms, making the anomaly detection system extra sturdy and dependable.

For those who’re pondering of implementing this technique, the excellent news is that the PyOD library already offers this performance. Let’s check out it subsequent.

4. Case Examine: Fraud Detection

On this part, let’s undergo a concrete case examine to see the supervised ensemble technique in motion. Right here, we think about a technique referred to as XGBOD (Excessive Gradient Boosting Outlier Detection), which is applied within the PyOD library.

For the case examine, we think about a bank card fraud detection dataset (Database Contents License) from Kaggle. This dataset accommodates transactions made by bank cards in September 2013 by European cardholders. In complete, there are 284,807 transactions, 492 of that are frauds. Word that on account of confidentiality points, the options offered within the dataset aren’t unique, however are the results of a PCA transformation. Function ‘Class’ is the response variable. It takes the worth 1 in case of fraud and 0 in any other case.

On this case examine, we think about three studying paradigms, i.e., unsupervised studying, XGBOD, and absolutely supervised studying, for performing anomaly detection. We’ll fluctuate the “supervision ratio” (proportion of anomalies which are accessible throughout coaching) for each XGBOD and the supervised studying method to see the impact of leveraging labeled anomalies on the detection efficiency.

4.1 Import Libraries

For unsupervised anomaly detection, we think about 4 algorithms: Principal Part Evaluation (PCA), Isolation Forest, Cluster-based Native Outlier Issue (CBLOF), and Histogram-based Outlier Detection (HBOS), which is an environment friendly detection methodology that assumes characteristic independence and calculates the diploma of outlyingness by constructing histograms. All algorithms are applied within the PyOD library.

For the supervised studying method, we use an XGBoost classifier.

import pandas as pd
import numpy as np

# PyOD imports
# !pip set up pyod
from pyod.fashions.xgbod import XGBOD
from pyod.fashions.pca import PCA
from pyod.fashions.iforest import IForest
from pyod.fashions.cblof import CBLOF
from pyod.fashions.hbos import HBOS

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (precision_recall_curve, average_precision_score,
                             roc_auc_score)
# !pip set up xgboost
from xgboost import XGBClassifier

4.2 Knowledge Preparation

Keep in mind to obtain the dataset from Kaggle and retailer it regionally underneath the title “creditcard.csv”.

# Load information
df = pd.read_csv('creditcard.csv')      
X, y = df.drop(columns='Class').values, df['Class'].values

# Scale options
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Cut up into practice/check
X_train, X_test, y_train, y_test = train_test_split(
    X_scaled, y, test_size=0.3, random_state=42, stratify=y
)

print(f"Dataset form: {X.form}")
print(f"Fraud charge (%): {y.imply()*100:.4f}")
print(f"Coaching set: {X_train.form[0]} samples")
print(f"Check set: {X_test.form[0]} samples")

Right here, we create a helper perform to generate labeled information for XGBOD/XGBoost studying.

def create_supervised_labels(y_train, supervision_ratio=0.01):
    """
    Create supervised labels primarily based on supervision ratio.
    """
    
    fraud_indices = np.the place(y_train == 1)[0]
    n_labeled_fraud = int(len(fraud_indices) * supervision_ratio)
    
    # Randomly choose labeled samples
    labeled_fraud_idx = np.random.selection(fraud_indices, 
                                         n_labeled_fraud, 
                                         exchange=False)
    
    # Create labels
    y_labels = np.zeros_like(y_train)
    y_labels[labeled_fraud_idx] = 1

    # Calculate what number of true frauds are within the "unlabeled" set
    unlabeled_fraud_count = len(fraud_indices) - n_labeled_fraud

    return y_labels, labeled_fraud_idx, unlabeled_fraud_count

Word that this perform mimics the practical situation the place now we have a number of identified anomalies (labeled as 1), whereas all different unlabeled samples are handled as regular (labeled as 0). This implies our labels are successfully noisy, since some true fraud instances are hidden among the many unlabeled information however nonetheless obtain a label of 0.

Earlier than we begin our evaluation, let’s outline a helper perform for evaluating mannequin efficiency:

def evaluate_model(mannequin, X_test, y_test, model_name):
    """
    Consider a single mannequin and return metrics.
    """
    # Get anomaly scores
    scores = mannequin.decision_function(X_test)
    
    # Calculate metrics
    auc_pr = average_precision_score(y_test, scores)
    
    return {
        'mannequin': model_name,
        'auc_pr': auc_pr,
        'scores': scores
    }

In PyOD framework, each educated mannequin occasion exposes a decision_function() methodology. By calling it on the inference samples, we are able to acquire the corresponding anomaly scores.

For evaluating efficiency, we use AUCPR, i.e., the world underneath the precision-recall curve. As we’re coping with a extremely imbalanced dataset, AUCPR is usually most well-liked over AUC-ROC. Moreover, utilizing AUCPR eliminates the necessity for an specific threshold to measure mannequin efficiency. This metric already incorporates mannequin efficiency underneath numerous threshold circumstances.

4.3 Unsupervised Anomaly Detection

fashions = {
    'IsolationForest': IForest(random_state=42),
    'CBLOF': CBLOF(),
    'HBOS': HBOS(),
    'PCA': PCA(),
}

for title, mannequin in fashions.gadgets():
    print(f"Coaching {title}...")
    mannequin.match(X_train)
    end result = evaluate_model(mannequin, X_test, y_test, title)
    print(f"{title:20} - AUC-PR: {end result['auc_pr']:.4f}")

The outcomes we obtained are as follows:

IsolationForest: – AUC-PR: 0.1497

CBLOF: – AUC-PR: 0.1527

HBOS: – AUC-PR: 0.2488

PCA: – AUC-PR: 0.1411

With zero hyperparameter tuning, not one of the algorithms delivered very promising outcomes, as their AUCPR values (~0.15–0.25) could fall wanting the very excessive precision/recall typically required in fraud-detection settings.

Nonetheless, we must always word that, not like AUC-ROC, which has a baseline worth of 0.5, the baseline AUCPR relies on the prevalence of the constructive class. For our present dataset, since solely 0.17% of the samples are fraud, a naive classifier that guesses randomly would have an AUCPR ≈ 0.0017. In that sense, all detectors already outperform random guessing by a large margin.

4.4 XGBOD Strategy

Now we transfer to the XGBOD method, the place we’ll leverage a number of labeled anomalies to tell our anomaly detection.

supervision_ratios = [0.01, 0.02, 0.05, 0.1, 0.15, 0.2]

for ratio in supervision_ratios:

    # Create supervised labels
    y_labels, labeled_fraud_idx, unlabeled_fraud_count = create_supervised_labels(y_train, ratio)
    
    total_fraud = sum(y_train)
    labeled_fraud = sum(y_labels)
    
    print(f"Identified frauds (labeled as 1): {labeled_fraud}")
    print(f"Hidden frauds in 'regular' information: {unlabeled_fraud_count}")
    print(f"Whole samples handled as regular: {len(y_train) - labeled_fraud}")
    print(f"Fraud contamination in 'regular' set: {unlabeled_fraud_count/(len(y_train) - labeled_fraud)*100:.3f}%")
    
    # Prepare XGBOD fashions
    xgbod = XGBOD(estimator_list=[PCA(), CBLOF(), IForest(), HBOS()],
                  random_state=42, 
                  n_estimators=200, learning_rate=0.1, 
                  eval_metric='aucpr')
    
    xgbod.match(X_train, y_labels)
    end result = evaluate_model(xgbod, X_test, y_test, f"XGBOD_ratio_{ratio:.3f}")
    print(f"xgbod - AUC-PR: {end result['auc_pr']:.4f}")

The obtained outcomes are proven within the determine under, along with the efficiency of one of the best unsupervised detector (HBOS) because the reference.

We are able to see that with just one% labeled anomalies, the XGBOD methodology already beats one of the best unsupervised detector, attaining an AUCPR rating of 0.4. With extra labeled anomalies turning into accessible for coaching, XGBOD’s efficiency continues to enhance.

4.5 Supervised Studying

Lastly, we think about the situation the place we straight practice a binary classifier on the dataset with the labeled anomalies.

for ratio in supervision_ratios:
    
    # Create supervised labels
    y_label, labeled_fraud_idx, unlabeled_fraud_count = create_supervised_labels(y_train, ratio)

    clf = XGBClassifier(n_estimators=200, random_state=42, 
                        learning_rate=0.1, eval_metric='aucpr')
    clf.match(X_train, y_label)
    
    y_pred_proba = clf.predict_proba(X_test)[:, 1]
    auc_pr = average_precision_score(y_test, y_pred_proba)
    print(f"XGBoost - AUC-PR: {auc_pr:.4f}")

The outcomes are proven within the determine under, along with the XGBOD’s efficiency obtained from the earlier part:

Normally, we see that with solely restricted labeled information, the usual supervised classifier (XGBoost on this case) struggles to differentiate between regular and anomalous samples successfully. That is significantly evident when the supervision ratio is extraordinarily low (i.e., 1%). Whereas XGBoost’s efficiency improves as extra labeled examples turn out to be accessible, we see that it stays constantly inferior to the XGBOD method throughout the examined vary of supervision ratios.

5. Conclusion

On this put up, we mentioned three sensible methods to leverage the few labeled anomalies to spice up the efficiency of your anomaly detector:

• Threshold tuning: Use labeled anomalies to show threshold setting from guesswork right into a data-driven optimization drawback.
• Mannequin choice: Objectively examine completely different algorithms and hyperparameter settings to search out what really works nicely to your particular issues.
• Supervised ensembling: Prepare a meta-model to systematically extract the anomaly signatures revealed by a number of unsupervised detectors.

Moreover, we went by a concrete case examine on fraud detection and confirmed how the supervised ensembling methodology (XGBOD) dramatically outperformed each purely unsupervised fashions and normal supervised classifiers, particularly when labeled information was scarce.

The important thing takeaway: a number of labels go a good distance in anomaly detection. Time to place these labels to work.