Neuro-Symbolic Fraud Detection: Catching Concept Drift Before F1 Drops (Label-Free)

Can the foundations act as a canary? Can neuro-symbolic idea drift monitoring work at inference time, with out labels?

Three Methods Fraud Can Change

Idea drift fraud detection is more durable than it sounds as a result of solely one of many three widespread drift sorts truly adjustments what the mannequin’s discovered associations imply. The experiment simulates three forms of drift on the Kaggle Credit score Card Fraud dataset (284,807 transactions, 0.17% fraud price) throughout 8 progressive home windows every [9].

Covariate drift. The enter characteristic distributions shift. V14, V4, and V12 transfer by as much as +3.0σ progressively. Fraud patterns keep the identical. The world simply appears just a little completely different.

Prior drift. The fraud price will increase from 0.17% towards 2.0%. Options are unchanged. Fraud turns into extra widespread.

Idea drift. The signal of V14 is progressively flipped for fraud circumstances throughout 8 home windows. By the top, the transactions the mannequin discovered to flag as fraud now appear to be official ones. The rule IF V14 < −1.5σ → Fraud is now pointing within the fallacious path.

That third one is the one that ought to fear you in manufacturing. With covariate and prior drift, there are exterior indicators. Enter distributions shift, or fraud charges visibly change. You’ll be able to monitor these independently. Idea drift leaves no such footprint. The one factor that adjustments is what the mannequin’s discovered associations imply. You’ll not know till F1 begins falling.

Until one thing sees it first.

The Drawback With the First Three Metrics

The mannequin from How a Neural Network Learned Its Own Fraud Rules: A Neuro-Symbolic AI Experiment produced three label-free monitoring indicators as a by-product of the symbolic layer. The thought: if the foundations are studying fraud patterns, adjustments in how these guidelines hearth ought to reveal when fraud patterns are shifting.

I anticipated the primary one to be the early warning. It was not.

The issue is particular to how this mannequin trains. All 5 seeds converged between epochs 3 and 10 (Val PR-AUC: 0.7717, 0.6915, 0.6799, 0.7899, 0.7951), when temperature τ continues to be between 3.5 and 4.0. At that temperature, rule activations are delicate. Each enter produces a near-identical activation rating no matter its precise options. In plain phrases: the foundations had been firing nearly the identical manner on each transaction, clear or drifted. A similarity metric on near-constant vectors returns 1.000 nearly on a regular basis. The primary sign solely fired in 2 of 5 seeds for idea drift, and in each circumstances it was the identical window as F1 or later.

Why excessive temperature makes monitoring more durable

The LearnableDiscretizer makes use of a sigmoid gated by temperature τ: σ((x − θ) / τ). At τ = 5.0 (epoch 0), that sigmoid is almost flat — each characteristic worth produces an activation near 0.5 no matter the place it sits relative to the discovered threshold. As τ anneals towards 0.1, the sigmoid sharpens right into a near-binary step. Early stopping fires at τ ≈ 3.5–4.0 — earlier than the foundations have absolutely crystallised. The end result: activation vectors are near-constant throughout all inputs, so any similarity metric between them stays close to 1.000 even when fraud patterns are genuinely shifting.

The second sign had the other downside. Absolutely the change in any characteristic’s contribution is tiny (values within the 0.001–0.005 vary) as a result of the rule weights themselves are small at an early-stopped checkpoint. In plain phrases: the sign was actual however invisible on the scale we had been measuring it. A hard and fast absolute threshold of 0.02 by no means fires.

Here’s what these three unique indicators are:

• RWSS (Rule Weight Stability Rating): cosine similarity between the baseline imply rule activation vector and the present one. In easy phrases: are the foundations nonetheless firing the identical manner they did on clear information?
• FIDI (Function Significance Drift Index): how a lot every characteristic’s contribution to rule activations has modified from the baseline. In easy phrases: has any particular characteristic grow to be kind of vital to the foundations?
• RFR (Rule Firing Fee): what fraction of transactions hearth every rule.

That analysis led to the correct query. As a substitute of asking “has FIDI modified by greater than X?”, the correct query is “has FIDI modified by greater than X customary deviations from its personal historical past?”

That query has a unique reply. And the reply is V14.

The Metrics: Constructing a Label-Free Drift Detection System

Three new metrics joined the unique three.

RWSS Velocity measures the per-window price of change: RWSS[w] − RWSS[w−1]. A sudden drop of greater than 0.03 per window fires an alert even earlier than absolutely the worth crosses the brink. If RWSS is falling at −0.072 in a single step, that could be a sign no matter the place it began.

FIDI Z-Rating is the one that really labored. Reasonably than a model new sign, it’s a easy extension of FIDI utilizing Z-score normalisation in opposition to the characteristic’s personal window historical past. As a substitute of asking whether or not absolutely the change crosses a hard and fast threshold, it asks whether or not the change is anomalous relative to what that characteristic has been doing. In contrast to conventional drift detection strategies that depend on enter distributions or output labels, this method operates purely on the symbolic layer, which implies it really works at inference time, with no floor reality required. It builds on differentiable rule-learning work together with ∂ILP [3], FINRule [4], RIFF [5], and Neuro-Symbolic Rule Lists [6], extending these representations with Z-score normalisation fairly than mounted thresholds. V14’s contribution to rule activations throughout the clear baseline home windows is small and flat. Close to zero, steady, predictable. When idea drift begins at window 3, it shifts. Not by a lot in absolute phrases. However by 9.53 customary deviations relative to the historical past it constructed throughout steady home windows. That is a gigantic relative anomaly, and no threshold calibration is required to catch it.

PSI on Rule Activations was designed to catch distributional shift within the symbolic layer earlier than the MLP’s compensation masks it on the output degree. It didn’t work right here. The delicate activations from early-stopped coaching (τ ≈ 3.5–4.0 on the saved checkpoint) cluster close to 0.5, producing near-uniform distributions that PSI can not distinguish. PSI_rules = 0.0049 all through the complete experiment. PSI_rules by no means fired. It’s within the codebase for when fashions with absolutely crystallised guidelines (τ < 0.5) can be found. On this experiment it contributed nothing.

The supposed detection order, from earliest to newest:

RWSS Velocity → FIDI Z-Rating → PSI(guidelines) → RWSS absolute → F1 (label-based)

Here’s what truly occurred.

Outcomes: What Every Metric Did

Idea Drift

FIDI Z-Rating fires in 5 of 5 seeds, all the time at window 3. F1 fires at W3 in three seeds and W4 in two. The imply FIDIZ detection lag is +0.40 home windows, which means it leads F1 on common. In seeds 7 and 2024 it fires one full window earlier than F1 drops. Within the remaining three seeds it fires concurrently. It by no means fires after F1 for idea drift. Not as soon as.

Throughout all drift sorts, FIDI Z-Rating is the one metric that detected idea drift in each seed and by no means lagged behind F1. For label-free drift detection fraud programs, that’s the headline end result.

RWSS fires in 2 of 5 seeds and in each circumstances concurrently with or after F1. Velocity matches RWSS precisely, similar window, each time. PSI on rule activations by no means fires in any respect.

Idea Drift vs Covariate Drift: Why Symbolic Monitoring Has Blind Spots

Covariate drift is the place the symbolic layer goes utterly silent.

Each symbolic metric: 0 of 5 seeds. Not one sign. Not one window. F1 finally fires in 4 of 5 seeds at W6 or W7, slowly and late, and the symbolic layer had nothing to do with it. This isn’t a spot that higher tuning will shut. It’s a elementary property of what the symbolic layer measures.

The reason being mechanical. When V14, V4, and V12 shift by +3.0σ, the shift is uniform throughout all samples. The learnable discretizer computes thresholds relative to the information. Every pattern nonetheless lands in roughly the identical threshold bin relative to its neighbours. Guidelines hearth on roughly the identical proportion of transactions. Nothing within the activation sample adjustments. Cosine similarity of imply activations stays at 1.0.

In easy phrases: if each transaction shifts by the identical quantity, the foundations nonetheless see the identical relative image. Transaction A was above the brink earlier than. It’s nonetheless above the brink after. The fraud-vs-legitimate ordering is preserved. RWSS measures that ordering, not absolutely the values. Consider it as a tide that lifts all boats equally. The boats keep in the identical order. RWSS solely measures the order.

If covariate drift is a priority in your deployment, you want a separate input-space monitor: PSI on uncooked options, a KS take a look at on V14, or an information high quality test. The symbolic layer can not aid you there. Symbolic layer drift monitoring has one blind spot, and covariate shift is it.

Prior Drift

FIDIZ fires in 5 of 5 seeds, all the time at W3. However prior drift causes F1 to drop at W0 (seed 123) or W2 (seed 2024) within the two seeds the place F1 fires in any respect. FIDIZ detection lag for prior drift: −2.00 home windows. It fires two home windows after F1.

This isn’t a calibration downside. FIDIZ wants a minimal of three clear home windows to construct a historical past earlier than its Z-score is significant. Prior drift that causes a right away fraud price soar is already seen in F1 earlier than FIDIZ may even begin computing. A rolling fraud price counter will all the time be quicker right here.

The Alert Demo: Window 3

Right here is the second the entire system was constructed for.

DriftAlertSystem is constructed as soon as from the validation set instantly after coaching. It shops the baseline. Then .test() is named on every new window. No labels. No retraining. That is inference-only drift detection: the system reads the symbolic layer and nothing else.

Seed 42, idea drift, 8 home windows:

Window 0: severity=none      RWSS=0.999  fired=False
Window 1: severity=none      RWSS=0.999  fired=False
Window 2: severity=none      RWSS=0.999  fired=False
Window 3: severity=warning   RWSS=1.000  fired=True   ← FIDI Z fires right here
Window 4: severity=vital  RWSS=0.928  fired=True   ← RWSS absolute confirms
Window 5: severity=warning   RWSS=0.928  fired=True
Window 6: severity=warning   RWSS=0.928  fired=True
Window 7: severity=warning   RWSS=0.928  fired=True

At window 3, RWSS is precisely 1.000. The activation sample is completely equivalent to baseline. Output possibilities haven’t modified. Nothing in the usual monitoring stack has moved.

And the alert fires at WARNING severity.

The reason being V14. Its Z-score is −9.53. Meaning V14’s contribution to rule activations has shifted to almost 10 customary deviations under the baseline it established throughout clear home windows. The mannequin’s output doesn’t know but. The MLP is compensating. However the rule path can not compensate. It was educated to specific a hard and fast symbolic relationship. It’s screaming.

One window later, the MLP stops holding. RWSS drops to 0.928. Velocity falls 0.072 in a single step. Severity escalates to CRITICAL.

═══════════════════════════════════════════════════════
  DRIFT ALERT  |  severity: CRITICAL
  Earliest sign: VELOCITY
═══════════════════════════════════════════════════════
  ── Early-Warning Layer ─────────────────────────────
  RWSS Velocity : -0.0720  [threshold -0.03]  ⚠ FIRED
  FIDI Z-Rating  : ⚠ FIRED
       V14  Z = -9.53
  PSI (guidelines)   : 0.0049  [moderate≥0.10]  steady
  ── Confirmed Layer ─────────────────────────────────
  RWSS absolute : 0.9276  [threshold 0.97]  ⚠ FIRED
  Guidelines gone silent: 0  OK
  Imply RFR change  : -0.001
  Advisable motion:
    → Retrain instantly. Don't deploy.
═══════════════════════════════════════════════════════

The report names VELOCITY because the earliest layer. That may be a precedence order within the inner logic. In precise window timing, FIDI Z-Rating fired one window earlier at W3. The W3 WARNING is the sooner human-facing alert. The one that provides you time to behave earlier than the CRITICAL fires.

Why FIDI Z-Rating Sees It Earlier than F1 Does

The mannequin has two paths working in parallel from the identical enter.

The MLP path carries 88.6% of the ultimate output (imply α = 0.886 throughout seeds; α is the discovered mix weight; 0.886 means the neural community does 88.6% of the prediction work and the symbolic guidelines do the remaining 11.4%). When idea drift progressively reverses V14’s relationship to fraud labels, the MLP, educated on 284,000 transactions, partially absorbs that change. Its inner representations shift. Output possibilities keep roughly steady for not less than one window. That is the MLP compensating.

The rule path carries 11.4%. It was educated to specific the MLP’s information in symbolic type: V14 under a threshold means fraud [2]. That relationship is mounted and specific. When V14 flips signal for fraud circumstances, the rule’s V14 contribution doesn’t modify. It merely stops working. The bit activations for V14 change path. The rule begins firing on the fallacious transactions.

The neural community adapts. The symbolic layer doesn’t. And that’s precisely why the symbolic layer detects the drift first.

That asymmetry is what FIDI Z-Rating exploits.

Absolutely the change in V14’s contribution is tiny (values within the 0.001 to 0.005 vary) as a result of rule weights are small at an early-stopped checkpoint. A hard and fast absolute threshold by no means catches it.

However V14’s historical past by way of the clear home windows is simply as flat. When idea drift strikes it at window 3, the Z-score is −9.53. Identical sample: near-zero absolute change, excessive relative shift.

The symbolic layer is much less compensating than the MLP, so it exhibits the drift first. FIDI Z-Rating makes the sign seen by evaluating every characteristic to not a hard and fast threshold, however to its personal historical past.

However this solely holds for one of many three drift sorts. The opposite two are a unique story fully.

What This System Can not Do

A system that claims early warning invitations overstatement. Here’s what the information truly says. That is label-free anomaly detection fraud monitoring, which implies the constraints are structural, not tunable.

Covariate drift is a whole blind spot. 0 of 5 seeds. The mechanism is defined within the Outcomes part above. Use PSI on uncooked options or a KS take a look at on V14 as an alternative.

FIDIZ fires late on prior drift by design. When the fraud price jumps, F1 reacts at W0 or W2. FIDIZ structurally can not hearth earlier than W3. It wants historical past that doesn’t but exist. A rolling fraud price monitor responds quicker.

PSI on rule activations produced nothing. PSI_rules = 0.0049 all through each window of each seed. Comfortable activations from early-stopped coaching cluster close to 0.5, and PSI on near-uniform distributions is insensitive regardless of what’s truly taking place. This metric is within the codebase and may match with absolutely annealed fashions (τ < 0.5). On this experiment it was silent.

5 seeds is proof, not proof. FIDIZ fires at W3 for idea drift throughout all 5 seeds. That’s constant and inspiring. It isn’t the identical as dependable in manufacturing throughout datasets, fraud sorts, and drift severities you haven’t examined. 5 seeds is a place to begin, not a conclusion. Extra seeds, extra drift configurations, and real-world validation are wanted earlier than sturdy deployment claims.

Outcomes Abstract

The sample is clearest when said plainly first. Consider this as an early warning idea drift system with three distinct modes relying on what’s altering. Covariate drift: the symbolic layer noticed nothing, F1 caught it slowly. Prior drift: the symbolic layer fired after F1, not earlier than. Idea drift: FIDI Z-Rating fired in each single seed, all the time at or earlier than F1, averaging +0.40 home windows of lead time.

Lag = home windows earlier than F1 alert. Constructive = FIDIZ fires first. Unfavorable = F1 fires first.

Constructing It

The system is designed for use in manufacturing, not simply in a pocket book.

# As soon as, instantly after coaching
X_val_t      = torch.FloatTensor(X_val)
alert_system = DriftAlertSystem.from_trained_model(mannequin, X_val_t, feature_names)
alert_system.save("outcomes/drift_alert_baseline_seed42.pkl")

# Each scoring run — weekly, each day, per-batch
alert_system = DriftAlertSystem.load("outcomes/drift_alert_baseline_seed42.pkl")
alert = alert_system.test(mannequin, X_this_week)

if alert.fired:
    print(alert.report())

No labels. No retraining. No infrastructure past saving a pickle file subsequent to the mannequin checkpoint. The .test() name computes RWSS velocity, FIDI Z-Rating, PSI on activations, and RWSS absolute in that order, utilizing PyTorch [7] and scikit-learn [8]. Severity escalates from none to warning to vital based mostly on what number of hearth and the way far RWSS has dropped.

The three early-warning computations are every just a few strains.

RWSS Velocity: price of change per window.

def compute_rwss_velocity(rwss_history: Listing[float]) -> float:
    if len(rwss_history) < 2:
        return 0.0
    return float(rwss_history[-1] - rwss_history[-2])

# Alert fires when drop > 0.03 per window
vel_fired = rwss_velocity < -0.03

FIDI Z-Rating: normalise characteristic contribution anomaly in opposition to historical past.

def compute_fidi_zscore(fidi_history, current_fidi, min_history=3):
    if len(fidi_history) < min_history:
        return {ok: 0.0 for ok in current_fidi}
    z_scores = {}
    for feat_idx, current_val in current_fidi.gadgets():
        history_vals = [h.get(feat_idx, 0.0) for h in fidi_history]
        mean_h = np.imply(history_vals)
        std_h  = np.std(history_vals)
        z_scores[feat_idx] = (current_val - mean_h) / std_h if std_h > 1e-8 else 0.0
    return z_scores

# Alert fires when any characteristic Z > 2.5
fidi_z_fired = any(abs(z) > 2.5 for z in z_scores.values())

PSI on Rule Activations: distributional shift within the symbolic layer (included for completeness).

def compute_psi_rules(baseline_acts, current_acts, n_bins=10):
    bins = np.linspace(0, 1, n_bins + 1)
    psi_per_rule = []
    for r in vary(baseline_acts.form[1]):
        b = np.histogram(baseline_acts[:, r], bins=bins)[0] + 1e-6
        c = np.histogram(current_acts[:,  r], bins=bins)[0] + 1e-6
        b /= b.sum(); c /= c.sum()
        psi_per_rule.append(float(np.sum((c - b) * np.log(c / b))))
    return np.imply(psi_per_rule)

V14: Three Articles, One Function

That is the half I didn’t plan. However V14 idea drift behaviour seems to be the thread that ties all three articles collectively.

Guiding Neural Networks with Domain Rules: I wrote guidelines about massive transaction quantities and anomalous PCA norms. Affordable intuitions. Nothing to do with V14.

How a Neural Network Learned Its Own Fraud Rules: The mannequin discovered V14 anyway. Given 30 anonymised options and no steering, the gradient landed on the one characteristic with the best absolute correlation to fraud. Twice, throughout two unbiased seeds.

This Article, I intentionally made V14 break. I flipped its signal for fraud circumstances, progressively, throughout 8 home windows. And FIDI Z-Rating registered the collapse at −9.53 customary deviations whereas RWSS was nonetheless 1.000 and F1 had not moved.

The identical characteristic, three completely different roles: ignored, found, then monitored as the very first thing to fail. That coherence was not engineered. It’s what reproducible multi-seed analysis on a constant dataset retains producing.

What to Do With This

Use FIDI Z-Rating for idea drift detection with out labels. It fires in 5 of 5 seeds, requires solely 3 home windows of historical past, by no means fires after F1, and desires no labels. Preserve the Z-score threshold at 2.5 and minimal historical past at 3 home windows.

Add a separate input-space monitor for covariate drift. PSI on uncooked options or a KS take a look at on vital options like V14. The symbolic layer is blind to distributional shifts that protect relative activation order.

Use a rolling fraud price counter for prior drift. FIDIZ structurally can not hearth earlier than W3. A label-based price counter fires at W0.

Construct the alert baseline instantly after coaching. Not after drift is suspected. Do it after coaching. If you happen to wait, you might have already misplaced your clear reference level. Put it aside alongside the checkpoint file.

One window of early warning is actual. Whether or not it’s one week or sooner or later will depend on your scoring cadence. For many manufacturing fraud groups, the distinction between a scheduled retrain and an emergency one is measured in precisely these items.

Three Issues That Will Catch You Utilizing This Idea Drift Early Warning System

The three-window blind interval. FIDIZ has no historical past to work with for the primary 3 home windows after deployment. You might be monitoring with RWSS and RFR solely throughout that point. Plan for it explicitly.

Comfortable activations will silence PSI_rules. In case your greatest checkpoint arrives when τ ≥ 1.0 (which occurs each time early stopping fires earlier than coaching is full), rule activations cluster close to 0.5 and PSI_rules returns noise. Examine τ at your saved checkpoint. On this experiment τ was nonetheless 3.5–4.0 at convergence. That’s the reason PSI_rules was silent all through.

Retrain means re-audit. This technique is a fraud mannequin retraining set off, not a retrain substitute. After retraining, the foundations change. V14 might now not dominate, or new options might have entered. The compliance sign-off from the earlier mannequin doesn’t carry ahead. Construct the audit into the retrain course of, not as a step after, however because the step that closes the loop.

Closing

Three articles. One characteristic stored showing.

Guiding Neural Networks with Domain Rules: I ignored it. How a Neural Network Learned Its Own Fraud Rules: The gradient discovered it. Article 3: When it broke, the symbolic layer observed earlier than the output layer did.

The experiment has a particular, trustworthy scope: FIDI Z-Rating detects idea drift in 5 of 5 seeds, typically one window earlier than F1, by no means after it, fully with out labels. For covariate drift it’s blind. For prior drift it’s late. These aren’t caveats added on the finish to melt the declare. They’re findings that inform you precisely the place to make use of this and the place to not.

A neuro-symbolic mannequin offers you two channels. The MLP is best at prediction. The symbolic layer is best at figuring out when prediction is about to go fallacious. They don’t seem to be redundant. They’re watching completely different elements of the identical downside.

The MLP compensates. The symbolic layer can not. That’s its weak spot. On this experiment, it turned out to even be its earliest warning.

Sequence