AI and ML basics

PyTorch · sklearn · TensorFlow | Classification · Anomaly · Patterns

1. Why Walmart Asks

JD "apply ML for classification/anomaly/pattern" = three realities:

You'll hear — each question is a disguised test of one section of this chapter: - "How would you use ML to detect phishing or malware at Walmart scale?" — tests end-to-end pipeline thinking (see §2.9): features, ensemble, threshold, feedback loop, not a model name-drop. - "Your model is 99% accurate but SOC hates it — why?" — tests base-rate fluency (see §2.4): do the flood math out loud and talk precision, never accuracy. - "Supervised vs unsupervised for insider threat? Which and why?" — tests trade-off judgment (see §2.1): insider threat is low-label and high-cost-of-miss, so unsupervised UEBA first, supervised confirmation second. - "How would you red-team our fraud classifier?" — tests adversarial ML (see Q16): poisoning, evasion, stealing, prompt injection, mapped to MITRE ATLAS. - "Your UEBA baseline got poisoned — how do you detect and fix drift?" — tests §2.6: drift monitoring, anchored baselines, label-gated retraining, canary rollback.

Bar: explain + trade-offs + SOC workflow + code.

2. Concepts + Examples

2.1 Paradigms — Supervised vs Unsupervised vs RL

Intuition first: supervised learning answers "does this look like things previously labeled bad?", unsupervised answers "is this weird for this entity?", and RL answers "which action sequence wins a long game?". Security detection work is mostly the first two questions; RL shows up in attack/defense simulation and SOAR tuning rather than in daily alerting.

Memorize: supervised = "known-bad?", unsupervised = "weird for this entity?", RL = "what action sequence wins?"

Which to use:

2.2 Classification in Security — Malware / Phishing

Classification = assign a label. Formally, a supervised model learns a function from features to a finite label set — for security usually binary (malware/clean, phish/legit) or a family label (Emotet / TrickBot / clean). The model outputs a per-class score, and a threshold turns that score into an alert. Everything hard about security classification lives in the labels and the threshold, not in the algorithm choice.

Malware example features (static + dynamic):

Phishing example features: URL length/entropy, punycode, subdomain depth, domain age, SPF/DKIM/DMARC fail, HTML form-to-external, language urgency score, attachment macro presence.

Labels drawback — the four reasons security labels are hard: - Costly and stale. Every label needs an analyst or a sandbox verdict, and by the time the corpus is labeled the attacker has mutated — the model learns yesterday's campaign and misses the current one. - 1-in-10k imbalance. Real malicious ratios run 1-in-10,000 or worse, so an "always legit" classifier is 99.99% accurate and useless; metrics and training must compensate explicitly. - Adversarial flips. Attackers poison labels directly — whitelist-then-weaponize: obtain a clean verdict on a benign file or domain, then ship malware through the same signer or reputation entry. - Concept drift. GenAI phish breaks old text-template features; P(y|X) moves under you and yesterday's labels quietly stop describing today's traffic.

Fix, three complementary moves: active learning (analyst labels only the alerts the model is least certain about, so a small labeling budget lands where it moves the decision boundary), weak supervision (existing detection rules generate noisy labels at scale — a rule hits, so the label is probably phish — with a label model reconciling rule conflicts), and layered detection (run the supervised classifier behind an unsupervised anomaly layer, so novel attacks that slip past the classifier still surface as anomalies).

2.3 Anomaly — IsolationForest / Autoencoder / Clustering / UEBA

Core idea: model normal per entity, emit a score, not a verdict. An anomaly detector never says "malware"; it says "this is far from how this user/host/peer-group normally behaves at this hour" — the number is a ranking signal, and a downstream threshold or an analyst turns it into an alert. That is exactly why these methods need no labels (only a clean-ish window of normal behavior), and why "anomalous ≠ malicious" is the permanent tax you pay for skipping labels.

UEBA: risk = w1·auth + w2·data + w3·net + w4·peer — the fusion fires, not one feature. Each term is itself an aggregate over the day (e.g. auth = failed/successful ratio plus off-hours logins), and the weights are set so no single noisy source can page an analyst alone. Three weak signals crossing the threshold together is a much stronger alert than one dramatic feature.

"Anomalous ≠ malicious": 2am admin patching is anomalous-but-benign; slow exfil is malicious-but-normal. Scores need context + analyst review before alerting. Practical consequences: enrich every score with peer behavior and asset value before thresholding (a domain controller outranks a test box), keep a human in the loop on anything that auto-blocks, and feed the analyst's TP/FP verdict back so the model learns which anomalies this environment generates naturally.

2.4 Metrics — Precision/Recall/F1/ROC, Why Accuracy Lies

Definitions for imbalanced SOC queues:

Worked example: a 10-sample confusion matrix

Ten scored emails, three of them actually phish (samples 1–3), seven legit (4–10). The model alerts on samples 1, 2, 4, and 7. Reading the outcomes: TP = 2 (samples 1, 2 — real phish caught), FN = 1 (sample 3 — missed phish), FP = 2 (samples 4, 7 — false alarms), TN = 5 (legit mail correctly ignored). Now the metrics: Precision = TP/(TP+FP) = 2/4 = 0.50 — half of every alert you send is noise. Recall = TP/(TP+FN) = 2/3 ≈ 0.67 — you catch two of every three attacks. F1 = 2PR/(P+R) = 2·0.50·0.67/(0.50+0.67) = 4/7 ≈ 0.57, the same as computing 2TP/(2TP+FP+FN) = 4/7 directly. Accuracy = 7/10 = 70% — and notice how much better accuracy looks than the model really is: it missed a third of the attacks and cried wolf half the time. To lift precision you raise the threshold (fewer alerts, more misses); to lift recall you lower it. There is no free lunch, only a chosen operating point.

ROC-AUC vs PR-AUC on imbalanced phishing data

Take a realistic base rate: 1,000,000 events with 100 real phish (0.01% positive). At a permissive threshold the model runs at TPR 0.90 and FPR 0.001 — on a ROC plot that point hugs the top-left corner and the ROC-AUC may read 0.99, which looks superb. Translate it into alert counts: TP = 90, FP = 0.001 × 999,900 ≈ 1,000, precision = 90/1,090 ≈ 8.3%. On a precision-recall plot the same operating point sits near the floor, and PR-AUC shows the model for what it is: a source of ~1,000 junk alerts per 90 real catches. The reason is arithmetic — the ROC x-axis divides by ~1M negatives, so FPR stays tiny no matter how much noise you generate, while precision divides by everything you actually alert on, which is the number that sets SOC workload. Report PR-AUC and precision@recall for detection work on rare events; keep ROC-AUC for roughly balanced problems such as malware-family classification.

99.9% flood math — memorize:

Walmart processes 1,000,000 logins/day, 100 malicious. Scenario 1 — 0.1% FPR ("99.9% accurate"): - TP ≈ 100 — assume perfect recall for the illustration, so every real attack is caught. - FP = 0.001 × 999,900 ≈ 1,000 false alarms — the 0.1% FPR applies to the 999,900 benign logins, which is where the noise volume comes from. - Precision = 100 / 1,100 ≈ 9% → 10 of 11 alerts are noise.

Scenario 2 — same math at a looser threshold, 1% FPR ("99% accurate"): FP ≈ 10,000, precision ≈ 1%. SOC will disable your model in a week.

Takeaways: report precision@recall / PR-AUC, never accuracy; the threshold is a business call tuned to SOC capacity, not a modeling default; narrowing scope (finance + external + new device) is often the cheapest precision win you have. In interview, convert any metric claim into alert counts — "that operating point is 1,000 alerts/day" beats "F1 = 0.8" — because the SOC manager's unit is reviews per shift, and the threshold is where model quality meets staffing.

2.5 Overfitting / Underfitting + Bias-Variance

Reading overfitting off the loss curves

Watch train and validation loss per epoch on a phishing classifier. Epochs 1–8: train 0.42 → 0.18, val 0.45 → 0.20 — both curves fall together with a stable gap; the model is learning signal that generalizes. Epochs 10–15: train 0.18 → 0.05 while val climbs 0.20 → 0.31 — the curves diverge and the gap widens. That divergence is overfitting in one picture: past epoch 8 the model spends its capacity memorizing training noise (the specific attacker IPs and hashes inside your window) and every further epoch costs generalization. The fix is early stopping at the validation minimum — early_stopping_rounds in XGBoost, EarlyStopping(restore_best_weights=True) in Keras, patience plus best-checkpoint in PyTorch — so training effectively rolls back to epoch 8's weights. Two caveats: validation PR-AUC on a thin positive class is noisy across epochs (smooth it or lengthen the validation window before concluding anything), and if train and validation both plateau high, that is underfitting — add features and depth, not patience.

Bias-variance: simple = consistent errors (bias), complex = wild errors on new data (variance); regularized XGBoost/RF hits the sweet spot on tabular logs. Security-flavored: a linear model with high bias always misses graph-shaped lateral movement no matter how much data you add, while a deep net with high variance keys onto one campaign's IP ranges and dies on the next. Regularized gradient boosting has enough capacity for interactions like failed_logins × off-hours, and its ensembling plus shrinkage keep it stable week to week.

Top mistake is leakage: features built from information that will not exist at scoring time — analyst_verdict/blocked, quarantine status — inflate val to near-perfect and collapse in prod the day the model meets an alert nobody has triaged yet. Leakage is silent: the metrics look great right up until deployment, so audit every feature with "would I know this at scoring time?"

2.6 Drift + Baseline Poisoning

Defenses: 1. Monitor drift explicitly — PSI, KS-test, embedding distance on features + predicted-positive rate; alert when drift > threshold. 2. Anchor baselines — fixed clean window + peer baselines + immutable audit log; don't blindly retrain on attacker-influenced data. Require analyst-labeled feedback before absorbing new "normal." 3. Canary + rollback — shadow-deploy new model, compare precision/recall on live stream, auto-rollback; version datasets/models.

Slow-ramp exfil is the textbook case: moving averages absorb it. Picture a defender who computes bytes_out against a rolling 7-day mean and alerts at z > 3. The attacker starts at 1.0 MB/day and adds 0.2 MB/day every day. Each new day sits only 0.8 MB above the trailing 7-day mean (the ramp runs 0.8 ahead of its own week-old echo), while a window of values spaced 0.2 MB apart has a spread of about 0.4 MB — so z is pinned at ≈ 2.0, every day, indefinitely. Daily exfil grows without bound and no alert ever fires. Fix = long lookback + absolute caps + peer comparison.

2.7 Log Feature Engineering — Windows, Profiles, Aggregations

Raw logs are useless — engineer entity-time features:

Example row for alice, 14:00:

failed_1h=4, success_after_fail=1, impossible_travel_km=8000,
bytes_out_24h_z=3.2, new_host_first_seen=1, peer_deviation=2.1, domain_age_min=2
→ risk_score=0.93

Three engineering facts decide whether any of these features survive production: entity resolution (the same human appears as alice, alice@corp, and a Windows SID — one entity, three rows, and three split baselines unless you unify them first), delayed labels (an alert's true disposition arrives days or weeks later, so training pairs assemble on a delay and the model in production always trains on the past), and train-serve parity (the identical window/aggregation code must run in batch training and stream scoring, or the model will be scored on features it never saw in training).

2.8 PyTorch / Scikit-learn Basics

Tensors: n-dim arrays with GPU + autograd — autograd records every operation on a tensor so gradients are computed for you instead of derived by hand. Loop: forward → loss → zero_grad → backward → step per batch, val + early-stop per epoch.

One gradient step, traced by hand. Take the simplest model, ŷ = w·x, one sample x = 2, y = 10, MSE loss L = (w·x − y)². Start at w = 1. Forward: ŷ = 2, loss (2 − 10)² = 64. Backward: dL/dw = 2(w·x − y)·x = 2(2 − 10)·2 = −32. Step with learning rate η = 0.01: w ← w − η·dL/dw = 1 + 0.32 = 1.32. Recheck: ŷ = 2.64, loss (2.64 − 10)² ≈ 54.2 — one step cut the loss from 64 to 54.2. That subtraction w − η·dL/dw is the whole of gradient descent; Adam just adapts the per-parameter step size from gradient history. zero_grad() before backward() matters because PyTorch accumulates gradients across batches — skip it and this −32 adds to the previous batch's gradient, corrupting every step size.

Splits: 60/20/20, stratified, time-based (train Jan–Mar, test Apr; random shuffle leaks future TTPs). Never tune threshold on test. The three slices have three jobs: train fits the weights, validation chooses hyperparameters, model family, calibration and the alert threshold, test is the untouched report you quote exactly once. On logs the slices must be real time windows — train Jan–Mar, validate Apr, test May — because random shuffling lets the model see April's TTPs while training on March, which is future leakage that inflates every metric. Keep the class ratio roughly constant across slices (stratify=y, applied to time-ordered data with care) so a thin positive class does not vanish from one slice. The 60/20/20 ratio is convention, not law; what matters is that the test window never influenced a single decision.

2.9 End-to-End Pipeline — Ingest → Features → Models → Alerts → Feedback

[Ingest] Splunk/Elastic/S3/Kafka: auth, DNS, proxy, EDR, email, IAM
   ↓ (parse, normalize, entity-resolve, dedupe)
[Feature Store] batch (Spark) + streaming (Flink): windows, profiles, graph, reputation
   ↓
[Models — ensemble, not one hero model]
   ├─ Supervised: XGBoost/RF (known malware/phish), fine-tuned BERT (email NLP)
   ├─ Unsupervised: IsolationForest + Autoencoder + HDBSCAN (novel / insider)
   └─ Sequential: LSTM / Transformer on event sequences (lateral movement, DGA)
   ↓ (calibrate, threshold per segment, fuse into risk score)
[Scored Alerts + SHAP] → SOAR/SIEM: score, top-3 reasons, raw evidence, suggested playbook
   ↓
[Analyst Feedback] TP/FP/Benign → active-learning queue → relabel → retrain + drift check → redeploy

SHAP: attach top contributors: bytes_out_z=+0.4, travel=+0.3, new_domain=+0.2 so analysts judge in 30s; required for audit. SHAP decomposes a model's score into per-feature contributions in the score's own units, which turns an opaque risk_score=0.93 into a story an analyst can confirm or reject against raw evidence — and supplies the "why was I flagged" answer that any model-risk or compliance review will demand.

Calibration: raw score ≠ probability — scale_pos_weight/SMOTE rebalancing distorts probabilities (XGBoost tuning guide: skip rebalancing if you need true probs) because the model trains on an artificial class prior and its outputs match that prior, not reality. If scores drive thresholds/risk ranks, fit Platt scaling (logistic regression on the score) or isotonic regression on a held-out time slice; then check a reliability curve — of the events scored ~0.8, roughly 80% should be real — plus the Brier score. A calibrated 0.8 meaning 80% is what lets you set thresholds in business language.

Retrain loop: dispositions flow back; weekly retrain on confirmed labels only; hold out the latest week as validation so the check is honest about the future; canary-deploy the new model beside the old one, compare precision/recall on the live stream, then promote or roll back.

2.10 Production Tools

Walmart answer: "Splunk/Sentinel spine + UEBA entity risk + Darktrace/Vectra network + EDR — one risk-scored SOAR queue with feedback."

3. Q&A — 18 Questions

Q1. Supervised vs unsupervised vs RL + examples? Supervised: phishing classifier on labeled emails. Unsupervised: IsolationForest on logins vs user baseline. RL: lateral-movement/response policy in a sim. Known-bad → supervised; unknown-weird → unsupervised; sequential → RL.

Q2. How would you build a phishing detector? Features: URL entropy/punycode, domain age, SPF/DKIM/DMARC, HTML/attachment signals, urgency embeddings. Model: XGBoost baseline → fine-tuned DistilBERT. Class weights + PR-AUC, threshold for precision@90% recall, gateway quarantine + active-learning loop.

Q3. Anomaly detection for logins/network? Entity-resolve, windowed features (bytes, ports, Geo velocity), IsolationForest/Autoencoder per peer-group, fused risk score, per-segment threshold + SHAP. Evaluate precision@k + analyst acceptance; shadow-run first.

Q4. Why does accuracy lie? Do the math. TN dominates: 1M logins, 100 attacks, 1% FPR → ~10k FPs, precision ~1% at 99% accuracy. Quote precision/recall/PR-AUC — SOC cares "how many alerts are real."

Q5. Precision vs recall for a SOC? Both — threshold is cost-driven. High-recall for crown jewels, high-precision for broad rollout. Use alert budget: "50 reviews/day → max recall within ≤50 alerts." Report F1 + PR curve.

Q6. ROC-AUC vs PR-AUC? ROC (TPR vs FPR) looks great when TN is huge. PR (precision vs recall) shows true alert quality. For rare attacks decide on PR-AUC.

Q7. Overfit vs underfit? Overfit (train≫val, memorized IPs): regularize, prune, time-split data, simplify. Underfit (both bad): richer features, deeper model. Detect via learning curves; overfit = "works on last campaign, dies on next."

Q8. Data leakage example? Training on inference-unavailable info (e.g., analyst_verdict feature, or shuffled logs leaking future TTPs). Fix: time splits + "would I know this at scoring time?" audit + identical offline/online features.

Q9. Drift and baseline poisoning? Drift: behavior/TTPs shift. Poisoning: attacker deliberately moves baseline (slow exfil ramp). Defend: PSI/KS monitoring, anchor to clean window + peers, label-gated retraining, canary + rollback.

Q10. Feature engineering from SOC logs? Parse → entity-resolve → windowed aggs (counts, entropy, ratios over 5m/1h/24h) → self z-score + peer deviation → sequence/graph + reputation joins. Keep train-serve parity.

Q11. IsolationForest vs Autoencoder vs clustering? IF: fast tabular baseline. Autoencoder: high-dim/sequences with clean data. Clustering: alert dedupe + peer groups. Production = ensemble + UEBA fusion.

Q12. How does UEBA catch insiders? Fuses weak per-entity signals over time (off-hours + bulk download + peer deviation) into one risk timeline — each alone benign, together critical.

Q13. PyTorch training loop? DataLoader → model → loss → Adam → forward, loss, zero_grad, backward, step → validate/early-stop on PR-AUC. Embed categoricals; LSTM/Transformer for sequences.

Q14. XGBoost vs RF vs deep learning on SOC data? Usually XGBoost/RF win on tabular (missing values, interpretable, fast). Deep nets win on text/sequences. Start XGB (scale_pos_weight); go deep only when it plateaus.

Q15. Handling class imbalance? Class weights/scale_pos_weight, focal loss, time splits, threshold on PR curve, PR-AUC eval, scope-narrowing, active learning. SMOTE only inside the training fold (after the time split) — resampling before splitting leaks.

Q16. (Red-teaming) How would you attack an ML model? Poisoning (corrupt training, plant backdoor) + evasion (mutate PE/phish to score clean) + stealing/inversion + prompt injection on LLM copilots. Defend: adversarial training, ensemble + rules + human review, provenance, rate limits, least-privilege tools. Map to MITRE ATLAS.

Q17. Making alerts explainable? Calibrated score + top-3 SHAP reasons + evidence + peer context + playbook + TP/FP feedback buttons. Track acceptance rate and time-to-triage.

Q18. End-to-end ML SOC pipeline? Kafka→Splunk/S3 → entity-resolve → feature store (batch+stream, same code) → ensemble (XGB + IF/AE + Transformer) → risk + SHAP → SOAR → feedback → gated retrain + canary. MLTK for prototype; Sentinel/Darktrace/Vectra as analogues.

Q19. No labels for insider threat — how do you know the model works? Backtest on past IR cases; red-team inject synthetic known-bad; silent labeled holdout from analyst TP dispositions; shadow-mode first; track precision@k vs review capacity, acceptance rate, alert-volume trend; report PR-AUC wherever labels exist.

Q20. How exactly do you pick the operating threshold? precision_recall_curve → max recall with alerts/day ≤ SOC budget; per-segment thresholds; re-tune on drift; never tune on test.

4. Hands-On — Code to Recite

4.1 sklearn: Time Split + XGBoost Phishing Classifier

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report, average_precision_score, precision_recall_curve
from xgboost import XGBClassifier

# Assume df sorted by time; time-based split (no future leakage!)
# features: url_entropy, url_len, domain_age_days, spf_fail, urgency_score, ...
X = df.drop(columns=["label", "timestamp"])
y = df["label"]  # 1 = phish/malware, 0 = benign (highly imbalanced)

# Time split: train on first 80% of time, test on last 20%
split = int(len(df) * 0.8)
X_train, X_test = X.iloc[:split], X.iloc[split:]
y_train, y_test = y.iloc[:split], y.iloc[split:]

# Imbalance-aware
pos, neg = (y_train==1).sum(), (y_train==0).sum()
clf = Pipeline([
    ("scaler", StandardScaler()),
    ("xgb", XGBClassifier(n_estimators=300, max_depth=6, learning_rate=0.05,
        subsample=0.8, colsample_bytree=0.8,
        scale_pos_weight=neg/max(pos,1), eval_metric="logloss", n_jobs=-1))
])
clf.fit(X_train, y_train)

probs = clf.predict_proba(X_test)[:, 1]
print("PR-AUC:", average_precision_score(y_test, probs))
# Threshold from PR curve: precision @ >=90% recall — no magic numbers
prec, rec, thr = precision_recall_curve(y_test, probs)
idx = rec[:-1] >= 0.90
thr_at_90 = thr[idx].max() if idx.any() else 0.5
print(classification_report(y_test, probs >= thr_at_90, digits=3))

Say: time split, scaler in pipeline, scale_pos_weight, PR-AUC + precision@recall.

Line by line: X = df.drop(columns=[...]) builds the feature matrix and drops anything unavailable at scoring time; split = int(len(df) * 0.8) cuts by row order, valid only because the frame was sorted by time first; scale_pos_weight=neg/max(pos,1) sets the positive-class weight to the imbalance ratio, so one phish counts like ~100 legit rows inside the loss; Pipeline guarantees the scaler is fit on train only and applied unchanged at scoring; predict_proba yields scores, not decisions, and the final block picks the highest threshold that still reaches ≥90% recall instead of defaulting to 0.5; classification_report then shows per-class precision/recall at that operating point.

4.2 sklearn: IsolationForest UEBA Baseline

import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Behavioral features per user-hour:
# [failed_1h, bytes_out_z, distinct_hosts_24h, impossible_travel_km, new_host_flag]
X_behavior = np.array([...])  # train ONLY on known-clean window!

iso = Pipeline([
    ("scaler", StandardScaler()),
    ("iso", IsolationForest(n_estimators=200, contamination=0.02,
                            random_state=42))
])
iso.fit(X_behavior)  # unsupervised — no labels

scores = -iso.named_steps["iso"].score_samples(
    iso.named_steps["scaler"].transform(X_behavior))
# Top-k → fuse with supervised score → queue
top_k = np.argsort(scores)[-20:][::-1]
print(top_k)  # prod: per-group threshold + top features

Train on clean window only; contamination ≈ expected rate; fuse + per-segment threshold.

Line by line: each row of X_behavior is one entity-time slice (e.g. one user-hour) of windowed features; fit sees no labels at all — IsolationForest learns a random-partitioning of "normal", not a decision boundary; contamination=0.02 is your prior on how much of the clean window is already weird, and it shifts the decision boundary, so tune it rather than accepting the default; score_samples returns normality (higher = more normal), so the negation makes higher = more anomalous; np.argsort(scores)[-20:][::-1] surfaces the 20 most anomalous entities for the queue — production replaces this global top-k with per-peer-group thresholds.

4.3 PyTorch: Train Loop Sketch (Autoencoder + Classifier)

import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset

device = "cuda" if torch.cuda.is_available() else "cpu"

# --- A) Autoencoder for log anomaly: high recon error = anomaly ---
class LogAE(nn.Module):
    def __init__(self, d=20):
        super().__init__()
        self.enc = nn.Sequential(nn.Linear(d, 12), nn.ReLU(), nn.Linear(12, 6))
        self.dec = nn.Sequential(nn.Linear(6, 12), nn.ReLU(), nn.Linear(12, d))
    def forward(self, x):
        return self.dec(self.enc(x))

torch.manual_seed(42)
X_clean = torch.randn(10000, 20)  # standardized behavior vectors, clean window
X_tr, X_val = X_clean[:8000], X_clean[8000:]  # hold out clean val for the threshold
loader = DataLoader(TensorDataset(X_tr), batch_size=256, shuffle=True)
ae = LogAE().to(device)
opt = torch.optim.Adam(ae.parameters(), lr=1e-3)
loss_fn = nn.MSELoss()

for epoch in range(10):  # early-stop on val in prod
    for (xb,) in loader:
        xb = xb.to(device)
        loss = loss_fn(ae(xb), xb)
        opt.zero_grad(); loss.backward(); opt.step()
    print(f"epoch {epoch} loss {loss.item():.4f}")

# Threshold at 99th pct of clean VAL — never the training window (optimistic)
with torch.no_grad():
    err = ((ae(X_val.to(device)) - X_val.to(device))**2).mean(1)
thresh = err.quantile(0.99)
print(" anomaly threshold:", thresh.item())

# --- B) Classifier loop is same shape, swap head + loss ---
# model = nn.Sequential(nn.Linear(20,64), nn.ReLU(), nn.Linear(64,1))
# loss_fn = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([neg/pos]))
# ... identical forward → loss → zero_grad → backward → step ...

Narrate: batched forward-loss-backward-step, val + PR-AUC checkpoint, pos_weight, threshold from clean-val quantile — never the training window.

Line by line: LogAE compresses a 20-dim behavior vector to a 6-dim bottleneck and reconstructs it — the bottleneck forces the network to learn "normal" instead of copying; DataLoader(TensorDataset(X_tr), batch_size=256, shuffle=True) feeds shuffled minibatches so consecutive gradients are not correlated; inside the loop loss_fn(ae(xb), xb) compares output to input — there are no labels anywhere; opt.zero_grad(); loss.backward(); opt.step() is the full optimizer step from the hand trace above; the anomaly threshold is the 99th percentile of reconstruction error on held-out clean X_val, because using the training window would be optimistic — the autoencoder has already memorized that data.

4.4 TensorFlow/Keras Parity (JD keyword)

model = tf.keras.Sequential([tf.keras.layers.Dense(64, activation="relu", input_shape=(20,)),
                             tf.keras.layers.Dense(1)])  # logits
model.compile(optimizer="adam", loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              metrics=[tf.keras.metrics.AUC(curve="PR")])
model.fit(X_tr, y_tr, validation_split=0.2, epochs=20, class_weight={0: 1, 1: neg/pos},
          callbacks=[tf.keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True)])

Same train-loop concept as PyTorch: class_weight = pos_weight, PR-AUC metric, early stopping; time-split your validation_split.

Line by line: from_logits=True pairs with a single-logit output (no sigmoid in the model) for numerical stability; AUC(curve="PR") makes PR-AUC the tracked metric; class_weight={0: 1, 1: neg/pos} is the Keras spelling of pos_weight; EarlyStopping(patience=3, restore_best_weights=True) rolls back to the best validation epoch, the Keras equivalent of keeping the best checkpoint; validation_split=0.2 takes the last 20% of rows, which is time-correct only if the arrays are already ordered by time.

5. Revision Table

6. Mistakes to Avoid

30-second pitch:

"For Walmart scale I'd ensemble: XGBoost for known malware/phish, IsolationForest plus autoencoder for novel insider anomalies, and an LSTM over event sequences — all on windowed entity features with time-based splits. I'd tune thresholds for precision at the SOC's alert budget, explain every alert with SHAP, and close the loop with analyst feedback and drift-gated retraining. And because I red-team, I'd also adversarially test it — evasion, poisoning, and prompt-injection on any LLM copilot — mapped to MITRE ATLAS."

Drill: flood math + pipeline + IF/XGB from memory.