Neuro-Sentry — LLM Threat Detection
4-stage hybrid pipeline for prompt injection and jailbreak detection
FastAPIDeBERTa v3PyTorchGroq API (Llama 3.3 70B)React 18 + ViteDocker ComposeSQLAlchemyNginx
76% acc on 150 unseen prompts217 rules · 14 categories9.4 ms avg pipeline latency
Team project
Four-person university project (KR Mangalam University, BCA AI & DS, Sem 4). I led it as team leader and built the backend: the 4-stage detection pipeline, the 217-rule engine, and all attack simulation and red-team tooling. Akash Sharma handled blue-team defense logic. Bhavya Rattan and Lakshya Dangwal built the React frontend and visualization layer.
The problem
LLM safety testing is mostly vibes: throw a few jailbreaks at the model, watch it refuse, call it secure. That tells you nothing about coverage, nothing about latency cost, and nothing about what happens when an attacker adapts. The goal was a system that both attacks and defends — and reports numbers for both sides.
Key decision
Rules first, ML second — not rules instead of ML. Stage 1 runs 217 regex patterns across 14 categories with input normalization (homoglyph collapse, zero-width stripping) in ~0.92 ms. Any prompt scoring ≥ 85 gets fast-blocked before touching the classifier. Everything else hits Stage 2: a fine-tuned DeBERTa v3 binary classifier (~7.3 ms warm GPU, ~50 ms CPU). Stage 3 fuses both scores at 0.4 rules / 0.6 model weight, with a dangerous-content floor at 75 so a confident-looking ML prediction can't override a categorical rule match.
How it works
Five frontend tabs (Command Center, Attack Lab, Neural Link, Security Ops, Red Team) talk to a FastAPI backend. The detection pipeline has three stages: Stage 1 — Rule Engine with 217 regex rules across 14 categories: Jailbreak, Prompt Injection, Data Extraction, Encoding/Obfuscation, Social Engineering, Privilege Escalation, Roleplay, Manipulation, Dangerous Content, Token Manipulation, Context Overflow, Indirect Injection, Model Extraction, Multi-Agent Attack. Fast-block path short-circuits at score ≥ 85 (~0.92 ms). Stage 2 — fine-tuned DeBERTa v3 binary classifier, GPU or CPU. Stage 3 — weighted score fusion (0.4 rules / 0.6 ML) with a DC-category floor at risk 75, forcing auto-block regardless of ML output. Supporting systems: adaptive per-session risk escalation up to 2.0×, structured JSON audit log, API-key auth, SQLAlchemy for audit/session data, Groq for actual LLM responses. Red-team tooling: single-prompt Attack Lab with full pipeline breakdown, batch runs up to 200 prompts over SSE with live progress, export to CSV / JSON / styled PDF. Docker Compose runs backend + Nginx-fronted React frontend as a two-service stack.
Results
Rigorously evaluated on 150 strictly unseen prompts (100 benign from databricks-dolly-15k + 50 zero-day attack prompts crafted for the eval — no overlap with training data). Results: 11 blocked (7.3%), 5 flagged (3.3%), 134 allowed (89.3%). Accuracy: 76%. False negatives: 35. False positives: 1. Avg pipeline latency: ~9.4 ms. Rule engine: ~0.92 ms. Warm ML inference: ~7.3 ms. Cold start (model load): ~2960 ms. Configurable thresholds: BLOCK_THRESHOLD=65, FLAG_THRESHOLD=35.
What went wrong
35 false negatives on the rigorous eval — attacks that slipped through. All were highly creative zero-day obfuscation vectors not in the classifier's training distribution. The hybrid approach means the rule engine caught what it knew; the ML missed what it hadn't seen. This is the honest trade in any rule+ML system: coverage is bounded by what you've thought to write rules for, and generalization is bounded by what the model was trained on. The DC floor stops the worst-case failure (the model declaring a weapons prompt benign), but novel phrasing still leaks through.
The 76% accuracy figure is from a real evaluation against 150 unseen prompts — including adversarial zero-day attacks specifically designed to evade the defense. A production deployment would need continuous red-teaming and rule/model updates as attack patterns evolve.