Basilisk: An Evolutionary AI Red-Teaming Framework for Systematic Security Evaluation of Large Language Models

Adversarial Prompting

Prompt injection attacks, first described by Perez and Ribeiro [1], exploit the inability of LLMs to distinguish between instruction context and user-supplied data. Subsequent work demonstrated that these attacks could be delivered indirectly through documents, web content, and tool outputs retrieved by agentic systems [2]. Jailbreaking — eliciting policy-violating content through adversarial prompts — has been studied extensively, with Wei et al. [5] identifying competing objectives and mismatched generalization as root causes.

Automated Red Teaming

Perez et al. [7] introduced the use of a separate "red LM" to automatically generate test cases for a target model. Mehrotra et al. [8] proposed Tree of Attacks with Pruning (TAP). Chao et al. [9] demonstrated PAIR, achieving high attack success rates with limited queries. These LLM-assisted approaches require access to a capable attacker model and do not produce deterministic, auditable sequences. Basilisk complements these with a deterministic evolutionary approach that does not require an attacker LLM (though one may optionally be configured).

Fuzzing and Mutation-Based Testing

Mutation-based fuzzing is well-established in traditional software security [10]. Yang et al. [6] applied fuzzing concepts to LLM testing. Basilisk extends this paradigm with evolutionary selection pressure, maintaining a population of candidates across generations rather than generating mutations independently — enabling the discovery of attack chains that require multiple coordinated mutations.

Evaluation Frameworks

Guo et al. [11] surveyed LLM safety evaluation benchmarks, finding significant gaps in adversarial robustness coverage. HarmBench [12] provides a standardized jailbreak benchmark but is not designed for operational security testing. OWASP's LLM Top 10 [13] provides a practitioner-oriented taxonomy. Basilisk is the first framework to systematically cover all applicable OWASP LLM Top 10 categories with evolutionary optimization and CI/CD integration.

Configuration Hierarchy

Configuration resolution follows a strict priority cascade: CLI arguments override YAML file values, which override environment variables, which override compiled defaults. This enables Basilisk to be used interactively, configured per-project, and operated in zero-configuration CI environments.

Provider Abstraction Layer

All LLM interactions are mediated through the BaseProvider interface, which exposes send_prompt() and stream_prompt(). Three concrete implementations are provided:

• ▸LiteLLM Adapter — Access to 100+ providers including OpenAI, Anthropic, Google, Cohere, Mistral, and self-hosted models.
• ▸Custom HTTP Provider — Targets arbitrary HTTP endpoints with configurable request/response schemas for proprietary or air-gapped deployments.
• ▸WebSocket Provider — Supports real-time bidirectional communication for streaming-native endpoints.

Audit Trail

Every scan produces a tamper-evident audit trail in JSONL format. Each entry contains a SHA-256 hash of the previous entry (previous_hash), forming a cryptographic chain. Sensitive values are automatically redacted. The audit system is enabled by default and can be suppressed with BASILISK_AUDIT=0.

Evolutionary Formulation

SPE-NL frames adversarial prompt generation as a discrete optimization problem. Let 𝒫 = {p₁, p₂, …, pₙ} denote a population of N candidate prompts, where each prompt pᵢ ∈ Σ* is a finite string over a natural-language token alphabet Σ. The algorithm seeks to maximise the fitness functional f: Σ* → [0,1] over successive generations t = 0, 1, …, T_max.

Fitness Function

The fitness of a prompt p is a convex combination of four response-derived signals:

α=0.35, β=0.30, γ=0.25, δ=0.10  |  α+β+γ+δ = 1

Selection

At each generation t, a mating pool ℳ⁽ᵗ⁾ ⊂ 𝒫⁽ᵗ⁾ of size |ℳ| = N/2 is formed via tournament selection. For each slot, k=3 prompts are drawn uniformly at random and the highest-fitness prompt is retained:

An elite set ℰ_elite ⊂ 𝒫⁽ᵗ⁾ of the top e ∈ [5, 20] individuals (configurable via elite_count) is carried forward unchanged to 𝒫⁽ᵗ⁺¹⁾.

Crossover Operators

Given two parent prompts pₐ, p_b ∈ ℳ⁽ᵗ⁾ selected with probability p_c (crossover_rate, default 0.5), a child p′ = χ(pₐ, p_b) is produced by one of five operators:

Mutation Operators

Each child p′ is subject to mutation with probability p_m (mutation_rate, default 0.3):

Generational Update

The population is assembled until |𝒫⁽ᵗ⁺¹⁾| = N, then evaluated against the target — updating the fitness landscape for generation t+1.

Termination Conditions

Relative Breakthrough Logic (v1.0.6)

Any individual prompt satisfying the relative breakthrough threshold is immediately flagged, logged to the forensic audit trail, and emitted as a finding:

This provides defensive researchers with "near-miss" data revealing the evolutionary trajectory of successful attack vectors — even before perfect bypass is achieved.

Reconnaissance Modules

Prior to attack execution, five reconnaissance modules characterize the target environment:

• ▸Model Fingerprinting — Identifies underlying model family (GPT, Claude, Gemini, Llama, Mistral) via behavioral probing.
• ▸Guardrail Profiling — Maps active safety filters across 8 content categories at 3 severity tiers.
• ▸Tool Discovery — Enumerates available function-calling schemas through structured elicitation.
• ▸Context Window Measurement — Determines effective token limit via binary search.
• ▸RAG Detection — Identifies retrieval-augmented generation patterns through response analysis.

Differential Testing

Basilisk's differential testing engine executes identical payloads against multiple LLM providers simultaneously and identifies behavioral divergences — cases where one provider refuses while another complies. This reveals inconsistencies in safety implementations across the LLM ecosystem. Per-provider resistance rates are computed and reported via the desktop and CLI interfaces.

Guardrail Posture Assessment

The posture assessment module provides a non-destructive characterization of a deployment's safety posture. It probes 8 content categories at 3 tiers of increasing adversariality (benign → moderate → adversarial) and assigns a strength rating (None / Weak / Moderate / Strong / Aggressive) and letter grade (A+ through F) per category. Unlike full scans, posture assessment never attempts exploitation — making it safe for scheduled production monitoring.

Experimental Setup

I evaluated Basilisk against a publicly accessible intentionally vulnerable LLM endpoint (basilisk-vulnbot.onrender.com) and three commercial LLM APIs, using standard scan mode (5 generations, full payload library) across all 29 attack modules. Metrics: Attack Success Rate (ASR), Breakthrough Rate (BR) at f(p) ≥ θ_rel, and mean Generations to Breakthrough (GtB).

Evolutionary Improvement

SPE-NL achieves a 92% relative improvement in ASR over static payloads at t=5, demonstrating consistent improvement across generations.

Mutation Operator Effectiveness

Encoding-based operators (μ₂, μ₈, μ₁₀: Encoding Wrap, Homoglyphs, Token Smuggling) contributed disproportionately to high-fitness breakthroughs, consistent with prior findings that LLM safety fine-tuning is more robust to semantic than syntactic attacks. Role injection (μ₃) and language shift (μ₄) produced the highest compliance scores c(p), suggesting safety-alignment gaps in persona and multilingual contexts.

Differential testing revealed that in the system prompt extraction category, divergence rates — cases where at least one provider satisfied f(p) ≥ θ_rel while others refused — exceeded 60% of payloads, highlighting the absence of industry-wide standards for LLM safety behavior.

Exploiting Dynamic Sparse Inference

A novel vulnerability class explored in this research is the adversarial manipulation of Dynamic Input Pruning (DIP). I demonstrate that SPE-NL can evolve prompts designed to induce "State-Collapse" attacks. By generating inputs that maximize activation sparsity in non-safety channels, Basilisk can force inference-time pruning mechanisms to discard weights associated with safety-alignment subspaces — effectively disabling the model's filters through its own optimization logic.

CI/CD Integration

The Basilisk GitHub Action integrates LLM security testing into continuous integration pipelines. Configurable inputs include target endpoint, provider, scan mode, failure threshold (fail-on), and baseline for regression comparison. SARIF reports are automatically uploaded to the GitHub Security tab via Code Scanning, enabling inline vulnerability annotations on pull requests.

Report Formats

Responsible Disclosure Posture

Basilisk is designed for authorized security testing. The framework targets a user-supplied endpoint; no scanning occurs without explicit configuration. Docker and PyPI distributions include usage guidelines emphasizing authorized-use-only. The intentionally vulnerable demo target (basilisk-vulnbot.onrender.com) is provided specifically for safe experimentation.

Dual-Use Considerations

Like all offensive security tools, Basilisk presents dual-use risks. I mitigate these through: (1) open publication enabling defensive research and guardrail improvement; (2) audit trails creating accountability for scanning activity; (3) AGPL-3.0 licensing requiring derivative works to remain open; and (4) the posture assessment module providing defensive value without requiring attack execution.

Research Ethics

My empirical evaluation was conducted against endpoints I operate or for which I hold authorization. No production systems were tested without consent. Findings from commercial provider testing are reported in aggregate without identifying specific payloads that could be directly weaponized.