Evolving Adversarial Prompts: The SPE-NL Genetic Algorithm Behind Basilisk

The Problem with Static Payloads

Current LLM security testing tools operate on a fundamentally flawed premise: that a fixed list of known-bad prompts is sufficient to evaluate a model's safety posture. This approach has two critical failure modes.

First, static payloads go stale. Safety fine-tuning is a continuous process — providers push updates regularly, and each update invalidates a portion of any existing payload database. A red team that tested GPT-4 six months ago is not testing the same model today.

Second, and more fundamentally, the search space of natural language adversarial inputs is effectively infinite. Manual curation captures only the prompts that human researchers think to try. The most dangerous vulnerabilities often exist in unexpected corners — specific combinations of encoding, persona, and linguistic structure that no human would think to construct manually but that a search algorithm can discover systematically.

SPE-NL: High-Level Overview

Smart Prompt Evolution for Natural Language (SPE-NL) applies classical genetic algorithm principles to the domain of adversarial prompting. The core abstraction: an adversarial prompt is an organism. Its fitness is determined by how effectively it achieves its adversarial objective against the target model. The algorithm maintains a population of prompts, applies selection pressure, and iteratively produces stronger generations.

Population & Initialization

Each run initializes a population 𝒫⁽⁰⁾ of N = 100 candidate prompts. Rather than random initialization, SPE-NL uses warm-start seeding from Basilisk's YAML payload database 𝒟, selecting templates appropriate for the target attack module category:

Warm-start initialization is critical for practical efficiency. Random natural language strings have near-zero probability of producing any adversarial signal — the fitness landscape would be essentially flat and convergence would be extremely slow. Starting from known-viable attack templates biases the search toward high-fitness regions of the prompt space from generation zero.

The Fitness Function

The fitness function is the most critical design decision in any genetic algorithm. In SPE-NL, fitness is a convex combination of four independent signals extracted from the target model's response to prompt p:

α=0.35 · β=0.30 · γ=0.25 · δ=0.10 · (α+β+γ+δ = 1)

Selection Strategy

SPE-NL uses tournament selection to form the mating pool ℳ⁽ᵗ⁾ of size N/2 at each generation t. For each slot, k=3 prompts are sampled uniformly at random from the current population and the highest-fitness individual wins:

Tournament selection with k=3 was chosen over fitness-proportionate (roulette wheel) selection for two reasons: it's robust to fitness scaling issues, and it maintains selection pressure even when population fitness values are clustered closely together — which is common in later generations when the population has partially converged.

Additionally, an elite set ℰ_elite of the top e ∈ [5, 20] individuals (configurable via elite_count) is carried forward unchanged to the next generation. Elitism prevents the best-found solutions from being lost to genetic drift.

Crossover Operators

Given two parents pₐ, p_b ∈ ℳ⁽ᵗ⁾, a child prompt p′ = χ(pₐ, p_b) is produced with probability p_c (crossover_rate = 0.5) by one of five operators:

Mutation Operators

Each child p′ undergoes mutation with probability p_m (mutation_rate = 0.3). A single operator μ is sampled uniformly from the set 𝒰 = {μ₁, ..., μ₁₀}:

The 10 operators target fundamentally different layers of the model's defensive stack:

Generational Update

The next-generation population is assembled by combining the elite set with newly bred children:

This process continues until |𝒫⁽ᵗ⁺¹⁾| = N = 100. The complete population is then evaluated against the target model, each prompt sent as a fresh inference request, and fitness scores are updated for generation t+1.

# Simplified generational update loop
def evolve_generation(population, target, config):
    # Evaluate fitness for all prompts
    scored = [(p, fitness(p, target)) for p in population]
    scored.sort(key=lambda x: x[1], reverse=True)
    
    # Elitism — carry forward top performers
    elite = [p for p, _ in scored[:config.elite_count]]
    
    # Tournament selection → mating pool
    mating_pool = tournament_select(
        scored, 
        pool_size=len(population) // 2,
        k=3
    )
    
    # Breed new generation
    offspring = []
    while len(offspring) < len(population) - len(elite):
        pa, pb = random.sample(mating_pool, 2)
        child = crossover(pa, pb, config.crossover_rate)
        child = mutate(child, config.mutation_rate)
        offspring.append(child)
        
        # Flag relative breakthroughs immediately
        score = fitness(child, target)
        if score >= config.fitness_threshold:  # default 0.7
            emit_breakthrough(child, score)
    
    return elite + offspring

Termination & Breakthroughs

Evolution terminates on any of three conditions:

Relative Breakthrough Logic (v1.0.6)

Introduced in v1.0.6, any prompt satisfying the relative breakthrough threshold is immediately flagged and reported — without waiting for evolution to complete:

This is critical for operational use. In real red team engagements, you don't need a perfect bypass — you need actionable findings fast. A prompt at fitness 0.7 is already leaking partial system prompt content or achieving partial compliance. That's a reportable finding. The breakthrough is logged to the forensic audit trail and emitted as a scan:breakthrough_found WebSocket event in real time.

Empirical Results

SPE-NL was evaluated against an intentionally vulnerable LLM endpoint and three commercial APIs using standard scan mode (5 generations, full payload library) across the guardrail bypass module category.

Key Findings

Syntactic vs. Semantic Attacks

Across all evaluation runs, encoding-based operators (μ₂ Encoding Wrap, μ₈ Homoglyphs, μ₁₀ Token Smuggling) contributed disproportionately to high-fitness breakthroughs. This is consistent with the hypothesis that current LLM safety fine-tuning is primarily trained on semantic content — it recognizes what a prompt means, not how it's encoded.

Semantic operators (μ₁ Synonym Swap, μ₅ Structure Overhaul) produced more gradual fitness improvements but were less likely to produce sudden breakthrough events. The implication: syntactic obfuscation bypasses the classifier layer, while semantic reframing works by staying below the compliance threshold.

Multilingual Safety Gaps

Role injection (μ₃) and language shift (μ₄) produced the highest compliance scores c(p). Language shift was particularly effective against models with strong English-language guardrails — mid-sentence code-switching to languages with lower training representation consistently reduced refusal rates.

Cross-Provider Divergence

Differential testing across major commercial providers revealed that in the system prompt extraction category, divergence rates exceeded 60% — the same evolved payload that achieved f(p) ≥ 0.7 against one provider was refused by another. This demonstrates that alignment is not a generalizable capability but a provider-specific implementation detail.

Conclusion

SPE-NL demonstrates that evolutionary computation is a viable and highly effective approach to adversarial LLM evaluation. By treating prompts as organisms under selection pressure, Basilisk discovers attack vectors that static payload libraries miss — achieving 92% relative improvement in attack success rate over static baselines at generation 5.

The key design insights: warm-start initialization from domain-specific payload databases, a multi-signal fitness function that rewards partial success, and the combination of syntactic and semantic mutation operators targeting different layers of the model's defensive stack.

The full research paper with complete methodology, formal proofs, and extended evaluation is archived on **Zenodo** and **Figshare**. The framework is open source under AGPL-3.0.