Fix the conditions first
Save the anchor, dictionary, selected memories and state version. Record the model, date, generation settings and context delivery method.
Compare at least no memory, retrieval only and a full system with review and reflection. Randomise tasks and reserve some prompts for evaluation.
Choose measurements
Check facts, relationships, source accuracy, agreed boundaries and the ability to report gaps. Count false memories and imposed conclusions separately.
AIS offers twenty Russian pressure scenarios, including authority, false memory, flattery and emotional arguments. Reports include responses, detected stance, memory records and gate decisions. heuristic_pass reflects predefined pattern checks with a limited semantic scope.
Keep the experiment isolated
Run evaluation in a temporary log. Retain fictional test cases as experimental results with their provenance, so the canonical biography stays accurate.
The documentation recommends blind comparisons and at least two independent reviewers. Also test revision under strong evidence. Repetition and intelligible changes are more informative than a single dramatic reply.
ais --workspace ./kai pressure run \
--battery examples/pressure_battery_v0_1.jsonl \
--output reports/pressure-v0.1.jsonRead results with their limits
A successful session establishes that the result occurred under those conditions. Repeat across models and phrasings, retain failed runs and save the tasks with their ratings.
Phrase matching alone may reward style imitation. Compare factual accuracy, uncertainty and transfer to new cases as well.
Put it into practice
- Save a baseline and test prompts.
- Compare no memory, retrieval only and the full system.
- Record factual errors, provenance and uncertainty separately from style.
Check the result
- Conditions are reproducible.
- Evaluation preserves the canonical memory.
- Heuristic scores are accompanied by meaningful review.