Psyche
Open-source psychometric persona profiling framework. Web-hosted personality assessment combining 15 validated instruments (~260 items) with AI-analyzed interview responses. Anonymous by design — no IP, no accounts, no cookies.
- 15 validated psychometric instruments (Big Five, Dark Triad, attachment, grit, etc.)
- AI-generated narrative personality reports
- Claude MCP integration for conversational interviews
- Cross-LLM benchmarking schema for research
- Two-level consent design for IRB-exempt research
Activity Timeline
- PsycheEval v0.3 verified clean after Opus 4.8 contamination purge; Phase B approved and pre-registered.
2,757 assistant outputs and 14,733 pairwise scores confirmed after regenerating contaminated Opus 4.8 outputs to 4.7. Phase B rescaling plan uses target-conditioned AB/BA with per-judge calibration.
- Psychoval evaluation battery restarted with Opus 4.7 and autonomous permissions.
Thousands of test cases queued for the restart run. Autonomous permissions granted to avoid interruption during the full battery.
- Behavioral directive regeneration launched via Fable 5 multi-run corpus analysis.
Framework built on 39-instrument psychometric battery. Personality audit methodology formalized with measurement-based claim validation against psychometric data and interview evidence. Three corpus analysis runs initiated; first completed, two recovering from environment failures.
- PsycheEval v0.2 blocked on v0.1; enum enforcement bug found in judge validation.
Pairwise judge RedFlag fields lacked enum enforcement, risking vocabulary drift across runs. v0.2 infrastructure prerequisites documented: anchored scores, dual scales, delta tracking. Python enum hardening added to v0.1 scope.
- PsycheEval schema bug found (hardcoded RedFlag vocab), pairwise validation errors blocked, test coverage added.
18-label RedFlag vocabulary hardcoded in prompts; v0.2 requires 24 — fix is dynamic enum generation. Pairwise judges returning free-text in enum fields causing validation failures. v0.1 completion blocked on C5 claim revisions; v0.2 generation deferred until v0.1 pairwise is clean.
- PsycheEval v0.3 judge calibration: paraphrased-anchor rubric and ground-truth criteria defined.
Evaluator consistency framework designed via anchor paraphrasing. Synthetic-user ground truth specifies helpful vs unhelpful response criteria. Validation threshold: ≥0.5 SD mean shift or Spearman ρ < 0.7 triggers failure flag.
- PsycheEval enum validation bug found and fix approach defined; v0.1 path complete; v0.2 planned.
Judges returning free-text in enum-typed fields — fix derives red_flag vocabulary from Python enum and adds sync tests. 28 sessions produced 30+ spec documents: judge prompt variants, anchor-sensitivity methodology, user archetype rubrics, and v0.2 scaffolding plan.
- PsycheEval v0.1: diagnosed pairwise judge red-flag enum mismatch, fixed and added logging safety valve.
Root cause was hardcoded vocabulary lists diverging from Python RedFlag enum. Fix generates vocabulary dynamically. _coerce_red_flags() added to surface mismatches to validation_warnings files rather than drop them silently. v0.1 completion path and v0.2 scope both finalized.
- PsycheEval v0.2 judge framework fully documented.
Anchored 0–10 rubric finalized, replacing v0.1's compressed ceiling. Synthetic user profiles defined with ground-truth preferences and known blind spots. Full v0.2 red-flag taxonomy extended and documented.
- PsycheEval v0.1 validation bug traced; completion path and v0.2 roadmap finalized.
Judges returning free-text in red_flags instead of enum values — root cause: unenforced schema. v0.1 scoped to cross-provider comparisons. v0.2 roadmap adds anchored scores, length-control deltas, targeted pairing. Remote Session Launcher workflow documented as a side effect of session recovery work.
- Psyche Public launched on app.ashitaorbis.com/psyche
Web-hosted Lite tier (15 instruments, ~260 items) with AI-generated personality reports via DeepSeek V3.2. Anonymous sessions, two-level consent flow, Claude MCP integration for conversational interviews.