GitHubIntelligence Index
regex-golfLower score is better

Regex Golf

Generate the shortest valid regular expression matching all 10 target strings while excluding all 10 distractor strings across 25 deterministically generated puzzles. Lower is better.

Lower score is better46 models scoredCached in Neon Postgres

Leaderboard

Intelligence Index
MethodologyHow scoring works

Prompt

The model receives the MATCH and REJECT string lists and must reply with only the raw regular expression in Python re syntax.

Score

Lower is better. Per puzzle: regex length plus 20 per missed MATCH, 20 per falsely matched REJECT, and a flat 200 for invalid regex or evaluation timeout. The benchmark score is the average across all 25 puzzles.

Execution

Benchmark runners execute locally, evaluate submissions with a sandboxed Python subprocess that is killed on timeout to catch catastrophic backtracking, use OpenRouter with reasoning output excluded, cache results in Neon, and skip recomputation for models that already have stored scores.