# OpenResearch: a research evaluator for the InferOptimal OpenEvolve tool **Status:** In development: still offered, but not recommended yet, because it is not performing well enough. Its measured results are below. Every harness measured side by side, on the same model: https://www.inferoptimal.com/tools/results OpenResearch is an opt-in add-on to the InferOptimal OpenEvolve tool. Use it when the objective is a research question with no fixed metric, especially one where the obvious answer may be overridden by something buried in the material: a later amendment filed under another title, a definition elsewhere, a court decision, an errata notice, another document that changes the rule. Nothing installs, starts or calls it for you. Connecting an InferOptimal account or endpoint never installs it, the OpenEvolve tool never installs or starts it, and it runs nothing in the background: it works only while an OpenEvolve run you started calls its evaluator, or while you run one of its commands. ## How it works Every OpenEvolve run needs an evaluator; OpenResearch is one driven by agents instead of a fixed metric. - **The evolved program is a research brief**: the question, the leading conclusion so far, the evidence it rests on, and the next jobs. - **Each evaluation launches research agents on your endpoint** through the InferOptimal subagent tool (`agents.mjs`, the same `run_agents` your coding agent uses), over a folder or repository read-only. - **The fanout mode (`--fanout`) pushes past a ruling the loop believes.** Each agent does one short directed job: read a slice of the documents in full, hunt for one kind of override (a later amendment, a higher-level law, a definition, a court decision, an effective date or exception), follow up a passage another agent found, or assume the ruling is wrong for one reason. Agents report exact passages, never an answer. A decider model rates whether the findings should change the ruling and, as OpenEvolve's proposer, writes the next brief. A job that found nothing is not queued again under the same ruling. - OpenEvolve's own islands, selection and migration build on the best-rated briefs. ## What we have measured Experiment 5, 2026-10-18 and 19, on our generated legislation benchmark, counting only questions where OpenResearch's default loop ended on a wrong ruling: the fanout loop ended right in 9 of 17 runs with a Claude Sonnet decider, 3 of 8 with a GPT-6.1 Sol decider and 3 of 10 with the endpoint's own model deciding, and the default loop in 0 of 14. On questions from two jurisdictions held back while the loop was built, the Sonnet decider ended right in 4 of 7. A single GPT-6 Astra pass got 3 of these 7 questions right. On Anthropic's public performance take-home, one run steered only by the Sonnet decider's judgment (no cycle count reached the loop) took the kernel from 147,734 to 2,653 cycles in 3 hours, and a second, with another seed, to 2,905 cycles by minute 170. One generated benchmark and a few runs per question, not a general accuracy figure. In the legislation runs the research agents ran on an InferOptimal endpoint, 16 agents per run, and the decider also wrote the briefs. In the take-home runs a frozen correctness check rejected wrong kernels and cycles were measured only afterwards. Answer keys and cycle counts were used only to grade; they never reached an agent, the decider or OpenEvolve. ## Good at, bad at, cost Each line is marked measured, with where it was measured, or designed for and not measured yet. The measured lines come from our generated legislation benchmark (hard questions whose obvious answer is overridden by something buried in about 2,830 documents, 1 to 4 runs per question) and from a first pass over public benchmarks that ended when the GLM box we measured on was torn down (1 to 4 items each), and the kernel lines from Anthropic's public performance take-home. A cost per run prices GLM time at $20 an hour, that box's rate. They are not a general accuracy figure. Good at: - Hard research where a conclusion has to be pushed past the obvious answer: with a Claude Sonnet decider the fanout loop was right 7 of 14 times on the test questions the other harnesses also ran, against 0 of 10 for the plain OpenEvolve loop and 1 of 15 for one plain agent, and 9 of 17 over all Experiment 5 test questions (4 of 7 on held-out ones). (Measured: Experiment 5, rounds 1 to 3, the held-out repeats, trap check (5.10) and outside harnesses (5.6), on our generated legislation benchmark, 2026-10-18 and 19.) - Right on the 3 short-task items it reached: tax law (DeonticBench), NBA rule decisions (RuleArena) and data analysis (KramaBench), 1 of 1 each. (Measured: Experiment 6, first pass on the public DeonticBench, RuleArena and KramaBench benchmarks, 2026-10-19.) - Steering a search by judgment when there is no score: one run steered only by the Sonnet decider took a kernel from 147,734 to 2,653 cycles in 180 minutes, and a second, with another seed, to 2,905 cycles by minute 170. (Measured: Experiment 5, kernel challenge holdout (5.5), on Anthropic's public performance take-home, 2026-10-19.) Bad at: - Slow: 30 to 40 minutes a run on the short benchmarks and a median 37 minutes, up to 63, on hard research. On a very large corpus it hit the 40-minute cap with no ruling, because its agents read whole 334 MB files. (Measured: Experiment 5, outside harnesses (5.6), on our generated legislation benchmark, 2026-10-18 and 19, and Experiment 6, first pass on the public DeonticBench, RuleArena, KramaBench and BRIGHT benchmarks, 2026-10-19.) - Overkill for short, well-specified tasks: on the same benchmarks one plain agent was right 2 or 3 times in 4 in a median 1 to 6 minutes for $0.44 to $2.28 a run, against 30 to 34 minutes and $12.29 to $13.26 a run for OpenResearch. (Measured: Experiment 6, first pass on the public DeonticBench, RuleArena and KramaBench benchmarks, 2026-10-19.) - Some questions it misses every time: on 3 of the 7 test questions every Sonnet-decider run ended wrong (0 of 6). In rounds 1 and 2, 13 of 19 wrong runs had already cited every document of the true chain, so the failures are judgment, not search. (Measured: Experiment 5, rounds 1 to 3 and the held-out repeats, on our generated legislation benchmark, 2026-10-18 and 19.) - The endpoint's own model as decider stalls on a full endpoint: its ratings queue behind the agents' calls, and its three runs there reached 2, 1 and 4 evaluations in 61, 44 and 56 minutes. (Measured: Experiment 5, round 3 (5.7, 5.8), on our generated legislation benchmark, 2026-10-19.) - On the 5 hard questions both ran, the Microsoft Agent Framework fan-out ended right more often: 9 of 14 test runs against 7 of 14 for the Sonnet-decider fanout loop. (Measured: Experiment 5, outside harnesses (5.6), on our generated legislation benchmark, 2026-10-18 and 19.) Cost: - With the Claude Sonnet decider, $12.29 to $14.67 a run on average per benchmark, with GLM time priced at the box's $20 an hour; about $1.6 to $2.2 of each run that reached a ruling is Claude usage for deciding and writing the briefs ($1.63 on average over all 32 Experiment 5 Sonnet runs: $1.16 writing briefs, $0.46 rating). (Measured: Experiment 5, all 32 Sonnet-decider runs and outside harnesses (5.6), on our generated legislation benchmark, 2026-10-18 and 19, and Experiment 6, first pass on the public DeonticBench, RuleArena, KramaBench and BRIGHT benchmarks, 2026-10-19.) - With the GPT-6.1 Sol decider, the longest runs: a median 57 minutes on dev questions and 65 on held-out ones; a Codex sign-in reports no price for its deciding or brief writing. (Measured: Experiment 5, round 3 (5.7, 5.8), on our generated legislation benchmark, 2026-10-19.) - With the endpoint's own model deciding, nothing beyond endpoint time. (Designed for, not measured yet.) ## Before you start - An InferOptimal endpoint that serves chat completions with tool calls. - The InferOptimal subagent tool installed for that endpoint, normally at `~/.inferoptimal/agents.mjs` (the console's "Run as subagents" command, or `get_codex_config` from the InferOptimal MCP). It needs Node 18 or newer. - The InferOptimal OpenEvolve tool installed, normally at `~/.inferoptimal/openevolve` ([its guide](https://www.inferoptimal.com/swarm/README.md)). - Python 3.12 or newer, `make` and `curl`. - For a Claude Sonnet decider, Claude Code (`claude`) signed in on this machine; for GPT-6.1 Sol, Codex (`codex`) signed in with access to that model. Their usage counts against your own Claude or ChatGPT account. ## Install ```sh mkdir -p ~/.inferoptimal/openresearch && cd ~/.inferoptimal/openresearch && curl -fsSLO https://www.inferoptimal.com/openresearch/Makefile && make fetch install ``` `make fetch` downloads the add-on's files from inferoptimal.com into the current folder; `make install` makes `.venv` with its pinned dependencies (`requirements.txt`: `ddgs` and `trafilatura` for web research, `openai` for the decider, `psutil` to stop silent agents). From a source checkout of the InferOptimal repository, `make -C openresearch install` does the same with the files already there. ## Start a fanout run 1. Write the run: the question, the folder of documents, a new run folder and your decider. ```sh ~/.inferoptimal/openresearch/.venv/bin/python -m openresearch init --fanout --decider sonnet --question "" --folder --out --agents-script ~/.inferoptimal/agents.mjs ``` `init` writes `brief.md` (the seed brief), `evaluator.py`, `openresearch.json` and `system_message.txt` into the run folder and prints the OpenEvolve tool command for them. For a fanout run that command is the run the measured runs used: at least `converge.min_evaluations` evaluations (12 in the shipped constants), the floor below which a run is not judged, with one wave of 4 evaluations in flight, 16 agents (`fanout.agents_per_run` over `fanout.slots`). 2. With a `sonnet` or `sol` decider, start the brief writer in its own terminal and keep it running for the run: ```sh ~/.inferoptimal/openresearch/.venv/bin/python -m openresearch writer --model sonnet ``` It prints one line of JSON whose `openevolve_flags` (`--base-url`, `--model`, `--key-file`) point the OpenEvolve tool at a local bridge that has your decider write each brief. Stop it with Ctrl-C when the run ends. 3. Run the printed command from `~/.inferoptimal/openevolve` with `.venv/bin/python` in place of `python`. Give it the writer's three flags, or, with the endpoint as the decider, your endpoint's base URL, model and key file. The research agents always run on your endpoint through the subagent tool. ## Deciders - `--decider sonnet`: Claude Sonnet through your Claude Code sign-in, suggested for Claude users. - `--decider sol`: GPT-6.1 Sol through your Codex sign-in, suggested for Codex and ChatGPT users. - No `--decider`: the endpoint's own model decides and writes the briefs, at no extra cost. Use it only when the endpoint has spare capacity: its ratings queue behind the agents' own calls, and on a full endpoint our runs with it stalled. The decider has no tools: it judges from the passages the agents quote. ## Reading the answer and stopping Every brief opens with a `LEADING CONCLUSION:` line, followed by the chain of rules and overrides with their files and exact passages, a `WHERE WE ARE STUCK` line and the next jobs. The run's answer comes from its briefs, not any one agent's report: ```sh ~/.inferoptimal/openresearch/.venv/bin/python -m openresearch status --run ``` prints the evaluations so far, the run's `conclusion` and `stop`. For a fanout run, the conclusion is the ruling most of the latest wave's briefs lead with, the rule the measured runs were judged by. `stop` turns true once the decider reports the corpus `swept` (every slice and hunt done under the current ruling with nothing left to run), or once the run has done at least `converge.min_evaluations` evaluations over `fanout.run_minutes` minutes (30) without reaching a ruling it had not held `converge.soft_window_seconds` earlier (10 minutes). Quiet or repeated waves alone do not stop a fanout run. For the other modes the conclusion is the latest brief's leading conclusion, and `stop` says the run has settled or stopped adding useful findings. Every evaluation, with its jobs, hits and the decider's notes, is in `/state/evaluations.jsonl`. To continue a run, resume the OpenEvolve tool from its checkpoint with `--checkpoint` into a new `--output-dir`. A stopped run is not proof that its ruling is right; for a question that matters, compare independent runs. ## Other modes Without `--fanout`, every agent researches the whole question from the current brief and the endpoint's model writes the briefs (`--team` has agents each push one line; `--rater`, `--arm` and `--style` change how work is graded and what agents look for; `init --help` lists them). These modes can also research the web (`--web`): the agents then run `ddgs` and `trafilatura` in their shell and need network access, which Codex's sandbox blocks unless the subagent tool was installed with `--network on` (or `--sandbox off`). Modes that fork an agent's session (the `C2` spawn type and the `R1` and `R2` raters) use the subagent tool's `fork_of`, in the published tool since 2026-10-19.99. `--writer-fanout` is an experiment we have not measured yet. Each candidate brief names its lines of inquiry under `LINES:`. Evaluating it runs one research agent per line, all at once, through `agents.mjs`; each reports exact passages and never answers. Then one writer agent gets every report, may re-check with its own tools, and answers with the quotations it rests on and the task's final line. The decider (`--decider`) scores that answer and its evidence and, as OpenEvolve's proposer, evolves the lines. `--lines` sets the agents per candidate (default 4), `--candidates` the candidates evaluated at once (default: the endpoint's agent count divided by `--lines`), `--endpoint-agents` that agent count, and `--seed-lines` the seed brief's lines (`override` hunts amendments, preemption, court decisions and exceptions; `neutral` suits other tasks). None has an upper limit. `status` reports the highest-scored writer answer of the latest wave as `conclusion` and says stop by the fanout rule above. No agent has a step limit: subagent workers compact a long context into a handoff instead of stopping. Every fixed number the add-on uses is in `openresearch/constants.toml`.